Python requirements conflict with PyPi

Question

I have a project that needs some DevOps TLC, so I am finally building my installation script. This will eventually be a package that will be install-able by pip locally, but may not end up in PyPI.

It has a dependency for a module called u2py. It is this package, created for U2 Database operations, not this package, for... something else. The one I want is only ever installed by a 3rd party vendor (Rocket), the one I don't want is in PyPI.

What should be the expected behavior of my package in this case? I will include a blurb about this in my readme doc, but is that sufficient?

I've thought about throwing an exception to identify when the wrong package is present, but that makes me feel weird. It seems that maybe the most pythonic thing is to NOT add this to my install script, and blindly assume import u2py results in a module I can use. If it quacks like a duck, parses DynArrays like a duck, and call()s SUBROUTINEs like a duck, then it's a duck, right? Otherwise, if there is an error the user will just go and actually read the docs.

I've looked a classifiers, but not sure if they apply here.

Popcorn

Also, maybe whoever answers this question would have a shot at answering another similar one: #41603369 – Popcorn

".. if there is an error the user will just go and actually read the docs." No, they will (not) post their code on Stack Overflow and say "it dosent work". Are you prepared to handle such questions? You could throw an error that tells that it's not the right package. Make it bold, if the terminal supports that. – Pogrom

@usr2564301, what is the best way to disambiguate which package it is? Should I just try to check for the existence of a unique function, or something similar? Is that breaking any implicit duck typing contract? Am I thinking way too hard about this? – Popcorn

Let's wait what the Python Powerhouses may have to tell. It is an interesting topic. – Pogrom

If you publish this package on PyPI, how do you expect to satisfy the dependency on the u2py package from Rocket Software? – Balladry

@DustinIngram, I have no idea.The target audience would be those already working with the u2py environment, and my package isn't likely to make any new converts to this very old and niche tech. But, that leaves a lot of people who don't know about the u2 environment already, install to try it out, and watch their code catch fire and never work. It seems that I need to solve for the second scenario more than the first. – Popcorn

Answer 1

Ideally there would be a way at install-time (in setup.py) to detect whether the package is being installed into a "u2 environment" or not, and could fail the installation (with an appropriate error message) if that's the case.

With this solution, you won't be able to provide built distributions (wheels) since they don't execute the setup.py file at install-time, but just publishing source distributions should be fine.

Balladry

Wouldn't pip build a wheel anyway and cache it locally for the following installations, so that setup.py would be executed only once? https://mcmap.net/q/1779033/-prevent-pip-from-caching-a-package/11138259 – Wallow

Possibly, but if it's built locally, the check at build-time would succeed, and the built wheel would only be used on the platform which it was built on. – Balladry

Answer 2

It's a case where it would be nice if Python projects had namespaces (pip install com.rocket.u2py and import com.rocket.u2py as u2py).

From my point of view there are 2 aspects to consider: at the project level, at the package level.

1. project (distribution package)

I believe it is a bad practice to force alternative download sources onto the end user of your project. By default, pip should download from PyPI and nowhere else, unless the user decides it themselves (via --find-links or similar options, which you could instruct your users to do in your documentation).

Since it is such a niche dependency, I think I would simply not add it to install_requires. I would assume the end users of your project know about the dependency already and are able to install it themselves directly.

Also I don't believe it is possible to check reliably at install-time if the correct dependency is installed, since setup.py does not always run (overriding the bdist_wheel command can help, but probably not 100% effective).

2. package (importable package)

I am not sure some specific action is needed. The code would most likely fail sooner or later naturally, because of module or function is not importable. Which might be okay-ish, maybe?

But probably detecting if the dependency is installed (and it is the correct one), is relatively easy and would provide a better user experience. Either check that some specific modules or functions are importable. Or inspect the meta-data (import importlib_metadata; importlib_metadata.distribution('u2py').metadata['Author']).

In case of an application, I would try to fail gracefully as soon as possible. In case of a library I would try to find one strategic spot to place the check and raise a custom exception (CannotFindU2pyException).

---

Links:

• Prevent pip from caching a package
• https://docs.python.org/3/library/importlib.metadata.html#distribution-metadata
• https://github.com/pypa/pip/issues/4187#issuecomment-415067034
• Equivalent for `--find-links` in `setup.py`

Wallow

Answer 3

You can specify the url to the package in install_requires using setuptools (requires pip version 18.1 or greater).

Requirement specifiers

Example

setup.py

import setuptools

setuptools.setup(
    name='MyPackage',
    version='1.0.0',
    # ...
    install_requires=[
        'requests @ https://github.com/psf/requests/archive/v2.22.0.zip'
    ]
    # ...
)

and do python setup.py install

Also

Since version 19.1, pip also supports direct references like so:

SomeProject @ file:///somewhere/...

Ref

• https://www.python.org/dev/peps/pep-0508/
• https://github.com/pypa/pip/pull/4175

Conventioner

The u2py dependency isn't pip-installable, it comes from the environment the parent package is being installed in. This also doesn't handle the case where the user has the wrong u2py installed from PyPI. – Balladry