How to include and install local dependencies in setup.py in Python?

Question

I am creating a setup.py to distribute my application. This application has a number of dependencies which can be installed via pip, it also has some custom dependencies which can not be installed from PyPI.

So, I have created a custom_package_0.1.whl which will be included into the distribution and must be installed as a dependency after setup.py installs everything from install_requires.

Imagine the following app structure:

my_app/
    win_deps/custom_package_0.1.whl
    my_app/
        __init__.py
        main.py
        setup.py
        setup.cfg

How do I do that?

Carangid

I had a similar problem and found a satisfying answer here – Caputto

It's a shame for such widely known language to not support local dependencies in a non hacky way... – Stanislaw

Answer 1

it is possible but not sure what setuptools version you should use. steps:

in setup.py

setup(
  ...,
  install_requires=['my-package'],
  dependency_links=[
    # location to your egg file
    os.path.join(os.getcwd(), 'deps', 'my_package-1.0.0-py3.5.egg')
  ]
)

important thing is that your location should not pass URL pattern test and egg file name should have structure <package_name_with_no_hyphens>-<version>-<py_version>.egg

Googol

It seems like this wouldn't work if you haven't already installed my-package because there is no .egg until you install it, right? – Nauplius

@Nauplius I don't think that you need to install my-package with conda or something different. setuptools reads it from local location I guess – Googol

dependency_links has been removed – Hydranth

this does not work for newer pip versions, scroll to my answer for something that works with pip >= 19.1 – Alissaalistair

Answer 2

There is a new technique (Since version 19.1) called Direct references. Just pretend like your file is hosted on localhost.

from setuptools import setup

path_to_my_project = "/home/user/projects/my_package"  # Do any sort of fancy resolving of the path here if you need to


setup(# ... other arguments
      install_requires=[f"my_package @ file://localhost/{path_to_my_project}#egg=my_package"]
      )

Alissaalistair

This does not seem to work with Python 3.10 and setuptools 59. I get the error message Couldn't find index page for <package>' (maybe misspelled?) with either localhost or localbuilds specified. – Duodenum

Nothing in the release history suggests that something changed here – Alissaalistair

Yeah, I think that this is an issue in setuptools, rather than pip. I stumbed over this here: #70650546 – Duodenum

Answer 3

Based on @RunOrVeith answer above this works for a local wheel file using a relative path. So it can be used on various hosts to install a third party package. Works on Windows and Unix.

setup(# ... other arguments
      install_requires=[
          f"my-lib @ file://localhost/{os.getcwd()}/libs/my_lib-xxx-none-any.whl"
      ]
)

Estevan

I'm not sure that if depending on 'cwd' is the correct way, what if cwd changes ? – Contortionist

Since the ".../libs" is relative to the setup.py seems ok to me. If you move the entire project to another folder run "pip install -e ." and you should be fine. – Estevan

Answer 4

Extending wiesiu_p's answer, you can install the dependency by linking to its source directory, which has its own setup.py.

Assume you have the source files of your dependency my-dependency, and the root of my-dependency has its own setup.py. In your application's setup.py:

setup(
  ...,
  install_requires=['other-dependency','my-dependency'],
  dependency_links=[
    # location to your my-dependency project directory
    ''.join(['file:\\', os.path.join(os.getcwd(), 'path', 'to', 'my-dependency#egg=my-dependency-1.0')])
  ]
)

Now if you run python setup.py install with your application's setup.py, it will install my-dependency.

Plaided

I know this comment is old but I cannot get it to work. I tried with both files:\` as you have it and files://` which I think is correct? doing pip install --find-links=<dep_link> my_package works, but not when I put it in setup.py – Eschalot

I've added some clarification to the answer. I hope it helps. – Plaided

Note that this might fail if the local dependency itself has local dependencies, because the getcwd() might be called from site-packages. – Atlas

This does not work for me: it says that ERROR: No matching distribution found for my-dependency when I do pip install -e .. Even if I do python setup.py install I get an error error: Could not find suitable distribution for Requirement.parse('my-dependency'). I'm using pip 20.3.3. – Lasley

See my answer for how to get this to work with newer pip versions – Alissaalistair

Should that be file:// instead of file:\\ ? – Clavichord

@Sören Pretty sure it's file:\\. Did that not work for you? – Plaided

I've never seen an URI written that way, that's all. – Clavichord

Answer 5

The following worked for me when installing a local namespace package.

This is my directory structure:

mynamespace_project
├───namespace-package-a
│   ├───setup.py
│   └───namespace
│       └───a
│           ├───__init__.py
│           └───foo.py
└───namespace-package-b
    ├───setup.py
    └───namespace
        └───b
            ├───__init__.py
            └───bar.py

Then to require namespace-package-a in namespace-package-b, b would have the following setup.py:

setup(
      # ... other arguments
      install_requires=[
          f"package-a @ {(Path(__file__).parent.parent/"namespace-package-a").as_uri()}"
      ]
)

Running pip install namespace-package-b will then correctly install namespace-package-a with b.

Then in your code, you can import under the common namespace like so:

from namespace.a import foo
from namespace.b import bar

Multimillionaire

Answer 6

There are several options that you can choose from:

1. Upload your package to some server, and provide the URL with dependency_links.
2. You could put your python package inside of your my_app package and link it with packages parameter, instead of using the wheel file.
3. A more hacky way would be to use the setuptools api, and install the package by yourself.

Berget

For 2, how should I do that? I moved "win_deps" into "my_app" and added packages=(['win_deps']) but running python2 setup.py install does not automatically install all the .whl packages from "win_deps" directory – Carangid

What I meant in 2 was that instead of supplying the whl package, you could move the entire package directory custom_package_0.1 into this package directory. – Berget

oh, you mean just move custom_package_0.1 source files into win_deps? – Carangid

exactly. The problem is that setup.py does not have any available option to install a wheel package. The only way I can see is what I suggested you to do. The other ways are more hacky ways and I would not recommend you to do so. – Berget

Can you add more info for the third? Thanks. – Creeper

this is a terrible accepted answer. it has no details on the solutions – Eschalot