How to write setup.py to include a Git repository as a dependency

Question

I am trying to write setup.py for my package. My package needs to specify a dependency on another Git repository.

This is what I have so far:

from setuptools import setup, find_packages

setup(
    name='abc',
    packages=find_packages(),
    url='https://github.abc.com/abc/myabc',
    description='This is a description for abc',
    long_description=open('README.md').read(),
    install_requires=[
        "requests==2.7.0",
        "SomePrivateLib>=0.1.0",
        ],
    dependency_links = [
     "git+git://github.abc.com/abc/SomePrivateLib.git#egg=SomePrivateLib",
    ],
    include_package_data=True,
)

When I run:

pip install -e https://github.abc.com/abc/myabc.git#egg=analyse

I get

Could not find a version that satisfies the requirement SomePrivateLib>=0.1.0 (from analyse) (from versions: ) No matching distribution found for SomePrivateLib>=0.1.0 (from analyse)

What am I doing wrong?

Addam

Note that setup.py and pip are completely different systems. One issue that I had was that I was able to get this working for pip but not for setup.py. – Chamfer

Answer 1

Note: this answer is now outdated. Have a look at this answer for up-to-date instructions: https://mcmap.net/q/142134/-how-to-write-setup-py-to-include-a-git-repository-as-a-dependency

---

You can find the right way to do it here.

dependency_links=['http://github.com/user/repo/tarball/master#egg=package-1.0']

The key is not to give a link to a Git repository, but a link to a tarball. GitHub creates a tarball of the master branch for you if you append /tarball/master as shown above.

Red

Is it possible to disable server certificate verification on downloading the dependency ? – Smallscale

@Eugen, there's a --trusted-host option, but I am not sure if it helps. You might get a good answer if you ask in a new question. – Red

I've found #29171130 – Smallscale

looks like this method is deprecated per github.com/pypa/pip/issues/3939 – Kaohsiung

This method is also useless for private repositories, since there's no way to authenticate. – Finella

@tedivm, according to the docs, it should in principle be possible to give a git url there instead of https, so I guess you can make it work with private repositories. (see setuptools.readthedocs.io/en/latest/…) If you manage to get it to work it may be worth to post this as a separate answer here. – Red

I did manage to get it working and have added another answer. – Finella

This doesn't seem to work (anymore?), @DickFox's answer is the way to go. – Kilowatthour

The /tarball/master method does not work for gitlab – Electrode

Deprecated. Correct answer is to use Pep508, answered by @Dick Fox below – Rouen

Answer 2

Note: this answer is now outdated. Have a look at this answer for up-to-date instructions: https://mcmap.net/q/142134/-how-to-write-setup-py-to-include-a-git-repository-as-a-dependency

---

After digging through the pip issue 3939 linked by @muon in the comments above and then the PEP-508 specification, I found success getting my private repo dependency to install via setup.py using this specification pattern in install_requires (no more dependency_links):

install_requires = [
  'some-pkg @ git+ssh://[email protected]/someorgname/[email protected]#egg=some-pkg',
]

The @v1.1 indicates the release tag created on github and could be replaced with a branch, commit, or different type of tag.

Endorsement

Note: This works fine for local/private packages, however, you cannot release a package to PyPI that uses this syntax in its setup.py – Towny

@Towny Could you please provide a link to official statement? – Retardation

Note you can do git+https://github.com if you don't want to use SSH. – Kilowatthour

So what is the correct approach for doing a --upgrade? Even though I specify a tag version an upgrade just ignores newer tag versions – Trug

@Retardation Not super official, but these are at least comments on the pip GitHub project from actual members of the PyPA: github.com/pypa/pip/issues/4187#issuecomment-415667805 and further explanation: github.com/pypa/pip/issues/4187#issuecomment-415067034 – Byler

how do you specify a commit instead of a release version? – Elastin

Is there a protocol that works both for pip requirements files and install_requires? I usually use the pattern install_requires=open("requirements.txt", "r").read().splitlines() – Envelope

thanks, how can we specify the whl file form the github location? – Subtype

@Trug Did you came across any solution to the upgrade path ? – Unlikelihood

Thanks a lot. This worked for me after 2 days of banging my head with setup.py. It worked like a charm for my private/internal repository. Just one difference is I didn't add the #egg part. – Corned

Could save someone's effort. my-dependent-sdk @ git+https://<mypersonaltoken>@github.private.com/myorg/my-dependent-sdk.git@master – Corned

This does not seem to work if then installing via python setup.py develop. Instead using pip install -e . worked for me. – Lima

In one of the answers below, it suggests removing the #egg= section. I needed that piece of information to get it to work for my repo. – Outhaul

@EduardoPignatelli if someone is looking for that as well you can run for each line: line = re.sub(r'(git\+.*egg=(.*))', '\2 @ \1', line) – Greenshank

Answer 3

This answer has been updated as Python has evolved over the years.

Scroll to the bottom for the most current answer, or read through to see how this has evolved.

---

Unfortunately the accepted answer does not work with private repositories, which is one of the most common use cases for this.

2019 - dependency_links (deprecated as of pip v19+)

I eventually did get it working with a setup.py file that looks like this method:

from setuptools import setup, find_packages

setup(
    name = 'MyProject',
    version = '0.1.0',
    url = '',
    description = '',
    packages = find_packages(),
    install_requires = [
        # Github Private Repository - needs entry in `dependency_links`
        'ExampleRepo'
    ],

    dependency_links=[
        # Make sure to include the `#egg` portion so the `install_requires` recognizes the package
        'git+ssh://[email protected]/example_org/ExampleRepo.git#egg=ExampleRepo-0.1'
    ]
)

Newer versions of pip make this even easier by removing the need to use "dependency_links"-

from setuptools import setup, find_packages

setup(
    name = 'MyProject',
    version = '0.1.0',
    url = '',
    description = '',
    packages = find_packages(),
    install_requires = [
        # Github Private Repository
        'ExampleRepo @ git+ssh://[email protected]/example_org/ExampleRepo.git#egg=ExampleRepo-0.1'
    ]
)

However, with the very latest pip you'll run into issues with the EGG format handler. This is because while the egg is ignored pip is now doing direct URL matching and will consider two URLs, one with the egg fragment and the other without, to be completely different versions even if they point to the same package. As such it's best to leave any egg fragments off.

2021 June - setup.py

So, the best way (current to June 2021) to add a dependency from Github to your setup.py that will work with public and private repositories:

from setuptools import setup, find_packages

setup(
    name = 'MyProject',
    version = '0.1.0',
    url = '',
    description = '',
    packages = find_packages(),
    install_requires = [
        # Github Private Repository
        'ExampleRepo @ git+ssh://[email protected]/example_org/ExampleRepo.git'
    ]
)

2022 February - setup.cfg

Apparently setup.py is being deprecated (although my guess is it'll be around for awhile) and setup.cfg is the new thing.

[metadata]
name = MyProject
version = 0.1.1


[options]
packages = :find

install_requires =
  ExampleRepo @ git+ssh://[email protected]/example_org/ExampleRepo.git

2022 June - pyproject.toml

setup.cfg is already "pre" deprecated. as setuptools now has experimental support for pyproject.toml files.

This is the future, but since this is still experimental it should not be used in real projects for now. Even though setup.cfg is on its way out you should use it for a declarative format, otherwise setup.py is still the way to go. This answer will be updated when setuptools has stabilized their support of the new standard.

2023 January - pyproject.toml

It is now possible to define all of your dependencies in pyproject.toml. Other options such as setup.cfg still work.

[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[project]
dependencies = [
    'ExampleRepo @ git+ssh://[email protected]/example_org/ExampleRepo.git',
]

[project.optional-dependencies]
dev = ['ExtraExample @ git+ssh://[email protected]/example_org/ExtraExample.git']

2024 April

At this point the pyproject.toml file is the standard. There is literally no reason to use setup.cfg or setup.py (but they still work). If you are using them you should consolidate to a single pyproject.toml file.

Finella

could you please elaborate what -0.1 stands for in your approach? Do you take the version number from a git release or from the setup.py description? – Saltsman

From the setup.py file- if you want to use a specific branch or tag you format things a little differently. – Finella

"Unfortunately the other answer does not work with private repositories" This is no longer true Fox's answer does work on private repo without needing dependency_links (which is deprecated) – Candracandy

Thanks @Keto! I don't know why your edit got rejected but the mods, but I went ahead and overrode that rejection to add the deprecation notice to the answer. – Finella

This really ought to be the top answer, it's actually relevant in the current time. – Furmenty

This doesn't seem to work when running python setup.py install - it only works for me for pip install -e [module_name]. Is that true for all? This is on pip 21.1.3 – Councilwoman

I recommend always using pip and not using setup.py install. This answer has a lot more details on why- https://mcmap.net/q/144361/-difference-between-39-python-setup-py-install-39-and-39-pip-install-39 – Finella

I get "Host key verification failed." using ssh, but using https works, e.g: pkg @ git+https://github.com/user/pkg.git - also best answer here – Dom

You can resolve the host key verification by adding the host key to your known key list. Using github as an example, in a shell run ssh-keyscan github.com >> ~/.ssh/known_hosts. – Finella

Thanks for keeping this up-to-date! – Foe

Hey @RobertHafner would I then just import the package as import ExtraExample? I follow your steps but still seem to get ModuleNotFoundError. Thanks! – Shooter

@Shooter - the name and the module names should match but that's not a requirement- there are some libraries, like pyyaml, that have different names than their modules. – Finella

I just saw that you can even install a dependency with it's optional dependencies using pip install foo[dev]@git+https://github.com/foo/foo – Handspring

Answer 4

A more general answer: To get the information from the requirements.txt file I do:

from setuptools import setup, find_packages
from os import path

loc = path.abspath(path.dirname(__file__))

with open(loc + '/requirements.txt') as f:
    requirements = f.read().splitlines()

required = []
dependency_links = []

# Do not add to required lines pointing to Git repositories
EGG_MARK = '#egg='
for line in requirements:
    if line.startswith('-e git:') or line.startswith('-e git+') or \
            line.startswith('git:') or line.startswith('git+'):
        line = line.lstrip('-e ')  # in case that is using "-e"
        if EGG_MARK in line:
            package_name = line[line.find(EGG_MARK) + len(EGG_MARK):]
            repository = line[:line.find(EGG_MARK)]
            required.append('%s @ %s' % (package_name, repository))
            dependency_links.append(line)
        else:
            print('Dependency to a git repository should have the format:')
            print('git+ssh://[email protected]/xxxxx/xxxxxx#egg=package_name')
    else:
        required.append(line)

setup(
    name='myproject',  # Required
    version='0.0.1',  # Required
    description='Description here....',  # Required
    packages=find_packages(),  # Required
    install_requires=required,
    dependency_links=dependency_links,
)

Dope

Answer 5

Actually if you like to make your packages installable recursively (YourCurrentPackage includes your SomePrivateLib), e.g. when you want to include YourCurrentPackage into another one (like OuterPackage → YourCurrentPackage → SomePrivateLib) you'll need both:

install_requires=[
    ...,
    "SomePrivateLib @ git+ssh://github.abc.com/abc/[email protected]#egg=SomePrivateLib"
],
dependency_links = [
    "git+ssh://github.abc.com/abc/[email protected]#egg=SomePrivateLib"
]

And make sure you have a tag created with your version number.

Also if your Git project is private and you like to install it inside the container, e.g., a Docker or GitLab runner, you will need authorized access to your repository. Please consider to use Git + HTTPS with access tokens (like on GitLab: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html):

import os
from setuptools import setup

TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')

setup(
    ....

    install_requires=[
            ...,
            f"SomePrivateLib @ git+https://gitlab-ci-token:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
    ],
    dependency_links = [
            f"git+https://gitlab-ci-token:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
    ]
)

Updated:

You have to put #egg=SomePrivateLib at the end of dependency line if you like to have this dependency in requirements.txt file. Otherwise pip install -r requirements.txt won't work for you and you wil get something like:

ERROR: Could not detect requirement name for 'git+https://gitlab-ci-token:[email protected]/abc/[email protected]', please specify one with #egg=your_package_name

If you use reuirements.txt, this part is resposible for name of dependency’s folder that would be created inside python_home_dir/src and for name of egg-link in site-packages/

You can use a environment variable in your requirements.txt to store your dependency’s token value safe in your repo:

Example row in requrements.txt file for this case:

....

-e git+https://gitlab-ci-token:${EXPORTED_VAR_WITH_TOKEN}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib
....

Remonstrance

Answer 6

I was successful with these three options in GitLab. I am using version 11 of GitLab.

Option 1 - no token specified. The shell will prompt for username/password.

from setuptools import setup

TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')

setup(
    install_requires=[
        "SomePrivateLib @ git+https://gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
    ]
)

Option 2 - user access token specified. The token generated by going to GitLab → account top right → settings → access tokens. Create the token with read_repository rights.

Example:

import os
from setuptools import setup

TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')

setup(
    install_requires=[
        f"SomePrivateLib @ git+https://gitlab-ci-token:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
    ]
)

Option 3 - repository-level token specified. The token generated by going to the repository → settings → repository → deploy tokens. From here, create a token with read_repository rights.

Example:

import os
from setuptools import setup

TOKEN_USER = os.getenv('EXPORTED_TOKEN_USER')
TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')

setup(
    install_requires=[
        f"SomePrivateLib @ git+https://{TOKEN_USER}:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
    ]
)

In all three, I was able to do simply: "SomePrivateLib @ git+https://gitlab.server.com/abc/SomePrivateLib.git" without the #egg marking at the end.

Transit

Answer 7

This solution works for me when I run python setup.py install:

setuptools.setup(
    ...,
    install_requires=[
        'numpy',
        'pandas',
        'my_private_pkg'
        ],
    dependency_links=["git+https://github.com/[username]/[my_private_pkg].git@main#egg=my_private_pkg"],
    ...
)

Warmonger