Python setuptools: How can I list a private repository under install_requires?

Question

I am creating a setup.py file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:

from setuptools import setup
setup(name='my_project',
    ...,
    install_requires=[
        'public_package',
        'other_public_package',
        'private_repo_1',
        'private_repo_2',
    ],
    dependency_links=[
        'https://github.com/my_account/private_repo_1/master/tarball/',
        'https://github.com/my_account/private_repo_2/master/tarball/',
    ],
    ...,
)

I am using setuptools instead of distutils because the latter does not support the install_requires and dependency_links arguments per this answer.

The above setup file fails to access the private repos with a 404 error - which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can't figure out how to make setuptools authenticate.

Here are some things I've tried:

1. Use git+ssh:// instead of https:// in dependency_links as I would if installing the repo with pip. This fails because setuptools doesn't recognize this protocol ("unknown url type: git+ssh"), though the distribute documentation says it should. Ditto git+https and git+http.

2. https://<username>:<password>@github.com/... - still get a 404. (This method doesn't work with curl or wget from the command line either - though curl -u <username> <repo_url> -O <output_file_name> does work.)

3. Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.

Currently I just have setup.py print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there's something obvious that I'm missing, but can't think what it might be. :)

Duplicate-ish question with no answers here.

Wrongdoer

Answer 1

I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508 standard should be used, from [2] I retrieved an example which actually does work (at least for my case).

Please note; this is with pip 20.0.2 on Python 3.7.4

setup(
    name='<package>',
...
    install_requires=[
        '<normal_dependency>',
         # Private repository
        '<dependency_name> @ git+ssh://[email protected]/<user>/<repo_name>@<branch>',
         # Public repository
        '<dependency_name> @ git+https://github.com/<user>/<repo_name>@<branch>',
    ],
)

After specifying my package this way installation works fine (also with -e settings and without the need to specify --process-dependency-links).

References [1] https://github.com/pypa/pip/issues/4187 [2] https://github.com/pypa/pip/issues/5566

Junto

If you use ssh:// and run into Could not resolve hostname change the : to / in your clone url. I had this error with gitlab. – Sorority

That doesn't seem to work anymore as setuptools seems to look for a package on PyPi with the dependency name: Reading https://pypi.org/simple/some-fake-name/, and then Couldn't find index page for 'some_fake_name' (maybe misspelled?). In the end, the last error displayed is error: Could not find suitable distribution for Requirement.parse('some_fake_name@ git+ssh://[email protected]/cglacet/quadtree.git') – Bearce

Emphasis that <dependency_name> is not the same as the <repo_name> AND you may replace 'ssh://git@' with 'https://' if it is a public repo – Xanthochroism

@Anusha Sorry, I am not able to reproduce cglacet's error. His/her repository is not private and I expect Phil's suggestion to resolve it. Have you tried Phil's suggestion? – Junto

@TomHemmes I am trying to install a local package, so in install_requires I have <package-name> @ file://localhost/lib/<package-name>/<package_name>.version.whl and I get the same error as @Bearce – Olnek

warning!!!! when you pip upgrade, it seems that even the branch is moving to some new commit, pip regard them as satisfy! I have posted a question here question – Consignment

To add a bit more info on @Sorority comment, the SSH URL that is given by Gitlab is not interpretable because it has : that separates the domain name from the group com:<group>. Eg : git@gitlab.<your_org>.com:<group>/<project>.git to be replaced by git@gitlab.<your_org>.com/<group>/<project>.git – Larue

Answer 2

Here's what worked for me:

  install_requires=[
      'private_package_name==1.1',
  ],
  dependency_links=[
      'git+ssh://[email protected]/username/private_repo.git#egg=private_package_name-1.1',
  ]

Note that you have to have the version number in the egg name, otherwise it will say it can't find the package.

Melleta

Hi vadimg - Which version of setuptools/distribute are you using? I get "Unknown url type: git+ssh" using distribute 0.7.3 (the latest version). – Wrongdoer

This did not work for me 3 years later... Got: Could not find a version that satisfies the requirement – Colp

As of Oct 28, 2016, this approach seems broken – Accommodating

It works for me with pip==9.0.1 and setuptools==34.3.1 after I set the same version number in install_requires and dependency_links. But it seems only work when I run python setup.py install, not working on wheels and eggs with pip install. – Alben

With modern pip's you need to include an option, something like pip install --process-dependency-links ... – Refresh

This also works with HTTPS: git+https://github.com/<company or username>/private_repo.git#egg=private_package-0.0.1 Of course, it will prompt for password (unless it is stored in a keyring). It also requires --process-dependency-links option if installing via pip. – National

--process-dpendency-links is deprecated, see my answer using PEP508 url specification – Junto

I have the following error: unknown url type: git+ssh or unknown url type: git+https (Python 3.7) – Bearce

Answer 3

I couldn't find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.

GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you're pointing to a git repo:

from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']

setup(
    # ...
    install_requires='package',
    dependency_links = [
    'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
        .format(github_token=github_token, package=package, version=master)
        ] 

A couple of notes here:

• For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
• You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
• You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
• version can be a branch, a tag, or a commit hash
• You need to supply --process-dependency-links if installing from pip

Ramage

I am getting a cannot find tag or branch message. Despite the fact that the private repo I am attempting to clone does have a tag. – Jerid

figured out what the problem was. The tag had a v prepended to it in github. So I needed to use v1.1.0 instead of 1.1.0 in my setup.py script. – Jerid

Answer 4

I found a (hacky) workaround:

#!/usr/bin/env python

from setuptools import setup
import os

os.system('pip install git+https://github-private.corp.com/user/repo.git@master')

setup( name='original-name'
     , ...
     , install_requires=['repo'] )

I understand that there are ethical issues with having a system call in a setup script, but I can't think of another way to do this.

Cornell

yes this was also an ugly workaround for us due to the following: github.com/pypa/pip/issues/2822 – Survance

This is the only way I could get it to work, although I went with import pip. Neither @vadimg's answer or this suggestion in pypa/pip worked. – Plan

This will install a dependency even if running something unrelated to installation like python setup.py --version. – Terrify

Answer 5

Via Tom Hemmes' answer I found this is the only thing that worked for me:

    install_requires=[
        '<package> @ https://github.com/<username>/<package>/archive/<branch_name>.zip']

Imperial

Answer 6

Using archive URL from github works for me, for public repositories. E.g.

dependency_links = [
  'https://github.com/username/reponame/archive/master.zip#egg=eggname-version',
]

Wingo

How do you find the eggname-version? – Bearce

Answer 7

With pip 20.1.1, this works for me

install_requires=[ "packson3@https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],

in setup.py

Jud

Answer 8

Here is another way (free):

1. Clone GitHub Repo to GitLab Repo (free account) link
2. Add GitLab Repo Access Key (read only) [repository -> settings -> access tokens]
3. Use as dependency: https://oauth2:GITLAB_ACCESS_TOKEN>@gitlab.com/<GITLAB_USERNAME>/<PROJECT_NAME>.git

---

Q&A:

Q: Why through GitLab? A: Because GitHub only has Account Access Keys, GitLab has Repository Access Keys!

Q: Access Tokens in git history? A: Yes, because the whole point is to use a privat repo in privat context, the token is read-only, and the token expires after maximum of 1 year.

Q: Why is the GitLab Repository not synced automatically with the GitHub Repo? A: Mirroring is a GitLab premium feature, but you can do it for free if you use GitHub Actions, here is a tutorial.

Forsythia

Answer 9

Edit: This appears to only work with public github repositories, see comments.

dependency_links=[
    'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1',
    'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2',
],

Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding "#egg=project_name-version" to VCS dependencies is documented in the link you gave to distribute documentation.

Perihelion

I still get the same 404 error. Are the repos in your case private? I am aware of the #egg= syntax but don't know that it affects authentication. – Wrongdoer

Sorry, it of course was a public repo project. There probably is not any way to do this with current setuptools if https//<username>:<password>@... tarball URLs don't work with githubs private repositories as this question seems also to indicate: having-trouble-downloading-git-archive-tarballs-from-private-repo – Perihelion

You could try also development version of setuptools and/or check the code. The relevant code seems to be: package_index:fetch_distribution():534 package_index.py:_download_url():736 – Perihelion

Answer 10

This work for our scenario:

1. package is on github in a private repo
2. we want to install it into site-packages (not into ./src with -e)
3. being able to use pip install -r requirements.txt
4. being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt

https://github.com/pypa/pip/issues/3610#issuecomment-356687173

Degree