Setuptools unable to use link from dependency_links

Question

I've been trying to install a package with the following setup configured:

setup(
    packages=find_packages(),
    include_package_data=True,
    install_requires=[
        'Django==1.5.1',
        'xhtml2pdf',
    ],
    dependency_links=[
        'https://github.com/chrisglass/xhtml2pdf/zipball/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947#egg=xhtml2pdf',
    ],
)

However it installs the XHTML2PDF package from PyPi, instead of using the specified link. According to the output (I ran the install using pip install -vvv package.tar.gz), it could either not parse the version from the link (at // 1 in code), or I've not specified the correct project name (at // 2 in code):

Downloading/unpacking xhtml2pdf (from mypackage==1.3)
  Getting page https://pypi.python.org/simple/xhtml2pdf/
  URLs to search for versions for xhtml2pdf (from mypackage==1.3):
  * https://pypi.python.org/simple/xhtml2pdf/
  * https://github.com/chrisglass/xhtml2pdf/zipball/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947#egg=xhtml2pdf
  Getting page https://github.com/chrisglass/xhtml2pdf/zipball/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947
  Analyzing links from page https://pypi.python.org/simple/xhtml2pdf/
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.1.tar.gz#md5=9f83a2bcb67858aca9e058761f3bea7b (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.1
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.2.tar.gz#md5=d640ccb9470942fd2e6d3ae740c27dc1 (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.2
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.1.zip#md5=4ad41c845735ae14da99085311d84c00 (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.1
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.4.zip#md5=5f035cd6532bef99b7d35054caaa6ef7 (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.4
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.3.zip#md5=32599c74f26f57ebd002765741ec64f7 (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.3
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.2.zip#md5=4047a8234eb6b77591d526dcb1f60161 (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.2
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.3.tar.gz#md5=13b0d6059b72c994473fddfa7a528451 (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.3
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.5.zip#md5=8db99aae8536436a2b7b0b3987197b99 (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.5
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.5.tar.gz#md5=1ef268b40c11bf966f7c6c5504299e3e (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.5
    Found link https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.4.tar.gz#md5=36b015a4e2918460711cbc5eebe026ce (from https://pypi.python.org/simple/xhtml2pdf/), version: 0.0.4
  Analyzing links from page https://codeload.github.com/chrisglass/xhtml2pdf/legacy.zip/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947
  Could not parse version from link: https://github.com/chrisglass/xhtml2pdf/zipball/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947#egg=xhtml2pdf  // 1
  Skipping link https://github.com/chrisglass/xhtml2pdf/zipball/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947#egg=xhtml2pdf; wrong project name (not xhtml2pdf)  // 2
  Using version 0.0.5 (newest of versions: 0.0.5, 0.0.5, 0.0.4, 0.0.4, 0.0.3, 0.0.3, 0.0.2, 0.0.2, 0.0.1, 0.0.1)
  Downloading xhtml2pdf-0.0.5.zip (118kB): 
  Downloading from URL https://pypi.python.org/packages/source/x/xhtml2pdf/xhtml2pdf-0.0.5.zip#md5=8db99aae8536436a2b7b0b3987197b99 (from https://pypi.python.org/simple/xhtml2pdf/)
...Downloading xhtml2pdf-0.0.5.zip (118kB): 118kB downloaded

If I add the version number to the package (xhtml2pdf-0.0.5) in both install_requires and dependency_links it finds the Git links and doesn't complain about them, but still installs from PyPi.

I've been searching for an answer for a while now, but haven't been able to find the solution. I need the Git version, because it has fixes that the PyPi version doesn't have yet.

Cupule

possible duplicate of How can I make setuptools install a package from another source that's also available on pypi with the same version number? – Juanjuana

Yes, it IS a duplicate of How can I make setuptools install a package from another source that's also available on pypi with the same version number?, which I already announced in the answer I gave to my own question. – Cupule

Answer 1

Ok, so I solved (or hacked, depending on how you look at it) my own question.

There were two issues with the code.

The first issue was that the link was incorrect. Although I've seen it formatted like this elsewhere, it wouldn't work for me (whether using setuptools mkvirtualenv mypackage or distribute mkvirtualenv --distribute mypackage). I figured it needed the zip extension at the end and found out on Github you can download a specific commit using the https://github.com/user/project/archive/commit.zip format.

Then I still had the problem that, although it would now recognise the link, it would still pick the PyPi version over the Github version. The reason for this is that the last commits (after the release to PyPi) on Github still have the same version number as the latest version on PyPi. So it turns out this question is actually a duplicate of How can I make setuptools install a package from another source that's also available on pypi with the same version number?. To try to solve it I tried the following settings in my setup.py (some of these have been mentioned in other Stackoverflow questions as working):

install_requires=[],
dependency_links=['https://github.com/chrisglass/xhtml2pdf/archive/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947.zip',],

This didn't install xhtml2pdf at all.

install_requires=['xhtml2pdf',],
dependency_links=['https://github.com/chrisglass/xhtml2pdf/archive/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947.zip#egg=xhtml2pdf',],

This installed the latest PyPi version (0.0.5).

install_requires=['xhtml2pdf==0.0.5',],
dependency_links=['https://github.com/chrisglass/xhtml2pdf/archive/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947.zip#egg=xhtml2pdf-0.0.5',],

This installed the 0.0.5 version in PyPi. It actually makes sense that setuptools and distribute would choose the PyPi version over the Github version if the version number is the same, because the PyPi version should be more stable.

install_requires=['xhtml2pdf==0.0.6',],
dependency_links=['https://github.com/chrisglass/xhtml2pdf/archive/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947.zip#egg=xhtml2pdf-0.0.6',],

This downloaded the package from Github (because version 0.0.6 doesn't exist on PyPi), however come install time it complained that the versions didn't match, because the version specified in the package is actually 0.0.5.

install_requires=['xhtml2pdf<=0.0.6',],
dependency_links=['https://github.com/chrisglass/xhtml2pdf/archive/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947.zip#egg=xhtml2pdf-0.0.6',],

This finally worked. Setuptools (and distribute) realised that the Github version must be newer since that's what I told it. And come install time it didn't complain about the fact that the actual version is 0.0.5 because it still meets the requirements.

You may consider this a bit of a hack. I'm not entirely happy with this solution. But I also have a deadline and it works. A word of caution: I will start having issues when version 0.0.6 is actually released. I want to properly pin my dependencies, so I don't run into issues in the future when I'm reinstalling, and this is not a proper pin. However I suppose you could fiddle around with the version number using this documentation: https://pythonhosted.org/setuptools/setuptools.html#specifying-your-project-s-version. Key is that the version should (pretend to) be higher than 0.0.5, but also a number (with optional characters) that you won't expect to show up as a release in PyPi in the future (and obviously lower than what the next release number will most likely be (0.0.6 in my case)).

I hope this was helpful to anyone else :).

Note: I use a requirements.txt file which gets processed by the setup.py file to extract the requirements. When you use https://github.com/chrisglass/xhtml2pdf/archive/28d12fcaafc4c47b13f1f6f42c2bfb73f90cc947.zip#egg=xhtml2pdf-0.0.6 in your requirements.txt file it will also still install fine using pip install -r requirements.txt.

Cupule

This is bound to break though when upstream moves to the next version increment as well? – Limey

Yes, I mentioned that in the last-but-two paragraph with a word of caution: "A word of caution: I will start having issues when version 0.0.6 is actually released. I want to properly pin my dependencies, so I don't run into issues in the future when I'm reinstalling, and this is not a proper pin. ..." – Cupule