'pip setup.py bdist_wheel' no longer builds forced non-pure wheels

Question

I have a project that compiles with C extensions on Linux, but without them on Windows. When I first generated the wheel files on Windows with python setup.py bdist_wheel, they became universal, and I could not upload them to PyPI as these universal wheels are preferred by pip for installation over the .tar.gz uploads (the result from python setup.py sdist).

The trick around this was to specify in the setup.py:

Distribution.is_pure = lambda *args: False

or by subclassing Distribution:

class BinaryDistribution(Distribution):
    def is_pure(self):
        return False

and calling setup() in setup.py with the extra keyword argument distclass=BinaryDistribution,.

This all worked fine on my VM running Windows XP 64 which has 32 and 64 bit versions of Python 2.6/2.7/3.3/3.4 and pypy installed just for this purpose. A simple batch file gives me:

dist/pkg-1.0-cp26-none-win32.whl
dist/pkg-1.0-cp26-none-win_amd64.whl
dist/pkg-1.0-cp27-none-win32.whl
dist/pkg-1.0-cp27-none-win_amd64.whl
dist/pkg-1.0-cp33-none-win32.whl
dist/pkg-1.0-cp33-none-win_amd64.whl
dist/pkg-1.0-cp34-none-win32.whl
dist/pkg-1.0-cp34-none-win_amd64.whl

and the appropriate package gets downloade and installed by pip when you run pip on Windows and when you run pip on Linux you get the

pkg-1.0.tar.gz

which includes the C sources which are compiled during installation.

The problem started with the fact that I don't have a spare Windows 7 licensed machine where I can install Python 3.5 (it doesn't install on the EOL XP). So I investigated Appveyor and created appveyor.yml:

environment:
  matrix:
    - PYTHON: C:\Python27
    - PYTHON: C:\Python33
    - PYTHON: C:\Python34
    - PYTHON: C:\Python35
    - PYTHON: C:\Python27-x64
    - PYTHON: C:\Python33-x64
      DISTUTILS_USE_SDK: '1'
    - PYTHON: 'C:\Python34-x64'
      DISTUTILS_USE_SDK: '1'
    - PYTHON: 'C:\Python35-x64'

install:
  - |
    %PYTHON%\python.exe -m pip install --upgrade pip
    %PYTHON%\python.exe -m pip install wheel

build: off

test_script:
  - echo Skipped for now

after_test:
  - |
    %PYTHON%\python.exe setup.py bdist_wheel

artifacts:
  - path: dist\*

With the exact same source the result from the above eight calls to python setup.py bdist_wheel are:

pkg-1.0-py2-none-any.whl
pkg-1.0-py3-none-any.whl

And if you upload these to PyPI, Linux prefers them over the .tar.gz leading to non-inclusion of the C extension code.

What causes this, and how can I use Appveyor to build my .whl files (or at least the ones for Python 3.5?

Burnett

I encoutered the exact same problem just now, with Windows 7 and python 3.5. Reverting to wheels 0.24.0 indeed fixes the problem. I tried looking for when wheel 0.25+ were released but cannot find the tag on the bitbucket repo. Any idea ? – Cynara

@Cynara That was 2015-09-17. such info can easily be found by looking up wheel on pypi (search on pypi itself will show you some older versions, or change the end of the URL from 0.26.0 yourself) – Burnett

@Cynara look at my updated answer, with 0.28 and the --plat-name option you can now even generate the .whl files on a Linux machine if they are pure. – Burnett

Answer 1

I've just run into this issue myself with Python v2.7 and wheel v0.29.0 on Windows 7 x64, where I build a Python package with some pre-compiled extensions (complicated VisualStudio setup with SWIG and external DLLs).

After examining the source code I have found that overriding Distribution.has_ext_modules works (automatically includes platform name and ABI tag):

from setuptools import setup
from setuptools.dist import Distribution

DISTNAME = "packagename"
DESCRIPTION = ""
MAINTAINER = ""
MAINTAINER_EMAIL = ""
URL = ""
LICENSE = ""
DOWNLOAD_URL = ""
VERSION = '1.2'
PYTHON_VERSION = (2, 7)


# Tested with wheel v0.29.0
class BinaryDistribution(Distribution):
    """Distribution which always forces a binary package with platform name"""
    def has_ext_modules(foo):
        return True


setup(name=DISTNAME,
      description=DESCRIPTION,
      maintainer=MAINTAINER,
      maintainer_email=MAINTAINER_EMAIL,
      url=URL,
      license=LICENSE,
      download_url=DOWNLOAD_URL,
      version=VERSION,
      packages=["packagename"],

      # Include pre-compiled extension
      package_data={"packagename": ["_precompiled_extension.pyd"]},
      distclass=BinaryDistribution)

Distinction

Answer 2

An alternative that seems to do the same as the accepted answer but more concisely is this:

from setuptools import setup

DISTNAME = "packagename"
DESCRIPTION = ""
MAINTAINER = ""
MAINTAINER_EMAIL = ""
URL = ""
LICENSE = ""
DOWNLOAD_URL = ""
VERSION = '1.2'
PYTHON_VERSION = (2, 7)


setup(name=DISTNAME,
      description=DESCRIPTION,
      maintainer=MAINTAINER,
      maintainer_email=MAINTAINER_EMAIL,
      url=URL,
      license=LICENSE,
      download_url=DOWNLOAD_URL,
      version=VERSION,
      packages=["packagename"],

      # Include pre-compiled extension
      package_data={"packagename": ["_precompiled_extension.pyd"]},
      has_ext_modules=lambda: True)

Steiermark

Answer 3

The difference, of course, is in the environment, on the correctly working Win XP there is an older version of the wheel package installed (0.24.0) whereas on Appveyor the latest and greatest (and broken) version 0.26 of wheel gets installed (0.25 is broken as well).

Changing the install stanza in the YAML file to fix the wheel version:

install:
  - |
    %PYTHON%\python.exe -m pip install --upgrade pip
    %PYTHON%\python.exe -m pip install wheel==0.24

is enough to get this to work quickly.

You should however upgrade your the wheel package on your Linux box to version 0.28 and then use the new commandline option --plat-name:

python setup.py sdist
python2 setup.py bdist_wheel --plat-name win32
python2 setup.py bdist_wheel --plat-name win_amd64
python3 setup.py bdist_wheel --plat-name win32
python3 setup.py bdist_wheel --plat-name win_amd64

that will generate:

pkg-1.1.tar.gz
dist/pkg-1.1-py2-none-win32.whl
dist/pkg-1.1-py2-none-win32.whl
dist/pkg-1.1-py3-none-win_amd64.whl
dist/pkg-1.1-py3-none-win32.whl
dist/pkg-1.0-cp34-none-win_amd64.whl

which you can upload to PyPI and results in the correct (.tar.gz) file downloading on Linux and the appropriate wheel on Windows. By just making sure that if the --plat-name win... is specified setup() is called with ext_modules=None. The resulting wheel files have minor (line endings in 3 files and their SHA256SUM), but install normally on Windows.

That way you no longer need to build these packages, that are essentially pure packages, on a Windows machine

For me this change by Nate Coraor brings my total build time down from 15+ minutes to about 7 seconds

Burnett