Build a wheel/egg and all dependencies for a python project

Question

In order to stage python project within our corporation I need to make an installable distribution.

This should include:

• An egg or whl for my project
• An egg or whl for every dependency of the project
• (optionally) produce a requirements.txt file listing all the installable components for this release

Is there an easy plug in, (e.g. an alternative to bdist_wheel) that will not only compile one wheel but also that project's components?

Obviously I can script this, but I was hoping that there might be a short-cut that builds the package + dependencies in fewer steps.

This needs to work on Python 2.7 on Windows + Linux.

Dossier

Answer 1

With poetry you can define your dependencies and metadata about your project in a file in the root of your project, called pyproject.toml:

[tool.poetry]
name = "my-project"
version = "0.1.0"
description = "some longer description"
authors = ["Some Author <[email protected]>"]

[tool.poetry.dependencies]
python = "*"

[tool.poetry.dev-dependencies]
pytest = "^3.4"

To build your project as a wheel, execute poetry build

$ poetry build

Building my-project (0.1.0)
- Building sdist
- Built my-project-0.1.0.tar.gz

- Building wheel
- Built my-project-0.1.0-py3-none-any.whl

a dist/ folder is created with a wheel for your project.

Globefish

this will still not add all the specified dependencies in the created wheel file. the individual dependencies will be downloaded when its installed. – Selfabsorbed

@Selfabsorbed a wheel is not meant to have all dependencies. Subsequently you could do pip install path/to/wheel.whl --target /path/to/some/folder and zip the contents of the 'folder' to have all your dependencies in the zip archive and ship that to the environment where you want to run your code. – Globefish

@VincentClaes What would you with that folder once it's on the target machine in order to invoke the program using the bundled dependencies? – Picco

Oh, I think I got it PYTHONPATH=some/dir python3 some/dir/myapp/__init__.py – Picco

Answer 2

You will need to create a setup.py file for your package. Make sure you have the latest setuptools and pip installed. Then run the following:

python setup.py bdist_wheel

This will create a wheel file for your package. This assumes you don't have C/C++ headers, DLLs, etc. If you do, then you'll probably have a lot more work to do.

To get dependencies, you will want to create a requirements.txt file and run the following:

pip wheel .

If your package isn't on PyPI, then you'll have to manually copy your package's wheel file into the wheel folder that this command creates. For more information see the following excellent article:

• http://lucumr.pocoo.org/2014/1/27/python-on-wheels/

Vail

Has the need to manually copy wheels for packages that aren't on PyPi changed from 2014 to 2018? I couldn't find any official discussion – Ruck

You can use pip to install using a path to a wheel file or a URL instead of PyPI. – Vail

To install the dependency wheels, use command: pip install --no-index --find-links=/local/wheels -r requirements.txt – Fretwork

Can you tell me how to include files such as dll, exe, etc. in it? Thanks! – Disputable

Your command creates one wheel per PyPi package, which is not what people want. – Tittivate

This now throws a SetuptoolsDeprecationWarning, use pip wheel . instead. – Heartfelt

Answer 3

With the latest pip and wheel, you can simply run

pip wheel .

within your project folder, even if your application isn't on PyPi. All wheels will be stored in the current directory (.).

To change the output directory (to for example, ./wheels), you may use the -w / --wheel-dir option:

pip wheel . -w wheels

All the options available are listed at the pip documentation.

Quach

It search my local packages on pypi so it fails. Instead, I run python setup.py bdist_wheel first, then run pip wheel -r requirements.txt for pypi packages. – Aniela

I just tried it and it writes the wheel files to your working directory by default (not ./wheelhouse). Also, any idea how to specify the name of the output file? I'm not seeing anything in the docs. – Luciusluck

This gives me a million wheels. How do I get one big wheel file? – Tittivate

@Tittivate Wheels are not meant to be one single installer for all dependencies. You can add --no-deps to only build your project. Actually pip wheel --no-deps -w dist . is quite similar to python setup.py bdist_wheel. – Noneffective

@Noneffective yes, I wanted a wagon - github.com/cloudify-cosmo/wagon - a zip file of all the wheels. Databricks and other systems use this format to avoid having a million wheels lying around for uploading or integration. – Tittivate