Setuptools "development" Requirements

Question

TL;DR: Is there a way to hook setuptool's 'develop' to install a set of development requirements when running python setup.py develop?

I'm building my first python package using setuptools. I'm specifying the requirements as:

requirements = [
    'click',
    'ansible',
    'fabric',
    'gitpython',
    'pyyaml',
    'jinja2',
    'yapsy'
]

test_requirements = [
    'pytest',
    'pytest-pep8',
    'pytest-cov',
]

setup(
...
    install_requires=requirements,
    tests_require=test_requirements,
...
)

During development, I've been installing the package (in a virtual environment) with:

python setup.py develop

and uninstalling with:

python setup.py develop -u

The package uses entry_points to install some command line scripts, so this sets up the commands for me and allows me to edit the package while testing the command at the same time.

I also have some dependencies that I use for development ... sphinx + extensions and a couple other things (things that aren't needed to use the package). I'm just manually installing them in the virtual environment at the moment. I didn't see any documentation (and haven't found any examples on the googles) about how to wire them in with setuptools.

Maybe there's a way to hook 'setup.py develop' to install an additional set of requirements? Another method I haven't read about?

Emory

For development purposes I would recommend using a requirements.txt to pin package versions for a development environment, then run pip install -r requirements.txt to deploy that environment. That's the kind of use case it's there for. – Bogbean

It would be really nice if setup.py had a develop_requires entry... – Tyrolienne

There seems to be a move towards a new pyproject.toml, but it is super hard to find information about that. Google consistently and stubbornly gives me information about setup.py. I don't want both setup.py and pyproject.toml. I'll rather give up on Python. – Sterilize

Answer 1

For more info on using setup.py vs requirements.txt, I found this article helpful.

Update: September 2016

I no longer use requirements.txt (see original answer below) for installing development only packages. The prevailing wisdom seems to be that requirements.txt should be used to pin deployments to specific version numbers, typically using pip freeze > requirements.txt. This ensures that the exact same versions of your project's dependencies and also your project's dependencies' dependencies are installed on all of your servers.

I instead use the extras_require option to setup.

requirements = [
    'click',
    'ansible',
    'fabric',
    'gitpython',
    'pyyaml',
    'jinja2',
    'yapsy'
]

setup({
    install_requires=requirements,
    extras_require={
        'dev': [
            'pytest',
            'pytest-pep8',
            'pytest-cov'
        ]
    }
})

Now, to install your package for development, you run pip install -e .[dev]. This installs all the regular required packages and those listed in the dev section of extras_require.

Production installs can still be done with python setup.py install or pip install . (or with a requirements.txt file).

Original Answer

Here is a way to do it that seems to be in keeping with the recommendations I've run into regarding setup.py vs requirements.txt. Specify all your production dependencies in the install_requires parameter of setup.py.

requirements = [
    'click',
    'ansible',
    'fabric',
    'gitpython',
    'pyyaml',
    'jinja2',
    'yapsy'
]

setup({
    # ...
    install_requires=requirements
    # ...
})

Then create a requirements.txt file that instructs pip to install your production dependencies from setup.py as well as your testing dependencies.

-e .

pytest
pytest-pep8
pytest-cov

Now you can install your package for development with pip install -r requirements.txt. The -e . line will install your package and its dependencies from setup.py in development mode. To install on production, you could use python setup.py install or pip install .. This will only install the dependencies listed in setup.py.

Edinburgh

This would probably confuse Heroku-like deployments, though, as they use requirements.txt for production requirements. – Tortile

I keep my development requirements in a file named Development -- there is no reason it has to be called requirements.txt if that confuses a host. – Semifluid

Downside: extra_require leaks to your clients. It is suggesting that you have an optional feature that can be enabled by doing pip install yourmodule[dev]. – Goat