Best practices: how do you list required dependencies in your setup.py?

Question

This is how I do it currently:

import os
from setuptools import setup, find_packages
here = os.path.abspath(os.path.dirname(__file__))

requires = [
    'pyramid',
    'pyramid_debugtoolbar',
    'waitress',
    'requests',
    'mock',
    'gunicorn',
    'mongoengine',
    ]

setup(name='repoapi',
      version='0.0',
      description='repoapi',
      packages=find_packages(),
      include_package_data=True,
      zip_safe=False,
      install_requires=requires,
      tests_require=requires,
      test_suite="repoapi",
      entry_points="""\
      [paste.app_factory]
      main = repoapi:main
      """,
      )

Is this an okay way? I have some troubles. For example, for pyramid, I cannot use the system-wide nosetests plugin to run tests. I need to install pyramid in the global python site-packages!

But I don't want that. So I must install nose in the virtualenv of this project. But I don't want it to be a dependency. I don't feel like it should belong to requires. It isn't. Yet, I also don't want to install by hand all the time. Yeah I know I have a lot of I don't want to do this and that...

But how would you solve that? I don't want to tamper the global python site-packages, but I want to install nose as part of the virtualenv.

Also, pip install requirement files. It's slightly more accurate because I don't need to specify the version manually and I don't need to be afraid of updating setup.py manually. Just throw pip freeze > file.txt and done.

However, pip can return garbage because we throw garbage packages into virtualenv.

So many blades. What's the best practice? How do you deal with these issues?

Maybe I missed it, but https://github.com/django/django/blob/master/setup.py, how did Django do it?

Lon

setuptools tests_require is deprecated since 41.5.0 (October 2019). We are now at 50.3.0. – Twelvetone

Answer 1

You can split up your requirements into "install" dependencies and "test" dependencies like this:

import os
from setuptools import setup, find_packages
here = os.path.abspath(os.path.dirname(__file__))

install_requires = [
    'pyramid',
    'pyramid_debugtoolbar',
    'waitress',
    'requests',
    'gunicorn',
    'mongoengine',
    ]

tests_require = [
    'mock',
    'nose',
    ]

setup(name='repoapi',
      ...
      install_requires=install_requires,
      tests_require=tests_require,
      test_suite="nose.collector",
      ...
      )

This way, when someone installs the package, only the "install" dependencies are installed. So, if someone only wants to use the package (and they aren't interested in running the tests), then they don't have to install the test dependencies.

When you do want to run the tests, you can use this:

$ python setup.py test

Per the docs:

Note that these required projects will not be installed on the system where the tests are run, but only downloaded to the project’s setup directory if they’re not already installed locally.

Once the "test" dependencies are in place, then it will run the "test_suite" command. Since you mentioned nose as your preferred test runner, I showed how you use "nose.collector" to configure that.

Incidentally, the Django setup.py is not the cleanest example for understanding the basics of setuptools. I think the Sentry setup.py is a better example to learn from.

Koren

Thanks for sharing. But wouldn't nose actually get installed into virtualenv? So at the end of the day, the best practices is still documentation, bug ticket to keep setup.py in sync? Thanks. – Lon

Thanks for linking to the docs for tests_require. Having lots of difficult managing dependencies in Python. Couldn't find the property in the docs I was originally referring to: packaging.python.org/guides/… – Ovotestis

What if I want to install the test dependencies but not run the tests just yet? Is there a python setup.py install --all option? – Falda

setuptools tests_require is deprecated since 41.5.0 (October 2019). We are now at 50.3.0. – Twelvetone

Answer 2

If you are using requirement files, then an alternative would be to read their content instead of duplicating it:

import pathlib
from setuptools import setup, find_packages

HERE = pathlib.Path(__file__).parent
INSTALL_REQUIRES = (HERE / "requirements.txt").read_text().splitlines()
TESTS_REQUIRE = (HERE / "test-requirements.txt").read_text().splitlines()[1:]

setup(...,
      install_requires=INSTALL_REQUIRES,
      tests_require=TESTS_REQUIRE,
      ...
      )

I find this approach much more stable and less prone to errors, since requirements change sometimes and one often forgets to update two places.

Note: notice the TESTS_REQUIRE start from the second line, this is due to the fact that the first line of test-requirements.txt is often -r requirements.txt. Feel free to change it if your case is different.

Resolutive

setuptools tests_require is deprecated since 41.5.0 (October 2019). We are now at 50.3.0. – Twelvetone

@MartinThoma So what should we use instead? Docs don't seem to say. – Crannog

Just document how to run your tests, e.g. in the readme – Twelvetone

Pytest is the most common choice. tox is also pretty well known. If I see a pytest.ini or a tox.ini I would know how to run the tests. – Twelvetone