How to run unittest discover from "python setup.py test"?

Question

I'm trying to figure out how to get python setup.py test to run the equivalent of python -m unittest discover. I don't want to use a run_tests.py script and I don't want to use any external test tools (like nose or py.test). It's OK if the solution only works on python 2.7.

In setup.py, I think I need to add something to the test_suite and/or test_loader fields in config, but I can't seem to find a combination that works correctly:

config = {
    'name': name,
    'version': version,
    'url': url,
    'test_suite': '???',
    'test_loader': '???',
}

Is this possible using only unittest built into python 2.7?

FYI, my project structure looks like this:

project/
  package/
    __init__.py
    module.py
  tests/
    __init__.py
    test_module.py
  run_tests.py <- I want to delete this
  setup.py

Update: This is possible with unittest2 but I want find something equivalent using only unittest

From https://pypi.python.org/pypi/unittest2

unittest2 includes a very basic setuptools compatible test collector. Specify test_suite = 'unittest2.collector' in your setup.py. This starts test discovery with the default parameters from the directory containing setup.py, so it is perhaps most useful as an example (see unittest2/collector.py).

For now, I'm just using a script called run_tests.py, but I'm hoping I can get rid of this by moving to a solution that only uses python setup.py test.

Here's the run_tests.py I'm hoping to remove:

import unittest

if __name__ == '__main__':

    # use the default shared TestLoader instance
    test_loader = unittest.defaultTestLoader

    # use the basic test runner that outputs to sys.stderr
    test_runner = unittest.TextTestRunner()

    # automatically discover all tests in the current dir of the form test*.py
    # NOTE: only works for python 2.7 and later
    test_suite = test_loader.discover('.')

    # run the test suite
    test_runner.run(test_suite)

Fiester

Just a word of caution to anyone who happens to come here. setup.py test is considered a code 'smell' and is also set to be deprecated. github.com/pytest-dev/pytest-runner/issues/50 – Gemoets

Answer 1

If you use py27+ or py32+, the solution is pretty simple:

test_suite="tests",

Neutrino

I wish this worked better, I ran into this issue: #6164504 "Test names should match module names. If there is a "foo_test.py" test, there needs to be a corresponding module foo.py." – Patronymic

I agree. In my case, where I'm testing a Python external where there literally is no such Python module with a .py, there seems to be no good way to achieve this. – Shears

This is the correct solution. I didn't have the issue @CharlesL. had. All my tests are named test_*.py. In addition I found out that it will actually recursively search through the given directory to find any class that extends unittest.TestCast. This is extremely useful if you have a directory structure where in you have tests/first_batch/test_*.py and tests/second_batch/test_*.py. You can just specify test_suite="tests", and it will pick up everything recursively. Note that each nested directory will need to have an __init__.py file in it. – Celio

You need to do from setuptools import setup instead of from distutils.core import setup for this option to exist. I don't know why official docs seems to use distutils.core, I guess that's the default? – Caddie

stackoverflow.com/a/58534041 has a good description of this and other arguments. – Merla

Answer 2

From Building and Distributing Packages with Setuptools (emphasis mine):

test_suite

A string naming a unittest.TestCase subclass (or a package or module containing one or more of them, or a method of such a subclass), or naming a function that can be called with no arguments and returns a unittest.TestSuite.

Hence, in setup.py you would add a function that returns a TestSuite:

import unittest
def my_test_suite():
    test_loader = unittest.TestLoader()
    test_suite = test_loader.discover('tests', pattern='test_*.py')
    return test_suite

Then, you would specify the command setup as follows:

setup(
    ...
    test_suite='setup.my_test_suite',
    ...
)

Kosel

There is a problem with this solution, because it creates 2 "levels" of unittest. Meaning that setuptools will create a 'test' command that will try to create a TestSuite from setup.my_test_suite, which will force it to import setup.py, which will run the setup() again! This second time it will create a new (nested) test command that runs your desired test. This might not be noticeable to most people, but if you try to extend the test command (I needed to modify it because I can't run my tests 'in-place') you might run into weird issues. Use https://mcmap.net/q/238016/-how-to-run-unittest-discover-from-quot-python-setup-py-test-quot instead – Celio

This causes the tests to run twice for me for reasons mentioned above. Fixed it by moving the function into the __init__.py of the tests folder and referencing that. – Gutturalize

The issue with tests being executed twice can be easily fixed by executing setup() function inside if __name__ == '__main__': block in setup.py script. The first time the setup script is being executed, so the if block will be called; the second time the setup script will be imported as a module, so the if block will not be called. – Lafave

Hmm, I realize that my setup.py does NOT contain that test_suite parameter at all, yet "python setup.py test" still works fine for me. That is different that what the docs says: "If you did not set a test_suite in your setup() call, and do not provide a --test-suite option, an error will occur." Any idea? – Truculent

Answer 3

You don't need config to get this working. There are basically two main ways to do it:

The quick way

Rename your test_module.py to module_test.py (basically add _test as a suffix to tests for a particular module), and python will find it automatically. Just make sure to add this to setup.py:

from setuptools import setup, find_packages

setup(
    ...
    test_suite = 'tests',
    ...
)

The long way

Here's how to do it with your current directory structure:

project/
  package/
    __init__.py
    module.py
  tests/
    __init__.py
    test_module.py
  run_tests.py <- I want to delete this
  setup.py

Under tests/__init__.py, you want to import the unittest and your unit test script test_module, and then create a function to run the tests. In tests/__init__.py, type in something like this:

import unittest
import test_module

def my_module_suite():
    loader = unittest.TestLoader()
    suite = loader.loadTestsFromModule(test_module)
    return suite

The TestLoader class has other functions besides loadTestsFromModule. You can run dir(unittest.TestLoader) to see the other ones, but this one is the simplest to use.

Since your directory structure is such, you'll probably want the test_module to be able to import your module script. You might have already done this, but just in case you didn't, you could include the parent path so that you can import the package module and the module script. At the top of your test_module.py, type:

import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

import unittest
import package.module
...

Then finally, in setup.py, include the tests module and run the command you created, my_module_suite:

from setuptools import setup, find_packages

setup(
    ...
    test_suite = 'tests.my_module_suite',
    ...
)

Then you just run python setup.py test.

Here is a sample someone made as a reference.

Mellar

The question was how to make "python setup.py test" use unittest's discovery capability. This doesn't address that at all. – Dialyze

Ugh... yeah I completely thought the question was asking something different. I'm not sure how that happened, I must be losing my mind :( – Mellar

Answer 4

One possible solution is to simply extend the test command for distutilsand setuptools/distribute. This seems like a total kluge and way more complicated than I would prefer, but seems to correctly discover and run all the tests in my package upon running python setup.py test. I'm holding off on selecting this as the answer to my question in hopes that someone will provide a more elegant solution :)

(Inspired by https://docs.pytest.org/en/latest/goodpractices.html#integrating-with-setuptools-python-setup-py-test-pytest-runner)

Example setup.py:

try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup

def discover_and_run_tests():
    import os
    import sys
    import unittest

    # get setup.py directory
    setup_file = sys.modules['__main__'].__file__
    setup_dir = os.path.abspath(os.path.dirname(setup_file))

    # use the default shared TestLoader instance
    test_loader = unittest.defaultTestLoader

    # use the basic test runner that outputs to sys.stderr
    test_runner = unittest.TextTestRunner()

    # automatically discover all tests
    # NOTE: only works for python 2.7 and later
    test_suite = test_loader.discover(setup_dir)

    # run the test suite
    test_runner.run(test_suite)

try:
    from setuptools.command.test import test

    class DiscoverTest(test):

        def finalize_options(self):
            test.finalize_options(self)
            self.test_args = []
            self.test_suite = True

        def run_tests(self):
            discover_and_run_tests()

except ImportError:
    from distutils.core import Command

    class DiscoverTest(Command):
        user_options = []

        def initialize_options(self):
                pass

        def finalize_options(self):
            pass

        def run(self):
            discover_and_run_tests()

config = {
    'name': 'name',
    'version': 'version',
    'url': 'http://example.com',
    'cmdclass': {'test': DiscoverTest},
}

setup(**config)

Fiester

Answer 5

Another less than ideal solution slightly inspired by http://hg.python.org/unittest2/file/2b6411b9a838/unittest2/collector.py

Add a module that returns a TestSuite of discovered tests. Then configure setup to call that module.

project/
  package/
    __init__.py
    module.py
  tests/
    __init__.py
    test_module.py
  discover_tests.py
  setup.py

Here's discover_tests.py:

import os
import sys
import unittest

def additional_tests():
    setup_file = sys.modules['__main__'].__file__
    setup_dir = os.path.abspath(os.path.dirname(setup_file))
    return unittest.defaultTestLoader.discover(setup_dir)

And here's setup.py:

try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup

config = {
    'name': 'name',
    'version': 'version',
    'url': 'http://example.com',
    'test_suite': 'discover_tests',
}

setup(**config)

Fiester

Answer 6

Python's standard library unittest module supports discovery (in Python 2.7 and later, and Python 3.2 and later). If you can assume those minimum versions, then you can just add the discover command line argument to the unittest command.

Only a small tweak is needed to setup.py:

import setuptools.command.test
from setuptools import (find_packages, setup)

class TestCommand(setuptools.command.test.test):
    """ Setuptools test command explicitly using test discovery. """

    def _test_args(self):
        yield 'discover'
        for arg in super(TestCommand, self)._test_args():
            yield arg

setup(
    ...
    cmdclass={
        'test': TestCommand,
    },
)

Dialyze

BTW, I'm assuming above that you're only targeting Python versions that actual support discovery (2.7 and 3.2+), since the question is about this feature specifically. You could, of course, wrap the insert in a version-check if you wanted to remain compatible with older versions, as well (thus using setuptools' standard loader in those cases). – Dialyze

Answer 7

This won't remove run_tests.py, but will make it work with setuptools. Add:

class Loader(unittest.TestLoader):
    def loadTestsFromNames(self, names, _=None):
        return self.discover(names[0])

Then in setup.py: (I assume you're doing something like setup(**config))

config = {
    ...
    'test_loader': 'run_tests:Loader',
    'test_suite': '.', # your start_dir for discover()
}

The only downside I see is it's bending the semantics of loadTestsFromNames, but the setuptools test command is the only consumer, and calls it in a specified way.

Blythe