How to test a Python CLI program with click, coverage.py, and Tox?

Question

I'm working on a CLI program using click, and I want to start adding some tests with code coverage analysis using coverage.py.

I thought a good way to implement the tests would be to run the CLI itself using subprocess. However, coverage.py reports zero code coverage, presumably because the Python instance spawned by subprocess doesn't have the coverage.py instrumentation.

I found this link that says I can drop a sitecustomize.py file in my PYTHONPATH to always force Python to start coverage measurement, but I'm using Tox to create a venv and run tests. I couldn't find any Tox settings that deal with this.

I found this answer that says I should just run my CLI through coverage run, but it looks like that only works if given a path to a Python script, and I'm trying to run my CLI through the entry point defined in setup.py. i.e. I have to change all my command lines in test code from myprogram to coverage run myprogram/cli/cli.py. I'd rather not do this because it's not the way I expect users to run the program.

So it seems like the two options are:

1. figure out a way to make sitecustomize.py work in Tox environments, or

2. all command lines in test code to use a script path instead of an entry point (probably easier in the long run, but makes tests slightly more brittle and harder to understand). Wondering if there's anything else I'm missing.

Undrape

FWIW when I've written similar programs, I've always written tests where my program runs in the same process as the test code. The tests just set up os.environ and sys.argv as expected and then call myprogram.main() directly. This approach works well with tox, coverage, and other tools I've used. – Molybdenous

That's a great point, I didn't think of changing sys.argv. You can add that as an answer if you want. This answer says you should mock sys.argv with unittest.mock, but that feels like overkill without knowing some specific reasons why... – Undrape

Also, looks like click.testing.CliRunner is supposed to do the right thing to invoke the CLI in unit tests. I didn't find that because I was looking for something that looked like argparse.ArgumentParser:parse_args(). – Undrape

Answer 1

CliRunner and a proper unit test framework is the way to go. Here is an example setup for PyTest which uses the PyTest Coverage plugin

test_click.py

from click.testing import CliRunner

from click_prog import hello


def test_hello_world():

    runner = CliRunner()
    result = runner.invoke(hello, ['--opt', 'An Option', 'An Arg'])
    assert result.exit_code == 0
    assert result.output == 'Opt: An Option  Arg: An Arg\n'

    result = runner.invoke(hello, ['An Arg'])
    assert result.exit_code == 0
    assert result.output == 'Opt: None  Arg: An Arg\n'


if __name__ == '__main__':
    test_hello_world()

click_prog.py

import click
import sys


@click.command()
@click.option('--opt')
@click.argument('arg')
def hello(arg, opt):
    """A Simple program"""
    click.echo('Opt: {}  Arg: {}'.format(opt, arg))


if __name__ == '__main__':
    hello(sys.argv[1:])

pytest.ini

[pytest]

# -- recommended but optional:
# python_files = tests.py test_*.py *_tests.py

Test Results:

===================== test session starts =====================
platform darwin -- Python 3.6.5, pytest-3.7.1, py-1.5.4, pluggy-0.7.1
rootdir: /Users/strauch/dev/fix_windows, inifile: pytest.ini
plugins: xdist-1.22.5, forked-0.2, cov-2.6.0
collected 1 item

test_click.py .                                         [100%]

---------- coverage: platform darwin, python 3.6.5-final-0 -----------
Name             Stmts   Miss  Cover
------------------------------------
click_prog.py        8      1    88%
test_click.py       12      1    92%
------------------------------------
TOTAL               20      2    90%


================== 1 passed in 0.07 seconds ===================

Mogilev