python unittest with coverage report on (sub)processes

Question

I'm using nose to run my "unittest" tests and have nose-cov to include coverage reports. These all work fine, but part of my tests require running some code as a multiprocessing.Process. The nose-cov docs state that it can do multiprocessing, but I'm not sure how to get that to work.

I'm just running tests by running nosetests and using the following .coveragerc:

[run]
branch = True
parallel = True

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
    # Have to re-enable the standard pragma
    pragma: no cover

    # Don't complain about missing debug-only code:
    def __repr__
    #if self\.debug

    # Don't complain if tests don't hit defensive assertion code:
    raise AssertionError
    raise NotImplementedError

    # Don't complain if non-runnable code isn't run:
    if 0:
    if __name__ == .__main__.:
    def __main__\(\):

omit =
    mainserver/tests/*

EDIT:

I fixed the parallel switch in my ".coveragerc" file. I've also tried adding a sitecustomize.py like so in my site-packages directory:

import os
import coverage
os.environ['COVERAGE_PROCESS_START']='/sites/metrics_dev/.coveragerc'
coverage.process_startup()

I'm pretty sure it's still not working properly, though, because the "missing" report still shows lines that I know are running (they output to the console). I've also tried adding the environment variable in my test case file and also in the shell before running the test cases. I also tried explicitly calling the same things in the function that's called by multiprocessing.Process to start the new process.

Strage

Answer 1

First, the configuration setting you need is parallel, not parallel-mode. Second, you probably need to follow the directions in the Measuring Subprocesses section of the coverage.py docs.

Myrwyn

The command switch for coverage is --parallel-mode so I assumed that's what it was in the config. Thanks for pointing out that page, I'll see if I can get that working. And thanks for developing such a great testing tool! – Strage

please see my edit above... I still can't seem to get it to work. – Strage

Answer 2

Another thing to consider is if you see more than one coverage file while running coverage. Maybe it's only a matter of combining them afterwards.

Bengurion

Unfortunately, I think I only ever got one file. – Strage

Could you pass full command line, I am trying to crack similar problem. However mine seems to be even more complex. If I find anything useful will post it here. – Bengurion

Sorry, I don't remember... It was probably nosetests --with-coverage – Strage

Answer 3

tl;dr — to use coverage + nosetests + nose’s --processes option, set coverage’s --concurrency option to multiprocessing, preferably in either .coveragerc or setup.cfg rather than on the command-line (see also: command line usage and configuration files).

---

Long version…

I also fought this for a while, having followed the documentation on Configuring Python for sub-process coverage to the letter. Finally, upon re-examining the output of coverage run --help a bit more closely, I stumbled across the --concurrency=multiprocessing option, which I’d never used before, and which seems to be the missing link. (In hindsight, this makes sense: nose’s --processing option uses the multiprocessing library under the hood.)

Here is a minimal configuration that works as expected:

---

unit.py:

def is_even(x):
    if x % 2 == 0:
        return True
    else:
        return False

---

test.py:

import time
from unittest import TestCase

import unit


class TestIsEvenTrue(TestCase):
    def test_is_even_true(self):
        time.sleep(1)  # verify multiprocessing is being used
        self.assertTrue(unit.is_even(2))


# use a separate class to encourage nose to use a separate process for this
class TestIsEvenFalse(TestCase):
    def test_is_even_false(self):
        time.sleep(1)
        self.assertFalse(unit.is_even(1))

---

setup.cfg:

[nosetests]
processes = 2
verbosity = 2

[coverage:run]
branch = True
concurrency = multiprocessing
parallel = True
source = unit

---

sitecustomize.py (note: located in site-packages)

import os
try:
    import coverage
    os.environ['COVERAGE_PROCESS_START'] = 'setup.cfg'
    coverage.process_startup()
except ImportError:
    pass

---

$ coverage run $(command -v nosetests)
test_is_even_false (test.TestIsEvenFalse) ... ok
test_is_even_true (test.TestIsEvenTrue) ... ok

----------------------------------------------------------------------
Ran 2 tests in 1.085s

OK

$ coverage combine && coverage report
Name      Stmts   Miss Branch BrPart  Cover
-------------------------------------------
unit.py       4      0      2      0   100%

Sherilyn