How can I ensure good test-coverage of my big Python proejct

Question

I have a very large python project with a very large test suite. Recently we have decided to quantify the quality of our test-coverage.

I'm looking for a tool to automate the test coverage report generation. Ideally I'd like to have attractive, easy to read reports but I'd settle for less attractive reports if I could make it work quickly.

I've tried Nose, which is not good enough: It is incompatible with distribute / setuptools' namespace package feature. Unfortunately nose coverage will never work for us since we make abundant use of this feature. That's a real shame because Nose seems to work really nicely in Hudson (mostly)

As an alternative, I've heard that there's a way to do a Python coverage analysis in Eclipse, but I've not quite locked-down the perfect technique.

Any suggestions welcome!

FYI we use Python 2.4.4 on Windows XP 32bit

Fontana

I tried reproducing this problem and could not. Salim, if you want help with it, drop me a line. – Runnels

Answer 1

Have you tried using coverage.py? It underlies "nose coverage", but can be run perfectly well outside of nose if you need to.

If you run your tests with (hypothetically) python run_my_tests.py, then you can measure coverage with coverage run run_my_tests.py, then get HTML reports with coverage html.

From your description, I'm not sure what problem you had with nose, especially whether it was a nose issue, or a coverage.py issue. Provide some more details, and I'm sure we can work through them.

Runnels

Answer 2

Ned has already mentioned his excellent coverage.py module.

If the problem you're having is something nose specific, you might want to consider using another test runner. I've used py.test along with the pytest_coverage plugin that lets you generate coverage statistics. It also has a pytest_nose plugin to help you migrate.

However, I don't understand exactly what the problem you're facing is. Can you elaborate a little on the "distribute / setuptools' namespace package feature" you mentioned? I'm curious to know what the problem is.

Totipalmate

We use the namespace packages feature of setuptools which allows multiple eggs to provide stuff in a signle namespace, so for example if myproduct_foo_1.0.egg might provide the namespace myproduct and myproduct.foo, and myproduct_bar_1.0.egg would provide the namespace myproduct.bar. When I try to do a coverage analysis on myproduct the coverage tool needs to detect which of the two eggs actually contains the source of every relevant module. At the moment nose / cover.py seems to be doing it wrong. – Fontana

Salim, I would love to fix coverage.py to get this right. Do you have sample I can try myself? – Runnels

BTW: I added a coverage.py ticket for this problem: bitbucket.org/ned/coveragepy/issue/75/… – Runnels