condition coverage in python

Question

Is there any tool/library that calculate percent of "condition/decision coverage" of python code. I found only coverage.py but it calculates only percent of "statement coverage".

Mook

Answer 1

Coverage.py now includes branch coverage.

For the curious: the code is not modified before running. The trace function tracks which lines follow which in the execution, and compare that information with static analysis of the compiled byte code to find path possibilities not executed.

Freeman

Coverage.py doesn't support condition coverage though: github.com/nedbat/coveragepy/issues/660 – Cressida

Answer 2

I don't know of any branch coverage tools for Python, though I've contemplated writing one. My thought was to start with the AST and insert additional instrumentation for each branch point. It's doable, but there are some tricky cases.

For example,

raise SomeException(x)

Branch coverage for this needs to check that SomeException(x) was fully instantiated and didn't raise its own exception.

assert x, "Oh No!: %r" % (x, y)

This needs to check that the text on the right side of the assertion statement is fully evaluated.

return args.name or os.getenv("NAME") or die("no name present")

Each of the first two terms has to be checked for the true/false path, but not the last. In fact, the last might not even return.

There were a lot of cases to worry about and I had no pressing need for it other than curiosity so I didn't go anywhere with it. I was also wondering if I would get a lot of false positives where I would need some way to repress specific warnings.

If you want to try this route, start with Python 2.6 or 3.0. In those releases the AST module is documented and you can create your own AST nodes before generating the code or .pyc file.

Astronavigation

"or die" example is quite fun to read, and quite possibly was fun to write ;) – Craal

I was a Perl programmer before I switched to Python. That's a Perl-ism. Perl was never known as a boring language. – Astronavigation

Answer 3

I haven't used it myself, but if you are willing to replace coverage analysis with mutation testing, I've heard of a mutation tester called "pester".

While I was doing googling, I also came across a list of python testing tools which mentions some possible code coverage tools.

Garganey

The link to testing tools doesn't work anymore. For mutation testing check out mutmut: github.com/boxed/mutmut – Adjunct

the other alternative for the mutation testing is cosmic-ray: github.com/sixty-north/cosmic-ray – Cressida

Answer 4

Are you looking for cyclomatic complexity (Wikipedia)? It basically computes the number of paths through a piece of code. There are some projects to compute that for Python code, for example PyMetrics or this one. Google will certainly bring up more.

But I don't know of any further integration with unit tests that will show you the coverage.

Campagna

> Are you looking for cyclomatic complexity (Wikipedia)? No. Thanks for interesting information. > But I don't know of any further integration with unit tests that will show you the coverage. Me too ;) – Mook

cyclomatic complexity is basically unrelated to condition coverage. – Abradant

Answer 5

The very same maintainer of coverage.py has an article discussing a way to get coverage information at the bytecode level. The method is a bit kludgey: it involves re-assembling .pyc files with tweaked line numbers. However, it provides about as much granularity in coverage measurement as you could ask for.

Rack

Answer 6

It looks like "instrumental" implements condition coverage:

Instrumental on Pypi

Link about coverage.py and instrumental

Has anyone tried it? It has a tiny version number. I need something I can trust.

Vitelline

instrumental is providing what the original question asked for, but the project is essentially abandoned. See comments in the issue tracker: bitbucket.org/desmaj/instrumental/issues/22/… – Cressida

Answer 7

Parse and modify the AST is the right answer, IMHO. See this paper for a complete description of what you need to do: "Branch Coverage Made Easy for Arbitrary Languages"

http://www.semanticdesigns.com/Company/Publications/TestCoverage.pdf

Abradant