How to get combined code coverage over multiple runs of Python script

Question

I've got a python program which is tested by running it several times with different inputs, and comparing the outputs against reference results.

I'd like to get code coverage of all the tests combined, so I can see if there are extra sets of inputs I should be using to get complete coverage. I've looked at the coverage module but can't work out how I can make it do this.

Any clues?

Biostatics

Answer 1

If running on the same machine, run it with the -a option, which accumulates coverage data across multiple calls.

Example:

coverage erase

coverage run -a <command> [arguments, ...]

coverage run -a <command> ... # repeat as many times as needed.

coverage report

coverage html

doc: http://coverage.readthedocs.org/en/latest/cmd.html#data-file

Hope this helps.

Trave

see also coverage combine – Scintillometer

Answer 2

Ned Batchelder's coverage.py has a feature to combine the results of multiple runs, which seems to be exactly what you are looking for.

Christo

Yes, this is exactly right. You can use this to run across multiple data sets, across multiple versions of Python, or multiple dependency choices (e.g. different versions of middleware or support modules). You can "orchestrate" multiple test runs in a shell script, but you can do that one better: The tox test runner will auto-create pristine virtual environments and run tests in those; highly recommended for anything beyond the simplest testing setup. – Incorporation

@opa The OP selected this answer because it apparently helped them. The other answer was added two years later. I'm a bit puzzled why this got you so angry – I'm happy to improve my answers based on friendly suggestions, but when being shouted at I don't really feel like volunteering more time to help people. – Christo

@opa I did not complain about your issue with this answer. You do have a point, and I never questioned that. I was rather complaining about the tone of your comment, which is unnecessarily rude. It would have been sufficient to say "You should include the gist of the linked page in this answer, since otherwise it will become useless if that page goes away" or similar. Instead you chose to say that the answer is "godawful", that it is unfortunate that the OP selected it etc, which sounds rather hostile. I will improve the answer when I find some time to look into this. – Christo

@SvenMarnach You're right, my hostility wasn't productive. I get particularly frustrated when I see experienced users have made the same mistakes that lead to new users producing poor quality content. – Maltreat