Django test coverage vs code coverage

Question

I've successfully installed and configured django-nose with coverage

Problem is that if I just run coverage for ./manage.py shell and exit out of that shell - it shows me 37% code coverage. I fully understand that executed code doesn't mean tested code. My only question is -- what now?

What I'm envisioning is being able to import all the python modules and "settle down" before executing any tests, and directly communicating with coverage saying "Ok, start counting reached code here."

Ideally this would be done by nose essentially resetting the "touched" lines of code right before executing each test suite.

I don't know where to start looking/developing. I've searched online and haven't found anything fruitful. Any help/guidelines would be greatly appreciated.

P.S.

I tried executing something like this:

DJANGO_SETTINGS_MODULE=app.settings_dev coverage run app/tests/gme_test.py

And it worked (showed 1% coverage) but I can't figure out how to do this for the entire app

Edit: Here's my coverage config:

[run]
source = .
branch = False
timid = True
[report]
show_missing = False
include = *.py
omit =
    tests.py
    *_test.py
    *_tests.py
    */site-packages/*
    */migrations/*
[html]
title = Code Coverage
directory = local_coverage_report

Vernonvernor

How about DJANGO_SETTINGS_MODULE=app.settings_dev coverage ./manage.py test app1 app2 ... appn? – Scarbrough

This wouldn't help. Read my 2nd sentence - as soon as manage.py is involved I get a faulty coverage. – Vernonvernor

"resetting the touched lines of code right before executing each test suite" is easy to do. Do you want help on that? -- Just make a new coverage instance for every test suite (you do that in your test runner specialization). – Scarbrough

Answer 1

since you use django-nose you have two options on how to run coverage. The first was already pointed out by DaveB:

coverage run ./manage.py test myapp

The above actually runs coverage which then monitors all code executed by the test command.

But then, there is also a nose coverage plugin included by default in the django-nose package (http://nose.readthedocs.org/en/latest/plugins/cover.html). You can use it like this:

./manage.py test myapp --with-coverage

(There are also some additional options like which modules should be covered, whether to include an html report or not etc . These are all documented in the above link - you can also type ./manage.py test --help for some quick info).

Running the nose coverage plugin will result in coverage running after the django bootstrapping code is executed and therefore the corresponding code will not be reported as covered.

Most of the code you see reported as covered when running coverage the original way, are import statements, class definitions, class members etc. As python evaluates them during import time, coverage will naturally mark them as covered. However, running the nose plugin will not report bootstrapping code as covered since the test runner starts after the django environment is loaded. Of course, a side effect of this is you can never achieve 100% coverage (...or close :)) as your global scope statements will never get covered.

After switching back and forth and playing around with coverage options, I now have ended up using coverage like this:

coverage run --source=myapp,anotherapp --omit=*/migrations/* ./manage.py test

so that

a. coverage will report import statements, class member definitions etc as covered (which is actually the truth - this code was successfully imported and interpreted)

b. it will only cover my code and not django code, or any other third-party app I use; The coverage percentage will reflect how well my project is covered.

Hope this helps!

Archi

I've updated the OP with my config. Do you think successful import means that the code is tested? To me the tested part is just 1 line, not every line in the imported file. – Vernonvernor

As far as there is no logic on these lines I would say yes, the code is covered. It's possibly a matter of perception, one could argue the opposite I guess. You still have the option of using the nose plugin, it is really convenient. – Archi

running manage.py --with-coverage actually seems to report a higher code coverage than running coverage run.... I'm still playing around with various flags, most interesting one is --cover-inclusive which seems to have a reverse behavior from what I expected. – Vernonvernor

Do you mean that --cover-inclusive actually results in a higher coverage? Are you using the --cover-package option as well? For the same set of packages --cover-inclusive should theoretically result in less (equal at most) coverage. That's strange. – Archi

Answer 2

The "Ok, start counting reached code here." can be done through the API of the coverage module. You can check this out through the shell. Stole directly from http://nedbatchelder.com/code/coverage/api.html:

import coverage

cov = coverage.coverage()
cov.start()

# .. call your code ..

cov.stop()
cov.save()

cov.html_report()

You can make your own test-runner to do this exactly to match your needs (some would consider coverage made from any unit-test to be OK, and others would only accept the coverage of a unit caused by the unit-test for that unit.)

Clarkin

Does this mean that I will have to write my own coverage runner? At the moment if the part where you say "... call your code..." includes an import statement, then that code would also be considered covered. – Vernonvernor

It is up to you if you find the existing runners sufficient or not. I am just suggesting that if you have special needs, you may need to write your own. coverage.py tracks code executed, not code parsed, i.e. code will not be tracked as executed just because it is imported. It is, however, trivial to make code that is executed when imported. – Scarbrough

Robert, the entire point of my post is that I'm aware of the issue of coverage tracking the code executed. I'm pointing out that not all executed code is tested code. – Vernonvernor

Then what of the code covered is considered tested in your point of view? – Scarbrough

Tested means that I write code that executes code in question, and then confirms that the execution results in some expectation. It has to include some assertion. If I just execute some code, it doesn't mean I tested it. Executing code: a = sum(2,2). Testing code: assert(a == 4). – Vernonvernor

Unit tests will often be on the form a = sum([2, 2]) self.assertEqual(a, 4). But it should be easy enough to override the assertion methods on your own custom TestCase specialization to do coverage recording. I am sure these can be combined through the API or by using the same coverage meta file. – Scarbrough

Answer 3

I had the same issue. I saved some time by creating a .coveragerc file that specified options similar to those outlined in the bounty-awarded answer.

Now running 'coverage run manage.py test' and then 'coverage report -m' will show me the coverage report and the lines that aren't covered.

(See here for details on the .coveragerc file: http://nedbatchelder.com/code/coverage/config.html)

Lyns

Answer 4

I'm a bit confused by what you are trying to achieve here.

Testing in Django is covered very well here: https://docs.djangoproject.com/en/dev/topics/testing/overview/

You write tests in your app as test.py - I don't see the need for nose, as the standard django way is pretty simple.

Then run them as coverage run ./manage.py test main - where 'main' is your app

Specify the source files for your code as documented here: http://nedbatchelder.com/code/coverage/cmd.html so that only your code is counted

e.g. coverage run --source=main ./manage.py test main

You'll still get a certain percentage marked as covered with the simple test provided as an example. This is because parts of your code are executed in order to start up the server e.g definitions in modules etc.

Demoralize

Nose is superbly faster and more flexible. And really it's not the core question. Your last sentence is exactly why I'm opening a bounty. "You'll still get a certain percentage marked as covered" -- that's a huge issue! As I pointed out this "certain percentage" is 37% in my case, of code covered, but not tested. I'm not ok with this. – Vernonvernor

Coverage tools are never going to be able to guarantee that any of your code is actually properly tested. This can only be done by carefully crafting your tests with an understanding of all parts of your system. Coverage merely helps to highlight holes in your testing. – Demoralize

how is everyone ok with this? It's a really crappy "help to highlight holes" because it doesn't highlight all the holes! It's like saying "oh this is part of computer science that doesn't work" – Vernonvernor

This is pretty old, but worth noting that we're all OK with this because a coverage tool can't tell if you've tested your code (just that lines of it ran). If it could tell you everything is tested fine, it could write the code for you as well... Ultimately you write characters which tell a processor what to do, you need to do the same for the tests. – Fruitless