I am running simple nose tests on a Django website (code available here). Depending on the order that the CLI commands are issued, I get vastly different results!
Looking at many posts, it seems related to when coverage started with respect to the load process. What I don't understand is:
- Why are the missed statements so different between the two methods?
- Why does the coverage go way up if I run one method after the other?
I am focusing on the coverage of minerals/views.py
The Django settings.py file contains the configuration info:
# Use nose to run all tests
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
# Nose will measure coverage on the folllowing apps
NOSE_ARGS = [
'--with-coverage',
'--cover-package=minerals',
'--cover-html',
]
I am running on Ubuntu 14.04LTS, with Python 3.5.1, using a virtualenv requirements file:
coverage==4.1
Django==1.9.6
django-debug-toolbar==1.4
django-nose==1.4.3
nose==1.3.7
sqlparse==0.1.19
Running test method 1:
When I run the tests using coverage run --source '.' manage.py test minerals, then minerals/views.py is missing statements: 12-13, 18-49.
$ coverage run --source '.' manage.py test minerals
nosetests minerals --with-coverage --cover-package=minerals --cover-html --verbosity=1
Creating test database for alias 'default'...
..
Name Stmts Miss Cover
--------------------------------------------------------------------
[...]
minerals/views.py 18 8 56%
--------------------------------------------------------------------
TOTAL 73 48 34%
----------------------------------------------------------------------
Ran 2 tests in 0.809s
Note: Yes, for some unknown reason, the number of missed statements between the run above and the report below changes.
The full coverage report -m (from the file report_source):
Name Stmts Miss Cover Missing
---------------------------------------------------------------------------
manage.py 6 0 100%
mineral_catalog/__init__.py 0 0 100%
mineral_catalog/settings.py 21 0 100%
mineral_catalog/urls.py 6 0 100%
mineral_catalog/views.py 4 1 75% 6
mineral_catalog/wsgi.py 4 4 0% 10-16
minerals/__init__.py 0 0 100%
minerals/admin.py 3 0 100%
minerals/apps.py 3 3 0% 1-5
minerals/migrations/0001_initial.py 6 6 0% 3-15
minerals/migrations/0002_auto_20160608_1153.py 5 5 0% 3-14
minerals/migrations/__init__.py 0 0 100%
minerals/models.py 24 1 96% 30
minerals/templatetags/__init__.py 0 0 100%
minerals/templatetags/mineral_extras.py 12 6 50% 13-16, 22-23
minerals/tests.py 20 20 0% 1-31
minerals/urls.py 5 0 100%
minerals/views.py 18 11 39% 12-13, 18-49
------------------------------------------------------------------------------
TOTAL 137 57 58%
Running test method 2:
If I run the tests using ./manage.py test minerals, then minerals/views.py is missing statements 1-10, 16, 42.
$ ./manage.py test minerals
nosetests minerals --with-coverage --cover-package=minerals --cover-html --verbosity=1
Creating test database for alias 'default'...
..
Name Stmts Miss Cover
--------------------------------------------------------------------
[...]
minerals/views.py 18 8 56%
--------------------------------------------------------------------
TOTAL 73 48 34%
----------------------------------------------------------------------
Ran 2 tests in 0.818s
The full coverage report -m (from the file report_manage):
Name Stmts Miss Cover Missing
------------------------------------------------------------------------------
minerals/__init__.py 0 0 100%
minerals/admin.py 3 3 0% 1-6
minerals/apps.py 3 3 0% 1-5
minerals/migrations/0001_initial.py 6 0 100%
minerals/migrations/0002_auto_20160608_1153.py 5 0 100%
minerals/migrations/__init__.py 0 0 100%
minerals/models.py 24 24 0% 1-30
minerals/templatetags/__init__.py 0 0 100%
minerals/templatetags/mineral_extras.py 12 8 33% 1-10, 19-23
minerals/tests.py 20 0 100%
minerals/urls.py 5 5 0% 1-10
minerals/views.py 18 8 56% 1-10, 16, 42
------------------------------------------------------------------------------
TOTAL 96 51 47%
Running test method 3:
If I remove the cover/ directory and the .coverage file between test runs, Methods 1 and 2 above produce the same results each time as to be expected. However, if I run Method 1, then run Method 2 (leaving the cover/ dir and .coverage in place from Method 1), I get much different results. minerals/views.py is only missing statement 42.
$ ./manage.py test minerals
nosetests minerals --with-coverage --cover-package=minerals --cover-html --verbosity=1
Creating test database for alias 'default'...
..
Name Stmts Miss Cover
--------------------------------------------------------------------
[...]
minerals/views.py 18 1 94%
--------------------------------------------------------------------
TOTAL 73 4 95%
----------------------------------------------------------------------
Ran 2 tests in 0.788s
The full coverage report -m (from file report_sourceTHENmanage):
Name Stmts Miss Cover Missing
------------------------------------------------------------------------------
manage.py 6 0 100%
mineral_catalog/__init__.py 0 0 100%
mineral_catalog/settings.py 21 0 100%
mineral_catalog/urls.py 6 0 100%
mineral_catalog/views.py 4 1 75% 6
mineral_catalog/wsgi.py 4 4 0% 10-16
minerals/__init__.py 0 0 100%
minerals/admin.py 3 0 100%
minerals/apps.py 3 3 0% 1-5
minerals/migrations/0001_initial.py 6 0 100%
minerals/migrations/0002_auto_20160608_1153.py 5 0 100%
minerals/migrations/__init__.py 0 0 100%
minerals/models.py 24 1 96% 30
minerals/templatetags/__init__.py 0 0 100%
minerals/templatetags/mineral_extras.py 12 2 83% 22-23
minerals/tests.py 20 0 100%
minerals/urls.py 5 0 100%
minerals/views.py 18 1 94% 42
------------------------------------------------------------------------------
TOTAL 137 12 91%
I'm not sure how to proceed from this point. I would like to be able to get consistent results using a single CLI comand.
Please let me know if there is more information I need to provide to help debug this!
Update:
Experimenting with py.test, I was able to get a single line coverage using:
$ coverage run --source minerals -m py.test minerals/tests.py
============================================ test session starts =============================================
platform linux -- Python 3.5.1, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
django settings: mineral_catalog.settings (from ini file)
rootdir: [...]/mineral_catalog, inifile: pytest.ini
plugins: django-2.9.1
collected 2 items
minerals/tests.py ..
========================================= 2 passed in 10.88 seconds ==========================================
$ coverage report -m
Name Stmts Miss Cover Missing
------------------------------------------------------------------------------
minerals/__init__.py 0 0 100%
minerals/admin.py 3 0 100%
minerals/apps.py 3 3 0% 1-5
minerals/migrations/0001_initial.py 6 0 100%
minerals/migrations/0002_auto_20160608_1153.py 5 0 100%
minerals/migrations/__init__.py 0 0 100%
minerals/models.py 24 1 96% 30
minerals/templatetags/__init__.py 0 0 100%
minerals/templatetags/mineral_extras.py 12 2 83% 22-23
minerals/tests.py 20 0 100%
minerals/urls.py 5 0 100%
minerals/views.py 18 1 94% 42
TOTAL 96 7 93%