running nose --with-coverage to get all the package files, but not other dependencies and libs

Question

My project folder(yeah - I know it's best practice) is something like:

.
├── app.py
├── otherscript.py
├── tests/
└── tools/
    ├── __init__.py
    └── toolfile.py

I need nose --with-coverage to test the .py scripts in the main folder, tools folder and exclude the tests folder (although I don't really care about excluding that)

When I run basic

nose --with-coverage

I get coverage on all installed dependencies and libs (flask, requests, etc)

when I run

nose --with-coverage --cover-package=folder name (or . or ./)

I get coverage for the tests folder. the tools/__init__.py file and app.py but not for the rest of the scripts:

> (opentaba-server) D:\username\pathto\opentaba-server>nosetests --with-coverage -- cover-package=./ ... Name                                      

> Stmts   Miss  Cover   Missing
> ----------------------------------------------------------------------- Tests\functional_tests\test_return_json      26      0   100%
> Tests\unit_test\test_createdb                 0      0   100%
> Tests\unit_test\test_scrape                   0      0   100% app     
> 63     15    76%   22, 24, 72-81, 8 8-106, 133 tools\__init__         
> 0      0   100%
> ----------------------------------------------------------------------- TOTAL                                        89     15    83%
> ---------------------------------------------------------------------- Ran 3 tests in 5.182s OK

When I run with the --cover-inclusive flag . It just fails with :

nosetests-scripts.py: error: no such option: --with-coverage

I'll be glad for any help with this

Imes

So it's saying that "--with-coverage" is not a valid option, but only when you ALSO use the option "--cover-inclusive" ? – Leveille

@smlstvnh strangely so. – Imes

this is actually the case with --with-xunit as well. I spent about 1 hour trying to get the --xunit-file option to work without specifying the --with-xunit option. Pretty big waste of time... – Knipe

Nose's version of coverage is supposed to ignore modules in the Python library directory. – Roorback

Might be too late, but the last error message looks like you are using an outdated version of nose. Most coverage options are quite recent. – Maurizia

Answer 1

I had a very similar problem with generated code. The solution was to exclude the generated code or tools code in your case only from the reports.

So we now use nosetests to run our tests like

nosetests --with-coverage --cover-inclusive --cover-package=$(PACKAGE)

and afterwards, we create the reports manually, so

coverage combine
coverage report --omit 'tools/*'

Thus, coverage.py will cover your tools package, but they won't show up in the reports.

Acker

Answer 2

my tests/nose_setup_coverage.cfg file:

[nosetests]
verbosity=1
detailed-errors=1

with-coverage=1
cover-html=1
cover-html-dir=../../out/cover

#this is the line that fixed my equivalent of your issue
#by "climbing up" from tests/ but skipping python's **site-packages**
cover-package=..   

where=/Users/jluc/kds2/py/tests

adding cover-package=.. (in the cfg file) and executing from within the tests directory got me to cover all my python directories, but not django and other 3rd party stuff.

This is my directory structure (minus some non-Python stuff):

.
├── lib
├── non_app
├── ps_catalog
├── pssecurity
├── pssystem
├── static
├── static_src
├── staticfiles
├── templates
├── tests
└── websec

last, though it doesn't seem documented, coverage, as run from nosetests, will pick and use a .coveragerc file in the current (test) directory (you can't pass it via commandline or in the nose cfg file, that's for the cover plugin).

In that file, the omit section allows you finer control on which directories to exclude:

omit=/Users/jluc/kds2/env/lib/python2.7/*
     */batch/*
     /Users/jluc/kds2/py/non_app/*
     */migrations/*

executing the tests in bash:

nosetests -c nose_setup_coverage.cfg

p.s. adding --cover-erase to the above, resets coverage

edit, FWIW:

I've generally replaced nose/nose2 with pytest as a test runner. If you don't bring in advanced pytest functionality, it will work just fine with plain old unittests and has a lot more plugins, support and active development. This is not to diss nose, or to promote pytest, but just saying that's an alternative to consider if you are repeatedly hitting edge cases with nose.

Pyrrolidine

you save me some shell commands with cover-html=1 (i added also cover-xml=1). thanks – Slaughterhouse

Answer 3

By default tests will not be included in the coverage report. You can make them show up (actually a very good idea to make sure your tests are properly executed, and no duplicate name tests are being ignored) with --cover-tests

In any case, nosetests --help is your friend. Most likely --cover-inclusive flag kills off coverage plugin and other options (for the plugin) become unavailable. You can try to debug it by launching nose through pdb.

As alternative you can run coverage as a standalone module launching nose tests.

Dkl