preventing python coverage from including virtual environment site packages

Question

I am new to coverage and ran into a strange problem. My coverage is taking my virtual environment site packages into account. Here is the output of the coverage run:

coverage run test.py
....................
----------------------------------------------------------------------
Ran 20 tests in 0.060s

OK
(atcatalog)- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -jmfrank63@fullstack-audio-text-catalog:~/workspace (git master)       [19:58:45]
$ coverage report
Name                                                                              Stmts   Miss  Cover
-----------------------------------------------------------------------------------------------------
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/__init__               18      0   100%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/_compat                38     20    47%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/app                   528    255    52%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/blueprints            156    118    24%
                             .
                             .
                             .
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/urls               412    215    48%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/utils              242    175    28%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/wrappers           568    298    48%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/wsgi               448    352    21%
atcatalog/__init__                                                                    7      0   100%
atcatalog/views/__init__                                                              0      0   100%
atcatalog/views/publang                                                               7      0   100%
atcatalog/views/pubtext                                                               1      0   100%
atcatalog/views/userlang                                                             13      0   100%
atcatalog/views/users                                                                 5      0   100%
atcatalog/views/usertext                                                             14      0   100%
test                                                                                120      0   100%
-----------------------------------------------------------------------------------------------------
TOTAL                                                                             12530   8044    36%
(atcatalog)- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -jmfrank63@fullstack-audio-text-catalog:~/workspace (git master)       [19:58:55]

Here is the structure of my project directory which resides under home:

workspace/
├── README.md
├── atcatalog
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── static
│   ├── templates
│   └── views
│       ├── __init__.py
│       ├── __init__.pyc
│       ├── publang.py
│       ├── publang.pyc
│       ├── pubtext.py
│       ├── pubtext.pyc
│       ├── userlang.py
│       ├── userlang.pyc
│       ├── users.py
│       ├── users.pyc
│       ├── usertext.py
│       └── usertext.pyc
├── requirements.txt
├── run.py
└── test.py

I had the virtual environment at first inside the project directory and now moved it out to ~/Envs with virtualenvwrapper, but the problem persisted. run.py and test.py are not special in any way, they both import app from atcatalog. I also tried to find ways to omit the virtual environment directory, but google gave no answer (surprisingly). I don't think it is the purpose of coverage to test already well tested site-packages. So I would exclude them from the run.

How can I accomplish to avoid coverage having testing my site-packages?

Oribella

I would try using py.test with coverage. – Pulmonary

Of course it is an option to switch to a different testing framework. But that doesn't explain what is going wrong here. None of the tutorials showed this behavior. And there are no records of other people having this problem. I am probably doing something stupid wrong. – Oribella

When you moved the virtual env, did you delete the old one? What is the result of ls -a in workspace/ – Pulmonary

Yes the old is deleted. I also got rid of the .coverage file. Also the output shows coverage is using my new virtual environment which is clearly outside my project directory. – Oribella

Have you given this a read? coverage.readthedocs.org/en/coverage-4.0b1/source.html#source – Pulmonary

Yes, that what puzzles me. From the docs: When running your code, the coverage run command will by default measure all code, unless it is part of the Python standard library. – Oribella

Strange, flask and werkzeug are definitely in the psl. Did you install with pip? – Pulmonary

Ok, your comments made me solve the problem. Flask and Werkzeug are not part of the standard library. So I tried coverage run --source /home/ubuntu/workspace test.py and the problem is gone. It might be ok for professionals, but I think anything under site-packages should not be tested on a standard basis. – Oribella

Ah, that's really interesting. Yea I would agree with you there. Glad it's fixed! – Pulmonary

Answer 1

Thanks to tknickman I figured it out: Use either

coverage run --source <path to project dir> test.py

or create a configuration file .coveragerc which resides in the directory you run coverage from, with the following content:

[run]
source =
    <path to project dir>

This provides you do not have your virtual environment installed under the project directory. If you have the virtual environment installed under the project dir you can use

coverage run --source <project path> --omit <pattern> test.py

Note that omit wants a file pattern like

~/projectdir/venv/*

instead of a path.

The corresponding .coveragerc would look like this:

[run]
source=
    <path to project dir>
omit=
    <path to project dir>/<name of virtual env>/*

I still think that like packages of the standard library any packages installed under site-packages should not be covered by default.

Oribella

Hello @JohannesMariaFrank, I have my folder with my python files and inside is venv - was trying to run it, but I do not know what to put as <project path> or <path to project dir> cause I am running it inside that folder where are the files - so nothing works for me, and if I create the .coveragerc file - still I would not know what to put as <path to project dir> - should I put "" - like empty string or? thank you – Mantoman

Hi @Mantoman An absolute path should be perfectly fine, in fact ~/projectdir/venv is an absolute path as ~ resolves to /home/radek or C:\Users\radek. – Oribella

I have did @JohannesMariaFrank as you said and got this error: Can't find '_ main _' module in '/[path-to-my-directory]/venv/include' – Mantoman

Hi @Mantoman sounds like #24724047 – Oribella

btw, if your .coveragerc file is in the same location, then do source=. omit=./venv/* in your .coveragerc file – Nevlin

Answer 2

In your setup.cfg file include:

[coverage:run]
omit=*/site-packages/*,*/tests/*,*/.eggs/*

Or any other folders that show up in your results that you want to hide from coverage.

Astrix

Answer 3

Try using py.test and then specifiying your test options in a setup.cfg file. You will need to pip install pytest first.

For example:

[pytest]
norecursedirs = build docs/_build *.egg .tox *.venv
python_files = tests/functional* tests/integration*
addopts =
    #--verbose
    --tb short
    # Turn on --capture to have brief, less noisy output
    # You will only see output if the test fails
    # Use --capture no if you want to see it all or have problems debugging
    --capture fd
    # --capture no
    # show extra test summary info as specified by chars (f)ailed, (E)error,      (s)skipped, (x)failed, (X)passed.
    - rfEsxX
    --junitxml junit.xml
    --cov workspace --cov-report xml --cov-report term-missing

You can read more about configuring py.test here: https://pytest.org/latest/customize.html

Pulmonary

This answer doesn't have anything to do with the original question which was about coverage.py and not py.test. – Warring

Answer 4

If using pytest, you can specify exclusive paths or files to test in setup.cfg (see docs):

[pytest]
# a directory
testpaths = tests

# exact file(s)
python_files = tests/test1.py tests/test2.py

It looks like if you include the python_files and testpaths parameters, then the python_files will only be used.

Rah

testpaths = tests - is tests a folder? – Mantoman