Is there a standard way to fail pytest if test coverage falls under x%

Question

Right now the way I'm running things is I have a set of test cases written with pytest that I run, if they fail then I fix and rework on. If they pass I use pytest-cov to get coverage and manually decide whether coverage is good enough. I was wondering if it was possible for pytest to fail if threshold for coverage is under x amount.

pytest --cov=myproject tests --cov-report=html
coverage report --fail-under=80
....
myproject/services/subnet.py                                                                        36     33     8%
myproject/severity.py                                                                                5      0   100%
--------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                             8843   8739     1%
....

Embouchure

Answer 1

You should use pytest for running tests, and failing if the tests fail. Then use coverage to assess the coverage amount, and fail if it is under:

pytest --cov=mypackage --cov-report= tests-or-whatever
coverage report --fail-under=80

Esp

Hmm, coverage report --fail-under=80 doesn't seem to do anything but spit out the sameoutput pytest had. This is what I'm running (edited) – Embouchure

"coverage report" will exit with a failure status if the total is too low. But now that you mention it, it seem obvious that it should also display a message in the output. I've written an issue: github.com/nedbat/coveragepy/issues/904 – Esp

So this one just tells me that the recipe failed. Is it possible to get a more friendly message? – Subir

@Subir for me, the console shows this message, so it's clear what the threashold and the actual coverage are: FAIL Required test coverage of 50% not reached. Total coverage: 38.10% – Milano

see --cov-fail-under=90 – Jobber

Answer 2

If you are using pytest-cov, you can use --cov-fail-under=MIN:

pytest --cov-fail-under=80 [...]

Kristy