How to count test cases written with pytest?

Question

My objective is to get the number of test methods in a package/folder. I'm able to do that by executing

py.test <folder> --collect-only|grep collected

This shows the test count as

collected 104 items

However this counts the parameterized test multiple times,e.g. if a method has two sets of parameter single test will be counted 2 times. Is there any way to tell pytest to count them as single?

Aqualung

Answer 1

How about

find . -type f -name 'test*.py' -exec grep -e 'def test_' '{}' \; | wc -l

or

ag 'def test_' | wc -l

Justiciar

It won't take into account the parametrized tests unfortunately – Geary

@Geary that's exactly what I needed, not to count the parameterised test as multiple tests. So the answer is exactly what I asked for. – Aqualung

Answer 2

If your tests in pytest use custom test collection method or use parametrization, regular grepping won't be helpful. Regular grepping can only give you test function and test classes. If you are interested in that number, other answers here should work fine for you. If you want to know the total number of collected tests, follow along.

You should run pytest --collect-only for target test directory and run grep on the output.

One possible solution is this:

pytest --collect-only | grep "<Function\|<Class" -c

It will return the number of lines which has <Function or <Class in the output of pytest --collect-only. Since all collected tests have either of these words, it will give correct number of tests.

Another hacky way to find the number of tests is to use -k switch. It searches for expression and run the tests which match the expression.

pytest -k "not test and not Test"

This will give you number of all tests. What it does is collect all the tests and tries to find the test which does not have test in the test name. Since all tests has test in the name, all the tests would be deselected and you would get your total number of tests. This method works with parametrization.

Darius

Answer 3

Loosely based on my other answer.

Count all tests

One-liner:

$ pytest --collect-only -q | head -n -2 | wc -l

Explanation

--collect-only combined with -q outputs one test per line, with a trailing info line. Example:

$ pytest --collect-only -q
test_eggs.py::test_bacon[1]
test_eggs.py::test_bacon[2]
test_eggs.py::test_bacon[3]
test_spam.py::test_foo
test_spam.py::test_bar

no tests ran in 0.00 seconds

The rest is just routine: head -n -2 strips the info line, wc -l counts the lines.

Applying further filtering works as usual, e.g.

$ pytest --collect-only -q -k "fizz" | head -n -2 | wc -l

will count only tests containing fizz in name,

$ pytest --collect-only -q buzz/ fuzz/ | head -n -2 | wc -l

will count only tests inside buzz and fuzz directories etc.

Count tests per test module

If you want to get the info about how many tests are in each module, use --collect-only combined with -qq:

$ pytest --collect-only -qq
test_eggs.py: 3
test_spam.py: 2

Count unique tests (test parametrizations counting as single test)

What the OP initially requested. This is a modification of the above command that strips the parametrization from test names and removes duplicate lines before counting:

$ pytest --collect-only -q | head -n -2 | sed 's/\[.*\]$//' | sort | uniq | wc -l

Packing

Answer 4

One more solution:

egrep -e 'def test_' -r ./ | wc -l

Hope it can help.

Westmoreland

yes that is what I was looking for, grepping the code without executing anything, thx. – Ultramontane