Is there a way to start unit tests which related only to changed code?

Question

In my Python project, we have a big number of unit tests (some thousands). Though they are logically distributed between files and classes, I need sometimes a lot of time in order to find ones, which cover the functionality I'm changing.

Of course, I can run all test from some specific file/class, but again because of big number of that tests, It'll be time-consuming to run them continuously (I'm executing unit tests each time after saving a file in my IDE).

So in general I need some solution which will do following activities at a time:

• Tracks which files have been changed since last file saving
• Traces dependencies between code that have been changed in those files and unit tests which cover that code
• Selectively executes only those unit tests which cover the code that has been affected

Does anyone have idea about something similar?

Mandle

Changes in code shouldn't effect proper unit tests... You don't need a unit tests for every function / method in your codebase. You should stick to unit testing just the public interface (API) for your code. This way when you change implementation details (say you decide to use a new algorithm to accomplish something) or when you refactor your code your tests will stick work. – Fablan

@CharlesP You're referring to black-box testing (end-to-end). The OP is probably talking about white-box tests. – Institution

Answer 1

You might like to checkout pytest-incremental:

The idea is to execute your tests faster by executing not all of them but only the “required” ones.

Install via pypi:

pip install pytest-incremental

Usage:

$ py.test --inc

I think it does what you are looking for, it "looks for imports recursively to find dependencies (using AST)" and runs only the changed tests.

Nonfulfillment

There also is github.com/eventbrite/nose-knows, which does something similar for nose. – Thermistor

Answer 2

pytest-testmon is a pytest plugin which selects only tests affected by changes since last execution. It uses Coverage.py to track individual tests dependencies and and compares and saves checksums of methods on each execution.

Lobachevsky

Answer 3

Assumption1: You can set your IDE to fire a script on file save.

Assumption2: That script can receive the name of the file being saved and access the file contents.

You could create a test coverage description "file" (storage location: irrelevant) something along lines of:

**module**         **tested by**
mymod1.py       testsuite1.py
mymod2.py       testsuite2.py
mysubmod1.py    testsuite3.py
mysubmod2.py    testsuite3.py

read your saved file to get the import statements and retrieve all needed values from the tested by column to cover your saved file and dependencies. Then run your test modules.

I suppose this could also work from a command line, taking the changed file name as a parameter.

I think maybe the dependency tracking needs to go the other way. If so, then have to parse the imports for everything in the directory looking for the changed module. The rest of the process would be the same.

Orthodontia