Making unit tests fail quickly for mutation testing

Question

One problem encountered with mutation testing is that it's slow, because by default you do a full test run (either a test file, or a suite of test files) for each mutation generated.

One way to make mutation testing faster would be to stop the test run for a given mutant once a single failure has been encountered (but only during mutation testing). Even better would be for the mutation tester to remember what was the first test to kill the last mutant, and give that first to the next mutant. Is there anything in ruby that does either of these things, or is my best bet to start monkey patching?

(Yes, I know unit tests ought to be fast. And showing all of the failed tests is useful outside of mutation testing, as it helps you not merely identify that something's wrong, but pinpoint where it's going wrong)

Edit: I'm currently using heckle with test/unit. If it's not possible for test/unit to remember which tests fail between runnings, maybe heckle or something running heckle could remember it.

Exanthema

and you run these tests how...? – Halibut

Answer 1

Your best best is to check out the heckle source from github, patch it, and submit that patch to the developers. You ought to be able to write a custom test runner for heckle.

Monkey patching is never the answer for something like this. In fact, monkey patching is almost never the answer for anything.

Fasces

Answer 2

My mutant tool uses the rspec2 --fail-fast option to stop immediately once a failing example was encountered. Together with the --rspec-dm2 strategy that only executes affected unit tests we get very fast mutation coverage testing. See this asciicast for a (speed) demonstration.

Chlori

Answer 3

One approach I've started using was writing unit tests for each method, and putting them in separate files, organized like rubyspecs. I run each spec in isolation, specifying the exact method I want to heckle. I have a rake task that manages all this and prints a report at the end with failing mutations, if any.

In the end I get full heckle coverage of each method, without waiting forever for the results. Plus it's even better than the normal approach everyone uses, because I don't get any incidental coverage -- each methods's specs must cover all mutations of that method.

Coaction