JUnit tests pass but PIT says the suite isn't green

Question

While trying to run a PIT mutation test I get the following error:

mutationCoverage failed: All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.

The tests run just fine when I do a normal test build but while running the mutation tests phase they supposedly fail but no details are provided as to why. I have gone through the reasons listed on the PIT Testing FAQ but I still have no clue what could be wrong.

I tried:

• adding the -Dthreads=1 option to rule of any multi threading issue
• could not find any system properties unique the couple tests that are failing
• the tests are not ignored under normal runs

What are some other things I should try? Or other ways to debug what could be going on here?

Painting

Have you tried to enable vebose logging in PIT to get more details about failing tests? – Jemimah

Yes I have. The additional output do not show anything more than helps beyond the normal output. It still just has a warning that indicates the test class and test method that is failing "4:30:45 PM PIT >> WARNING : Description [testClass=com.bobbylough.RandomTest, name=testToResource(com.bobbylough.RandomTest)] did not pass without mutation." – Painting

Answer 1

The common causes of tests failing at the coverage stage are

1. PIT picking up tests not included/are excluded in the normal test config
2. Tests rely on an environment variable or other property set in the test config, but not set in the pitest config
3. Tests have a hidden order dependency that is not revealed during the normal test run
4. PIT doesn't like something in your tech stack - possibly a JUnit test runner

It sounds like you've eliminated 1 & 2. so that leaves 3 and 4.

Test order dependencies can be hard to spot. If the answer is yes to any of these you may have one.

• Does your codebase include mutable static state? (e.g in singletons)
• Do your tests hit a database (in memory or otherwise) where it is possible for state to persist between tests?
• Do your tests modify files on disk?

There are probably also many other causes not listed above.

If you are confident that order dependencies are impossible in your code base, that leaves a problem with these particular tests.

It's hard to guess what this might be without some code. Can you post a simplified version of the test that still fails?

Scant

Thanks Henry. I will keep debugging. Your right it is likely a maven dependency issue which are always fun to debug – Painting

By "order dependency" I mean the suite will not pass when the tests are run in a certain orders. I.e if the suite runs tests in the order A,B,C,D everything is fine, but if ever C is run before B you find that B fails. This occurs when test C changes some state in the jvm that B implicitly relies on. – Scant

ah okay now I understand what you meant. Regrettably I do not have a simplified version that fails because it is on two of over 4,000 JUNIT tests in my project and I haven't been able to distill what is unique about them. You created a wonderful tool for mutation testing. I will keep digging. It may be something like a system property for a static method in an external library that I cannot mock. Thank you for your help – Painting

@Scant - is there any way we can pass/set environment variables to pit. in my case we are using surefire plugin to set environment variables to be used while executing unit test. As this variable is not being passed to pit when it runs test, tests are failing. Please suggest on this.Thanks – Criminology

found the solution to set environment variable as pit plugin configuration - <environmentVariables> <environment>test</environment> </environmentVariables> – Criminology

@Scant Do your tests modify files on disk? 1、There is a file upload in my code, and this error is also reported. what should I do. 2、And in my code there is a configuration to get the configuration center, there is a timeout. How to solve this? – Pignut