How to avoid surefire test to be run in parallel

Question

I have an application here where the unit tests are written in a way that they cannot be run in parallel.

When running the tests with maven some of them fail for that reason. I could verify that they are run in parallel by doing a

System.out.println(System.currentTimeMillis() +">>> executing testXXX");
System.out.println(System.currentTimeMillis() +">>> finished  testXXX");

at the start and end of each method. The output is:

1530602546964>>> executing testInstantiation                                                                                                                         
1530602547036<<< finished  testInstantiation                                                                                                                         
1530602547042>>> executing testSimilarNamedResources                                                                                                                 
1530602547050>>> executing testTranslateWithMissingKey                                                                                                               
1530602547051>>> executing testTryTranslateWithMissingKey                                                                                                            
1530602547051<<< finished  testTryTranslateWithMissingKey                                                                                                            
1530602547051>>> executing testTranslationMapWithMissingKey                                                                                                          
1530602547055>>> executing testSilentlyIgnoringExceptionTranslationMapWithMissingKey                                                                                 
1530602547055<<< finished  testSilentlyIgnoringExceptionTranslationMapWithMissingKey   

As we can see after testSimilarNamedResources started, some other tests are started, too.

I tried to configure the surefire plugin to not run in parallel:

<build>                                                                                                                                                           
   <plugins>
     <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-surefire-plugin</artifactId>
       <version>2.22.0</version>
       <configuration>
         <!--parallel>false</parallel-->
         <threadCount>1</threadCount>
         <perCoreThreadCount>false</perCoreThreadCount>
       </configuration>
     </plugin>
   </plugins>
</build>

But it still executes these tests in paralled. I ran mvn with the -X option to see whether my configuration was applied and got this output:

$ mvn -X test | grep -iE "(parallel|threadcount)"                                                                                                                
  <parallel>${parallel}</parallel>                                                                                                                               
  <parallelMavenExecution default-value="${session.parallel}"/>                                                                                                  
  <parallelOptimized default-value="true">${parallelOptimized}</parallelOptimized>                                                                               
  <parallelTestsTimeoutForcedInSeconds>${surefire.parallel.forcedTimeout}<parallelTestsTimeoutForcedInSeconds>                                                   
  <parallelTestsTimeoutInSeconds>${surefire.parallel.timeout}<parallelTestsTimeoutInSeconds>                                                                     
  <perCoreThreadCount default-value="true">false</perCoreThreadCount>                                                                                            
  <threadCount>0</threadCount>                                                                                                                                   
  <threadCountClasses default-value="0">${threadCountClasses}</threadCountClasses>                                                                               
  <threadCountMethods default-value="0">${threadCountMethods}</threadCountMethods>                                                                               
  <threadCountSuites default-value="0">${threadCountSuites}</threadCountSuites>                                                                                  
[DEBUG]   (f) parallelMavenExecution = false                                                                                                                     
[DEBUG]   (s) parallelOptimized = true                                                                                                                           
[DEBUG]   (s) perCoreThreadCount = false                                                                                                                         
[DEBUG]   (s) threadCount = 0                                                                                                                                    
[DEBUG]   (s) threadCountClasses = 0                                                                                                                             
[DEBUG]   (s) threadCountMethods = 0                                                                                                                             
[DEBUG]   (s) threadCountSuites = 0                    

Do I miss something in the plugin configuration?

UPDATE:

I have given up. The behaviour was too strange. Trying to create simple sample didn't work. Those tests were not run in parallel. I didn't found out why this was the case here. We will revamp the whole code and therefore also the unit tests. Not need to find a solution anymore, but it still puzzles me why it showed this strange behaviour…

Zackzackariah

I have an application here where the unit tests are written in a way that they cannot be run in parallel - They are written the wrong way. A test must be able to run independently from other tests. If your test depends on the results of other tests it's not a good test, you should rework it to be independent. – Jacobian

@Jacobian I know that. It's not my code. But it is as it is. Until the tests (and maybe the code) have been reworked to support parallel runs, I want to get this to correctly execute the tests. – Zackzackariah

I'm not sure about that, but I'd expect it to not be possible, since tests are made to run parallelly without dependencies on other tests. So it might be that the library won't allow you to do that. I could be wrong though. – Jacobian

BTW: The tests do not depend on each other. They interfere each other when run in parallel. This seems to caused by the usage of some static mutable data. – Zackzackariah

It must not. as by default : surefire (JUnit runner here) doesn't execute test in parallel. Sequential test execution is indeed the default behavior. So you don't need to set anything to get it. Besides, the debug output confirms that no parallel execution is enabled : > [DEBUG] (f) parallelMavenExecution = false. Please provide a Minimal, Complete, and Verifiable example. – Inelegant

@backslash There are many reasons why you might not want test run in parallel. In my case I can't read the log files if the test run in parallel. I would be happy with a temporary solution for debugging only. Why is everybody so quick to dismiss the need for a certain feature? – Catanddog

@Catanddog I understand your point. In a scenario like yours I'd stub (part of) the log library so that each test writes logs in a stream that you own and can control, independent from what other tests are doing. I don't think it's about dismissing the need for a certain feature, in my opinion it's a matter of understanding how tests work. A unit test should be self contained, so libraries take this for granted, shared resources aren't an option. The fact that you need a feature doesn't mean it's missing, instead you might need to re-engineer your solution to meet the library's standards. – Jacobian

Answer 1

I just spent a few hours dealing with this and ended up resolving it by explicitly setting forkCount to 1. The default is supposed to be 1, but it sure doesn't act like it.

I confirmed this by printing unique names for the running JVM (using ManagementFactory.getRuntimeMXBean().getName()) in a few test cases, each in their own class, along with 30 second Thread.sleep() calls. Without explicitly setting forkCount, sleep() did not block execution of other test cases - teach test clearly ran simultaneously, and getName() returned clearly different values. When I explicitly set forkCount to 1, the sleep() calls blocked execution as expected and getName() returned the same value every time.

Also, just wanted to add @NotThreadSafe is apparently not working as intended either. I tried it first with no luck.

Skepful

Answer 2

I would recomend you follow this guidance:

Since of maven-surefire-plugin:2.18, you can apply the JCIP annotation @net.jcip.annotations.NotThreadSafe on the Java class of JUnit test (pure test class, Suite, Parameterized, etc.) in order to execute it in single Thread instance. The Thread has name maven-surefire-plugin@NotThreadSafe and it is executed at the end of the test run. https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html#Parallel_Test_Execution_and_Single_Thread_Execution

Like others have said they should really be rewritten and by exception the tests that absolutely cannot run in parallel should be annotated.

For your plugin definition i suggest you do let it run in parallel as it can be quite a bit faster on modern machines.

I have mine defined as:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <configuration>
        <useUnlimitedThreads>true</useUnlimitedThreads>
        <runOrder>random</runOrder>
        <redirectTestOutputToFile>true</redirectTestOutputToFile>
        <rerunFailingTestsCount>1</rerunFailingTestsCount>
        <parallel>methods</parallel>
        <forkedProcessExitTimeoutInSeconds>2</forkedProcessExitTimeoutInSeconds>
    </configuration>
</plugin>

Manakin

„should really be rewritten“ — There are many reasons why you might not want test run in parallel. In my case I can't read the log files if the test run in in parallel. I would be happy with a temporary solution. Don't dismiss the need for a certain feature that quickly. – Catanddog