Does Maven surefire plugin run tests using multiple threads?

Question

I'm wondering if the Maven surefire plugin either runs tests multi-threaded by default (and if so can the number of threads be controlled? ) or if it runs tests from the Test classes in a random order or predictable order, or if the order can dictated by some means.

I haven't verified this yet (I'll do so tomorrow just looking for some heads up guidance and verification at this point), but it looks as if my various JUnit Test classes are getting the tests run in some intermixed order. Which makes it a real pain to orchestrate the creating of the test resources (which are quite hefty in my case).

Its probably a classic problem I run my suite with the Eclipse JUnit runner and everything runs very linear and plays nice. I go to Maven cmd line and things seems to be stepping all over each other.

Annoying

Answer 1

By default, Maven runs your tests in a separate ("forked") process, nothing more (this can be controlled using the forkMode optional parameter).

If you are using TestNG or Junit 4.7+ (since SUREFIRE-555), it is possible to run tests in parallel (see the parallel and the threadCount optional parameters) but that's not a default.

Now, while I'm not sure if the surefire plugin behaves the same as JUnit, it is possible to get some control by manually creating a TestSuite and specify the order in which tests are executed:

TestSuite suite= new TestSuite();
suite.addTest(new MathTest("testAdd"));
suite.addTest(new MathTest("testDivideByZero")); 

You are however strongly advised never to depend upon test execution order, unit tests really should be indeed independent.

P.S.: Just in case, there is also this request SUREFIRE-321 (to run tests in alphabetical order) that you might want to vote for.

Grover

Hi, aware of avoiding interactive tests, trying to get there... Voted for both issues. Looks like 555 is now part of release 2.5 repo1.maven.org/maven2/org/apache/maven/plugins/…. – Annoying

please make use of surefire plugin version 2.4.2 or above in case you dont wanna face problems related to OOM – Paulownia

Answer 2

First of all, your unit tests should be independent of each other. This is because the order of execution is not guaranteed even by JUnit, so each test should set up and tear down its context (aka test fixture) independent of what happens before or after.

The order of execution is definitely not random though, in JUnit it tends to be the same (I would guess alphabetical order), but you should not build on it - it can change anytime, and apparently in Surefire the order is different.

Here is a good link on why interacting tests are not a good idea.

Ikey

Hi, aware of avoiding interactive tests, trying to get there... thanks for the advice and the link. – Annoying

Answer 3

JUnit runs tests in the order in which they appear in the .java file (not alphabetically). Maven-surefire runs them in a different order, but not predictably (as far as I can tell).

Ideally, tests would be independent of one another, but singletons and static context can complicate things. A helpful way to get new static contexts between executions of separate TestCase's (but not individual tests) is to set the forkMode variable in your pom.xml..

<forkMode>always</forkMode>

Dulaney

Answer 4

I have found that if you are using the -T option in your maven command, Surefire will then fork into forkCount * <specified number of threads by the -T option> number of concurrent processes.

To make them all run in one process despite having multiple threads specified by -T, you can force forkCount to be 0 by adding the option -Dsurefire.forkCount=0

Lesh

"Regarding the compatibility with multi-module parallel maven builds via -T, the only limitation is that you can not use it together with forkCount=0. When running parallel maven builds without forks, all system properties are shared among the builder threads and surefire executions, therefore the threads will encounter race conditions when setting properties, e.g. baseDir, which may lead to changing system properties and unexpected runtime behaviour."—maven.apache.org/surefire/maven-surefire-plugin/examples/… – Stiletto