Running junit tests in parallel in a Maven build?

Question

I'm using JUnit 4.4 and Maven and I have a large number of long-running integration tests.

When it comes to parallelizing test suites there are a few solutions that allow me to run each test method in a single test-class in parallel. But all of these require that I change the tests in one way or another.

I really think it would be a much cleaner solution to run X different test classes in X threads in parallel. I have hundreds of tests so I don't really care about threading individual test-classes.

Is there any way to do this?

Johiah

Answer 1

From junit 4.7 it's now possible to run tests in parallel without using TestNG. Actually it has been possible since 4.6, but there are a number of fixes being made in 4.7 that will make it a viable option. You may also run parallel tests with spring, which you can read about here

Johiah

The linked page says "for most dual-core solutions, running with parallel threads is currently never any faster than running non-threaded". Is that still the case? – Bedizen

I would think that if your tests do any IO they would still benefit. For example, if your unit tests are more like integration tests and hit the database, running in parallel should speed them up. – Kazoo

@Bedizen Dont expect too much for the short non-io-bound unit tests is what I'm trying to say. Newer versions of surefire are also better/more efficient than 2.5 described in the post, so you may get slightly better results. – Johiah

You state it's possible, but can you include a link to an explanation of how? Your second link is for "with spring", which I'm not interested in. – Stomatology

@Johiah link? I'm struggling to find a built-in solution. – Seacoast

Answer 2

Use maven plugin:

<build>
    <plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.7.1</version>
        <configuration>
            <parallel>classes</parallel>
            <threadCount>5</threadCount>
        </configuration>
    </plugin>
    </plugins>
</build>

Schroeder

<parallel> is actually supported by surefire if you're using Junit 4.7 or later. surefire guide – Pellerin

Answer 3

Inspired by JUnit's experimental ParallelComputer runner I've built my own ParallelSuite and ParallelParameterized runners. Using these runners one can easily parallelize test suites and parameterized tests.

ParallelSuite.java

public class ParallelSuite extends Suite {

    public ParallelSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {

        super(klass, builder);

        setScheduler(new RunnerScheduler() {

            private final ExecutorService service = Executors.newFixedThreadPool(4);

            public void schedule(Runnable childStatement) {
                service.submit(childStatement);
            }

            public void finished() {
                try {
                    service.shutdown();
                    service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
                } catch (InterruptedException e) {
                    e.printStackTrace(System.err);
                }
            }
        });
    }
}

ParallelParameterized.java

public class ParallelParameterized extends Parameterized {

    public ParallelParameterized(Class<?> arg0) throws Throwable {

        super(arg0);

        setScheduler(new RunnerScheduler() {

            private final ExecutorService service = Executors.newFixedThreadPool(8);

            public void schedule(Runnable childStatement) {
                service.submit(childStatement);
            }

            public void finished() {
                try {
                    service.shutdown();
                    service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
                } catch (InterruptedException e) {
                    e.printStackTrace(System.err);
                }
            }
        });
    }
}

Usage is simple. Just change @RunWith annotations value to one of these Parallel* classes.

@RunWith(ParallelSuite.class)
@SuiteClasses({ATest.class, BTest.class, CTest.class})
public class ABCSuite {}

Merc

Can i run parallel for all my test classes with out mentioning its name? – Lenes

Answer 4

tempus-fugit offers something similar, check the docs for details. It relies on JUnit 4.7 and you just mark your test to @RunWith(ConcurrentTestRunner).

Cheers

Archaimbaud

Answer 5

You can check out the open source library - Test Load Balancer. It does exactly what you ask for - run different test classes in parallel. This integrates at the ant-junit level so that you do not have to change your tests in anyway. I am one of the authors of the library.

Also, think about not running them in threads as you may need a process level sandbox. For example, if you are hitting a DB in your integration tests, you do not want one test to fail because another test added some data in a different thread. Most of the times, tests are not written with this in mind.

Finally, how have solved this problem till now?

Intrastate

Answer 6

You can run the tests in parallel using ParallelComputer provided by Junit itself. Here's a small snippet to get you started.

Class[] cls = { TestCase1.class, TestCase2.class };
Result result = JUnitCore.runClasses(ParallelComputer.classes(), cls);
List<Failure> failures = result.getFailures();

This will help when you need to run tests from code as it has no dependencies on Maven or any other build management tools.

Please note that, this will run all test cases in parallel, if you have any dependencies between different test cases it might result in false positives. You SHOULD NOT have interdependent tests anyway.

Impassion

Answer 7

TestNG can do that (this was my first reflex - then I saw you're already having a lot of testcases).

For JUnit, look at parallel-junit.

Inexorable

Unfortunately this is not the answer to the question I am asking. parallel-junit only runs within a single test class. TestNG also only runs within a single class, and my tests are not TestNG tests. – Johiah

@PlatinumAzure: I updated the link. I do not know how this project is maintained. Another question was asked recently to distribute junit tests execution on several machines. – Inexorable

Answer 8

Another choice: Punner, a new parallel junit runner and maven plugin. You don't have to change your code, copy it to your pom.xml:

<!-- Disable default surefire based testing -->
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.20</version>
  <configuration>
    <skip>true</skip>
  </configuration>
</plugin>

<plugin>
  <groupId>com.github.marks-yag</groupId>
  <artifactId>punner-maven-plugin</artifactId>
  <version>${version}</version>
  <configuration>
  </configuration>
  <executions>
    <execution>
      <id>test</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Punner can run test methods in parallel, can keep test outputs separately and clean.

Punner will reduce your mvn console outputs, like this:

[INFO] --- punner-maven-plugin:0.9.13:test (test) @ ipc ---
[INFO] Punner report directory: /Users/guile/workspace/ipc/target/punner-reports
[INFO]
[INFO] com.github.yag.ipc.IPCTest.testConnectionHandler.............. PASSED
[INFO] com.github.yag.ipc.IPCTest.testSequence....................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testPartialContent................. PASSED
[INFO] com.github.yag.ipc.IPCTest.testResponseContent................ PASSED
[INFO] com.github.yag.ipc.IPCTest.testPingPong....................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testServerClose.................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testServerSideHeartbeatTimeout..... PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientSideHeartbeatTimeout..... PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientSideHeartbeat............ PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientReconnect................ PASSED
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.952 sec, Time saved: 25.919 sec.

Punner produce surefire compatible outputs, you can also get raw log data and a markdown format report from reports directory:

➜  ipc git:(develop) ll target/punner-reports
total 104
-rw-r--r--   1 guile  staff    11K Oct 15 23:07 TEST-com.github.yag.ipc.IPCTest.xml
-rw-r--r--   1 guile  staff   298B Oct 15 23:07 com.github.yag.ipc.IPCTest.txt
drwxr-xr-x  12 guile  staff   384B Oct  8 00:50 logs
-rw-r--r--   1 guile  staff    33K Oct 15 23:07 report.md

Punner is my personal project, I written Punner to speed up unit test phase of some other projects such as IPC framework, fine-grained locking, journal service, distributed workflow engine, etc. It saved a lot of my waiting time.

Punner don't support some advanced feature yet. I'm very glad if you could try it and give me some feedback.

Deccan

I have problem finding source code of given plugin or website. Please add link to homepage of project. – Crandale

Answer 9

You can change your test to be TestNg test in a minute (you just need to change imports), TestNG is the best in parallel testing.

Sermonize

Answer 10

You could try Gridgain that lets you run distribute your tests across a compute grid.

Kenay

I've tried out the GridGain solution and have had two major problems. Firstly, you have to tell GridGain to exclude from your grid task's classpath anything that GridGain also uses, e.g. Spring and lots of Apache Commons stuff. Secondly, the network classloading, while a brilliant idea, doesn't work for libraries that want to search the classpath, e.g. Spring – Bellyful