JUnit and Surefire Parallel Tests - ForkCount & ThreadCount

Question

I'm running Selenium tests on the Selenium Grid using the Surefire Plugin to execute tests. In terms of my test breakdown I have several classes, some of which have 1 test in there and some more than one test.

So on my Grid i have 30 chrome web drivers and I want to execute all tests within all classes in parallel.

I've read how to do this using the parallel parameter which i have set as:

            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.17</version>
                <configuration>
                    <includes>
                        <include>${testSuite}</include>
                    </includes>
                    <parallel>all</parallel>
                    <useSystemClassLoader>false</useSystemClassLoader>
                    <perCoreThreadCount>false</perCoreThreadCount>
                    <threadCount>20</threadCount>
                    <browser>${browser_type}</browser>
                </configuration>
            </plugin>

However this doesnt seem to fill all the Chrome web drivers I have available.

If i then use forkCount, like:

<forkCount>20</forkCount>
<reuseForks>true</reuseForks>

Then when the test execution first starts, all web drivers are filled however it quickly starts dropping and behaving one at a time.

So my questions:

• Is there a relationship between forkCount and threadCount
• Is there anything additional I need to do to really get this running in parallel?

Thanks.

Ultramicroscope

Are you using @NotThreadSafe on your tests? Everything you need is here maven.apache.org/surefire/maven-surefire-plugin/examples/… – Derk

No - not using that @NotThreadSafe – Ultramicroscope

selenium scripts are designed to run in parallel? otherwise all the thread actions will happen in single chrome browser. – Chon

You are going to need to write a custom junit runner to allocate the tests to the different parallel threads correctly and clean up after yourself. It doesn't work out of the box, it will appear to do so at first but the threads won't get reused. – Cardamom

Answer 1

You have to provide explicit junit test provider:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.18.1</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>2.18.1</version>
        </dependency>
    </dependencies>
    <configuration>
        <parallel>all</parallel>
        <useUnlimitedThreads>true</useUnlimitedThreads>
        <useSystemClassLoader>false</useSystemClassLoader>
        <includes>
            <include>${testSuite}</include>
        </includes>
        <systemPropertyVariables>
            <browser>${browser_type}</browser>
        </systemPropertyVariables>
     </configuration>
</plugin>

And you should use JUnit 4.7+ as older versions does not work with parallel testing correctly.

Also you can omit fork-related parameters if your tests do NOT affect JVM runtime (usually it's not the case).

Or migrate your tests to TestNG - it is more elegant framework and it works with parallel testing much better, then JUnit (imo).

Mongolia

surefire-junit47 is automatically selected if you use junit 4.8+ and are using a <groups> parameter in your configuration: maven.apache.org/surefire/maven-surefire-plugin/examples/… – Cardamom

@Cardamom - true/false. true, because you are right. false, because OP does not use groups. however explicit declaration guarantees the correct test provider will be used. – Mongolia

Using dependecnies withing plugin made no difference. (I'm using jUnit 4.11) – Ultramicroscope

As I wrote before - you should use JUnit 4.7+ as older versions does not work with parallel testing correctly (via surefire plugin). Consider to update JUnit 4.11 => 4.7. – Mongolia

Isnt 4.11 more recent that 4.7 ? (11 > 7 )? – Ultramicroscope

@Ultramicroscope Huh... Sometimes I'm being stupid. And you opened my eyes. – Mongolia

Answer 2

There are so many configuration for running test in parallel.

According to the documentation:

forkCount

The parameter forkCount defines the maximum number of JVM processes that Surefire will spawn concurrently to execute the tests. It supports the same syntax as -T in maven-core: if you terminate the value with a C, that value will be multiplied with the number of available CPU cores in your system. For example forkCount=2.5C on a Quad-Core system will result in forking up to ten concurrent JVM processes that execute tests.

...

The default setting is forkCount=1/reuseForks=true, which means that Surefire creates one new JVM process to execute all tests in one maven module.

threadCount

When using reuseForks=true and a forkCount value larger than one, test classes are handed over to the forked process one-by-one. Thus, parallel=classes would not change anything. However, you can use parallel=methods: classes are executed in forkCount concurrent processes, each of the processes can then use threadCount threads to execute the methods of one class in parallel.

As fair as I understand, threadCount is like a sub threading for each fork.

You can tweak these params to improve your test performance, for instance you can have:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.17</version>
    <configuration>
         <includes>
              <include>${testSuite}</include>
         </includes>
         <parallel>all</parallel>
         <useSystemClassLoader>false</useSystemClassLoader>
         <perCoreThreadCount>false</perCoreThreadCount>
         <forkCount>2.0C</forkCount>
         <reuseForks>true</reuseForks>
         <threadCount>20</threadCount>
         <browser>${browser_type}</browser>
    </configuration>
</plugin>

You can find more details in its site:

https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

Eirena

I've experimented with these however the issue I get is all my chrome drivers are not being filled even though I am throwing enough tests at it. – Ultramicroscope

@Ultramicroscope seems a problem with your test cases setup and not with the way you want to run them concurrently. For instance, you can remove all parallelism and check if your drivers are filled, if not then you have another problem. As far as I can tell, you may have a problem within your @Before or @BeforeClass annotated method, are you loading your driver there? – Eirena

I removed any parallel configuration and it only ran in one web driver so my config is being used. Also there are no Before or @BeforeClass methods in my code whatsoever – Ultramicroscope

@Ultramicroscope it seems you have a different problem and it isn't related to the concurrency what your question asks for. Your question aims to run your test concurrently and the answers here solves that. If you are saying you have no @Before or @BeforeClass, then I think you are not configuring well your test cases setup. Using both annotation will help you to setup whatever you want (like your drivers you are testing), once you configure that then you can run them concurrently. Unless you are loading every driver individually in each test method you will have an issue – Eirena

@Ultramicroscope Therefore, I'd recommend you to create another question with your unit test samples and ask the problem you are facing. Btw, you can let me know I'd be glad to help you out. – Eirena