Is there a way to "fail fast" for junit with the maven surefire plugin?
Asked Answered
P

8

66

I'm currently working on a java project using maven. We use the maven surefire plugin to run our junit suite as part of the build process.

Our test suite is rapidly growing, in both coverage and execution time. The execution time is very frustrating and time consuming when you end up waiting ten minutes to find out that a test failed in the first minute of testing.

I would like to find a way to make the build process fail upon the first error/failure in the test suite. I know that this is possible for other build tools, but I have been unable to find a way to do this with maven surefire.

I know that there is an unresolved ticket for this functionality in the surefire jira, but I'm hoping that there is an existing solution out there for this.

Parrisch answered 17/12, 2009 at 19:0 Comment(0)
B
71

As of September 6th 2015 it appears there is, -Dsurefire.skipAfterFailureCount=1.

As of October 19th 2015 version 2.19 has been released.

Birgitbirgitta answered 17/9, 2015 at 21:46 Comment(7)
Documentation: maven.apache.org/surefire/maven-surefire-plugin/…Mohur
This option appears to keep processing all the remaining tests to mark them as skipped. Very odd, and not as fail-fast as one might hope.Nottinghamshire
@MatthewRead I am unsure how this works, I have never actually used it (or maybe I did 7 years ago and am forgetting the specifics haha). Is it actually "running" the tests only to say they were skipped at the end? If so I would consider opening a bug with Surefire.Birgitbirgitta
Unfortunately, this does not work. I use surefire-plugin 2.22.2, and it still executes every test, even if the first one fails...Baroque
@Baroque I'd suggest you open a bug with Surefire then, that sounds like a regression.Birgitbirgitta
This seems to be only Junit 4 feature? :(Restful
This does not work yet with JUnit 5, see github.com/apache/maven-surefire/pull/503.Hazzard
G
13

As far as I know, no, and this really requires the resolution of SUREFIRE-580. If you want to make this happen faster, you should at least vote for the issue and, optionally, submit a patch ;)

Gragg answered 17/12, 2009 at 22:31 Comment(0)
H
13

There may be a suitable workaround, but it depends on your requirements and you need to use a CI server which can handle jvm process return codes.

The basic idea is to stop Maven's JVM process altogether and let the OS know that the process has stopped unexpectedly. Then, a continuous integration server like Jenkins/Hudson should be able to check for a non-zero exit code and let you know that a test has failed.

The first step is to make surefire exit the JVM at the first test failure. You can do that with JUnit 4.7 or higher by using a custom RunListener (put it in src/test/java):

package org.example
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
public class FailFastListener extends RunListener {
        public void testFailure(Failure failure) throws Exception {
                System.err.println("FAILURE: " + failure);
                System.exit(-1);
        }
}

Then, you need to configure that class so surefire will register it with the JUnit 4 Runner. Edit your pom.xml and add the listener configuration property to the maven-surefire-plugin. You will also need to configure surefire to not fork a new JVM process for executing the tests. Otherwise, it will just go on with the next test cases.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.10</version>
    <configuration>
        <forkMode>never</forkMode>
        <properties>
            <property>
                <name>listener</name>
                <value>org.example.FailFastListener</value>
            </property>
        </properties>
    </configuration>
</plugin>

If this does not help, I'd try to fork the maven surefire junit provider plugin.

Btw, unit tests, by definition, should run faster than 0.1 seconds. If your build really takes so long due to the unit tests, you will have to make them run faster in the future.

Halvaard answered 18/10, 2011 at 7:7 Comment(2)
excellent solution. Btw, it looks like the <forkMode>never</forkMode> restriction can be relaxed by running maven with the -ff option. After System.exit() surefire will immediately abort the build with a message "goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?"Supine
this looks like the solution to my problem, but i can't get it working, tests keep on running after catching an error. On the other and, I don't get any error message regarding my RunListener or my pom file. does anyone have an idea what could be wrong?Anniceannie
R
7

You can run maven with --fail-fast option:

The -ff option is very useful for developers running interactive builds who want to have rapid feedback during the development cycle.

an example can be:

mvn clean test -ff

http://books.sonatype.com/mvnref-book/reference/running-sect-options.html

Reich answered 11/10, 2017 at 10:55 Comment(1)
This is Maven's default behavior. This causes the build for multi-project builds to fail as soon as one of the projects fails. All of that project's unit tests will continue to run after the first failure.Birgitbirgitta
S
3

This isn't a direct answer to the question, but it can also be useful to feed the output of maven through grep, to strip out most stuff and help you to see where the test failures are.

Like this:

mvn test | grep -w 'Running\|Tests'

Which produces output (for my code) like this:

Running scot.mygov.pp.test.dashboard.DashboardJsonSerDesCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.029 sec
Running scot.mygov.pp.test.dashboard.DashboardBaselineCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 sec
Running scot.mygov.pp.test.dashboard.DashboardDatabaseValidationTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.264 sec
Running scot.mygov.pp.test.dashboard.DashboardServiceWebServiceIsolationCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec

Much easier to see where the first error of failure is.

Sarge answered 16/11, 2015 at 15:57 Comment(0)
T
2

For Junit 5, you can fail fast using this gist:

https://gist.github.com/alexec/7e065b4049a8787fa9ac3f2ca522acf3

You annotated your tests classes with @ExtendWith(FailFast.class) and then your test. Test annotated with @Order(2) only runs if any @Order(1) succeeded.

Tunnel answered 20/1, 2023 at 16:10 Comment(0)
P
0

This doesn't solve the question exactly, but the solution that my workplace ultimately came up with was to use Atlassian's Clover to run specialized builds of just tests pertaining to changed code.

We have a Clover build that runs the tests for the changed code and in turn kicks off the full test build.

This has proven to be a satisfactory solution.

Parrisch answered 16/3, 2010 at 21:21 Comment(0)
Q
0

A few ways to speed it up if not exactly what you need:

Quintillion answered 20/12, 2010 at 17:11 Comment(1)
--fail-fast is the default behavior.Birgitbirgitta

© 2022 - 2024 — McMap. All rights reserved.