Making Maven run all tests, even when some fail

Question

I have a project with several modules. When all tests pass, Maven test runs them all.

When tests fail in the first module, maven will not continue to the next project. I have testFailureIgnore set to true in Surefire settings, but it doesn't help.

How do I make maven run all tests regardless of earlier failures?

Seafood

What version of the maven-surefire-plugin? – Mallon

testFailureIgnore should work, maybe do a mvn help:effective-pom to make sure it's in there in the right place... – Stateroom

Answer 1

I just found the -fae parameter, which causes Maven to run all tests and not stop on failure.

Seafood

Sadly, this parameter doesn't work when I pass it to TeamCity. – Seafood

In a multi-module project, modules that depend on module that has failed tests will be skipped. Use -fn instead. – Jingo

@Seafood did you found a way to make it work on TeamCity? – Obey

Can you configure this in the pom file? – Unsocial

This only works for modules that don't have dependencies on other modules within the same build tree, see https://mcmap.net/q/98887/-making-maven-run-all-tests-even-when-some-fail – Stateroom

Answer 2

From the Maven Embedder documentation:

-fae,--fail-at-end Only fail the build afterwards; allow all non-impacted builds to continue

-fn,--fail-never NEVER fail the build, regardless of project result

So if you are testing one module than you are safe using -fae.

Otherwise, if you have multiple modules, and if you want all of them tested (even the ones that depend on the failing tests module), you should run mvn clean install -fn.
-fae will continue with the module that has a failing test (will run all other tests), but all modules that depend on it will be skipped.

Brickyard

additionally you could add -e for the cmd to provide some error information. – Brickyard

I just tested the option --fail-never. Maven will not fail the build even if there are compile errors. If I use this option on Jenkins, the build looks successful even if it has lots of compile errors. I prefer -Dmaven.test.failure.ignore=true in this case and let Jenkins analyze the surefire reports. – Briton

@wlnirvana I've edited the post now with this link (so feel free to delete your comment...) – Inhospitality

Can you configure this in the pom file? – Unsocial

@BluE according to the list of tags it doesn't seem so. Maybe there's a way to specify -fae directly in pom.xml? – Pinpoint

Testing "one" module I think doesn't fit...With one module test the build will be interrupted when a failure (unit test, compile) occurs with or without either setting. Unless you're using -Dmaven.test.failure.ignore=true. However it does effect modules as described (skips dependent ones unless you use --fail-never). --fail-never happens to also return a "success" process exit code FWIW :) – Stateroom

Sadly neither of these is enough if there are "unresolvable dependencies" or pom syntax error type failures. Oh well... – Stateroom

I'm using -fn, which seems to do what I want, but then the error code after the build is 0, which isn't what I want. How can I have all the tests run but still get the correct error code? – Domination

Answer 3

Either configure Surefire with <testFailureIgnore>true</testFailureIgnore>.

Or on the command line:

mvn install -Dmaven.test.failure.ignore=true

Mallon

IIRC this has the effect of the whole build not failing, rather than running all tests and failing in the end. – Germanophile

This is the only option that worked for me ... The "-fae" option did nothing on test failures – Brody

Yes if you only have unit test failures this will make the whole build "look like" it passed with success. --fail-at-end or -fae should make it continue on and just fail the build at the end, though it also skips dependent modules see https://mcmap.net/q/98887/-making-maven-run-all-tests-even-when-some-fail – Stateroom

The great thing about this option is that you can specify it under "Global Execution Options" for Maven in NetBeans IDE, which is great when you can't go around modifying every module's POM (I'm working on a project with 198 modules). – Josephinejosephson

Answer 4

Try to add the following configuration for surefire plugin in your pom.xml of root project:

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <testFailureIgnore>true</testFailureIgnore>
        </configuration>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

Tibold

As explained in other responses, this will execute all tests but also mark the build as success even if there are failures – Aceydeucy

Answer 5

A quick answer:

mvn -fn test

Works with nested project builds.

Schnur

Is this better than mvn install -Dmaven.test.failure.ignore=true? – Gonnella