I am having a problem trying to configure Maven to excluded some categories of unit tests based on multiple profiles.
I have two categories defined for unit testing: SlowTest
for those unit tests that take a very long time to run, and WindowsTest
for those unit tests that can only be executed within a Windows environment.
I have two profiles defined in my project's pom.xml
file as follows:
<profiles>
<!-- Exclude any tests in `SlowTest` category when profile 'skipSlowTests' is specified. -->
<profile>
<id>skipSlowTests</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludedGroups>SlowTest.class</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<!-- Skip any tests in 'WindowsTest' category when not running on Windows. -->
<profile>
<id>skipWindowsTests</id>
<activation>
<os><family>!windows</family></os>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludedGroups>WindowsTest.class</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So for example when I want to run the tests and exclude any slow tests, I can execute mvn -PskipSlowTests test
. And If I want to skip any Windows tests I can execute mvn test
on a non-Windows OS (or explicitly specify -PskipWindowsTests
on the command line).
The problem here is that when both the skipSlowTests
and skipWindowsTests
profiles are activated then only the Windows tests are skipped. The slow tests still run. When maven constructs the effective pom.xml
file, it applies the skipSlowTests
profile and configures maven-surefire-plugin
to exclude the groups SlowTest.class
. Then the skipWindowsTests
profile is applied and this overwrites the excluded groups to be WindowsTest.class
.
What I really want is the excluded groups to be SlowTest.class,WindowsTest.class
when both profiles are activated, but I cannot figure out a way to do this.
There's no way that I can see in maven of appending a value to a property, e.g.
<properties>
<excludedGroups>IgnoreTest.class</excludedGroups>
</properties>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludedGroups>${excludedGroups}</excludedGroups>
</configuration>
</plugin>
<profile>
<id>skipSlowTests</id>
<property>
<excludedGroups>${excludedGroups},SlowTest.class</excludedGroups>
</property>
...
</profile>
<profile>
<i>skipWindowsTests</id>
<property>
<excludedGroups>${excludedGroups},WindowsTest.class</excludedGroups>
</property>
...
</profile>
This pom.xml
is broken, because there's recursion in assigning a value to the excludedGroups
property.
And there's no way to set up a third profile (e.g. skipSlowTestsAndWindowsTests
) that activates whenever both the skipSlowTests
and skipWindowsTests
are activated.
Any suggestions here? Maybe adding separate phases to maven-surefire-plugin
(one for slow tests, one for windows tests, one for all others) and using profiles to determine which phases run?
Edit #1
I tried creating separate execution phases for the maven-surefire-plugin
as follows:
<!-- By default, do not skip slow tests or windows tests -->
<properties>
<skipSlowTests>false</skipSlowTests>
<skipWindowsTests>false</skipWindowsTests>
</properties>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<executions>
<!-- Default execution will skip SlowTest and WindowsTest categories -->
<execution>
<id>default-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludedGroups>SlowTest,WindowsTest</excludedGroups>
</configuration>
</execution>
<!-- Slow Test execution will run the slow tests only (if not skipped) -->
<execution>
<id>slow-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>${skipSlowTests}</skip>
<groups>SlowTest</groups>
</configuration>
</execution>
<!-- Windows Test execution will run the windows tests only (if not skipped) -->
<execution>
<id>windows-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>${skipWindowsTests}</skip>
<groups>WindowsTest</groups>
</configuration>
</execution>
</executions>
</plugin>
<profiles>
<!-- Skip slow tests when skipSlowTests profile enabled -->
<profile>
<id>skipSlowTests</id>
<properties>
<skipSlowTests>true</skipSlowTests>
</properties>
</profile>
<!-- Skip windows tests when skipWindowsTests profile enabled -->
<profile>
<id>skipWindowsTests</id>
<properties>
<skipWindowsTests>true</skipWindowsTests>
</properties>
...
</profile>
</profiles>
This configuration will run tests in three separate executions: the default-test
execution will run every test except for slow tests and windows tests; the slow-test
execution will run the slow tests only; and the windows-test
execution will run the windows tests only. And if both the skipSlowTests
and skipWindowsTests
profiles are enabled, then both slow tests and windows tests will be skipped.
The problem here is when there is a test marked with both SlowTest
and WindowsTest
categories:
@Test @Category({SlowTest.class,WindowsTest.class})
public void slowWindowsTestCase() { ... }
When the skipSlowTests
profile only is enabled, then the windows-test
execution is still run, and since the slowWindowsTestCase
method is marked as a windows test, it will be executed, even though we want to skip slow tests. Similarly, this test case will also be executed when the skipWindowsTests
profile only is enabled. The only way to skip this test case is to specify both the skipSlowTests
and skipWindowsTests
profiles.
I tried modifying the executions as follows:
<execution>
<id>slow-test</id>
...
<groups>SlowTest</groups>
<excludedGroups>WindowsTest</excludedGroups>
...
</execution>
<execution>
<id>windows-test</id>
...
<groups>WindowsTest</groups>
<excludedGroups>SlowTest</excludedGroups>
...
</execution>
But now the slowWindowsTestCase
method is never executed.
Edit #2
And even if I could get the test executions working correctly, I would need to add a dependency on maven-surefire-plugin
so that the SlowTest
and WindowsTest
interfaces can be loaded during testing. I have put maven-surefire-plugin
in the pluginManagement
section of my parent POM, but any dependency defined for this plugin cannot have test
scope. As a result, running a test outside of the module that defines the SlowTest
and WindowsTest
interfaces will not work. I can add a test
scope dependency for the plugin in the profiles, but that will not cover the case when no profiles are active.
<excludedGroups>
: it should not be a type ofString
but rather anArray
e.g.<excludedGroups><excludedGroup>A</excludedGroup><excludedGroup>B</excludedGroup></excludedGroups>
. Now one can easily combine items into one list. You're welcome to fire an issue for improvement. – Epilate