Running all tests from a @Category using Maven

Question

I want to run only a subset of my unit tests, the ones defined by a specific @Category.

So I read several SO questions, such as this one (which is exactly what I am looking for), and also this one.

The solution of my problem seems to be provided by the ClasspathSuite project. So I started to write the NewTest and OldTest interfaces that will define my test categories. Then, I created the AllTests suite:

@RunWith(ClasspathSuite.class)
public class AllTests { }

After that, I created a AllNewTests suite:

@RunWith(Categories.class)
@IncludeCategory(NewTest.class)
@SuiteClasses( { AllTests.class })
public class AllNewTests { }

Finally, I create two JUnit classes, one per category:

@Category(NewTest.class)
public class SomeNewTests {
    // some tests...
}

@Category(OldTest.class)
public class SomeOldTests {
    // some tests...
}

---

Now, if I run AllTests, Eclipse launches all the tests of my project, while Maven fails as no test are found:

mvn test -Dtest=AllTests

...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running my.company.AllTests
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.093 sec
There are no tests to run.

If I run AllNewTests (which is the correct thing to do, right?), in Eclipse everything is fine (i.e. it only run the tests annoted with @Category(NewTest.class)) but Maven returns an error:

mvn test -Dtest=AllNewTests

...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running my.company.AllNewTests
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.125 sec <<< FAILURE!

Results :

Tests in error:
  initializationError(my.company.AllNewTests)

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

The exception thrown is the following:

org.junit.runner.manipulation.NoTestsRemainException
    at org.junit.runners.ParentRunner.filter(ParentRunner.java:256)
    at org.junit.experimental.categories.Categories.<init>(Categories.java:142)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:35)
    at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:24)
    at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
    at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
    at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
    at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:33)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:146)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
    at $Proxy0.invoke(Unknown Source)
    at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:145)
    at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:70)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)

---

My question is what I did wrong?

---

Technical details: Java 6, Maven 3.0.2, JUnit 4.8.1, Surefire plugin 2.7.1, cpsuite-1.2.5

Anal

Answer 1

As an update: as of Surefire plugin v2.11, JUnit 4.8+ style categories are now supported.

The release notes for Surefire v2.11 mention the new feature. The surefire:test goal can is configured using groups.

Hopson

Have you actually succeeded in doing this ? I'm can't seem to make it work. Surefire (2.11 / 2.12) seem to ignore the groups. It simply runs all tests. – Muck

@JanGoyvaerts This was a bug and is now fixed. In fact, to run the categories in 2.11 and 2.12, you have to specify a dependency on the junit 47 provider (surefire-junit47). This is fixed in 2.13-SNAPSHOT, and should work without the explicit dependency. – Twinge

I've made it work in the end as you said. BUT ... there's a big performance problem with the provider. It buffers the logs (that's not too bad) and eats all memory and cpu when the tests last too long. So I switched to the naming convention - as it was meant to be originally. That works without a problem. I've signaled the problem. – Muck

Answer 2

I have solved my problem by creating my own JUnit Runner, that extends the Suite.

The idea is close to the principle by the Classpath Suite project, i.e. looking for the classes existing in the classpath, and keep only the ones that are annotated with a given annotation (@NewTest for example).

If you are interested, you can read the full story on my blog.

Anal

This method is now completely obsolete, with the evolutions made in surefire plugin (see user1034382 answer). – Anal

Answer 3

After reading some blog posts and stackoverflow questions, I finally was able to make this work with the surefire plugin, as user1034382 answered. In my case with version 2.17 of maven-surefire-plugin.

Just to add my two cents, the more up-to-date explanation can be found here: Using JUnit Categories to group tests

But you may run with the following surefire plugin problem:

[ERROR] java.lang.RuntimeException: Unable to load category:

That can be fixed with this other stackoverflow question/answer: Where should I put interface class for Junit @Category?

My answer is just to gather all these info here and avoid googling/reading to many diferent solutions. Al least, this worked to me.

Wellbalanced