Run JUnit test *methods* in isolation

Question

Is maven-surefire-plugin able to run all JUnit test methods in isolation? I.e., is it able to fork a JVM for each test method rather than for each test class?

The deprecated option

<forkMode>pertest</forkMode>

and the current

<forkCount>1</forkCount>
<reuseForks>false</reuseForks>

seem to only fork for each test class.

PS: Test methods should be independent and therefore no one should need to run each one on a new JVM (and not to say that it would be very expensive). But I was wondering whether there is such option or not.

Evered

IntelliJ seems to be able to do it. I'm waiting for a Surefire solution too. – Hypogenous

Answer 1

The Maven Surefire Plugin supports this configuration:

<parallel>methods</parallel>

For JUnit versions >= 4.7

For example:

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.20</version>
    <configuration>
      <parallel>methods</parallel>
      <threadCount>10</threadCount>
    </configuration>
  </plugin>

More details here.

Edit 1 based on your comment:

I would like to run each test method on a new JVM, and not on another thread on the same JVM

When using reuseForks=true and a forkCount > 1, test classes are handed over to the forked process one-by-one. So, with parallel=methods classes are executed in forkCount concurrent processes, each of these processes uses a threadCount number threads to execute the methods of one class in parallel.

So, it looks like surefire does not support your use case.

FWIW, spawning a JVM for every test method sounds like it could get expensive. Are you perhaps dealing with an issue where test methods have unwanted side effects within a JVM (maybe they set System properties or change statics) and these side effects cannot be isolated? If so, perhaps revisting the tests to eliminate these side effects might be more desireable than implementing a bespoke and potentially expensive test runner?

Ragsdale

Thanks @glitch. According to the link you pointed out, "The important thing to remember with the parallel option is: the concurrency happens within the same JVM process.". So, by defining <forkCount>1</forkCount><reuseForks>false</reuseForks> I can get a new JVM per test class, but then if I also include <parallel>methods</parallel><threadCount>10</threadCount> I only get 10 test methods executing concurrently on the same JVM, and not each test method on a new JVM. Right? – Evered

Yes, the classesAndMethods indicates your intent for parallelisation and the forkCount limits the extend of the concurrency. – Ragsdale

Thanks, but that it isn't what I'm looking for. I would like to run each test method on a new JVM, and not on another thread on the same JVM. – Evered

@Evered I've updated my answer following this clarification: "I would like to run each test method on a new JVM, and not on another thread on the same JVM" – Ragsdale

I totally agree with your "Edit 1": running each test method on a new JVM would be insanely expensive, and as you said there could also be some unwanted side effects of doing that. And I also agree if test methods do actually need to be executed isolation (i.e., on a new JVM) they most likely need to be re-written. I was just wondering whether there is an answer (i.e., a sufire option) to my question, not that I actually need or want to implement such thing. – Evered

Ok, well then the answer is no :) – Ragsdale

Answer 2

I found this project which claims to do it, though it did not seem to work when I tried (maybe out of date). Anyway I imagine this could not work for anything but basic @Test-annotated methods: no @RunWith(Parameterized.class) etc.

Tepic