Parallelize test execution in a @Rule

Question

I want to reuse some integration tests for load testing purposes. I implemented a rule which is parameterized by an annotation:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Parallel {

    int invocations() default 1;

    int rampUpTime() default 0;
}

In my rule implementation the annotation is evaluated and a statement is set up, which has a evaluate method like that:

@Override
public void evaluate() throws Throwable {
    ScheduledExecutorService exe = Executors.newScheduledThreadPool(invocations);

    for (int i = 0; i < invocations; i++) {
        ScheduledFuture<?> scheduledFuture = exe.schedule(new Runnable() {

            @Override
            public void run() {
                try {
                    invocated++;

                    // Request test = Request.method(description.getTestClass(), description.getMethodName());
                    // new JUnitCore().run(test);

                    statement.evaluate();
                } catch (Throwable e) {
                    e.printStackTrace();
                }
            }
        }, i * rampUpTime, this.timeUnit);
        futures.add(scheduledFuture);
    }
}

So, the evaluate call gets wrapped in a Runnable() and scheduled as described in the annotation. The thing is: In my rule, only the scheduling takes place, the runner doesn´t know (or care) about all the runables which only run as long as it takes to set up the whole test suite. So I´m trying to add the calls to evalute() to the test runner. First try was to use JUnitCore.run(...) which of course ends in a recursion.

Next try was to collect all the futures and wait for them to finish. This works fine on a per test basis, but I want to execute a whole test suite in parallel. And also, in my test report, I only see the test being executed once.

So then I thought I use a parameterized suite with a context (an object which collects all the futures from all tests) as parameter, but I didn´t found a way to promote this context object to the tests, each test has to have its own parameters.

I´m now asking for a way to add multiple executions from my rule to the test runner which is executing it.

Berneta

LoadTest from clarkware.com/software/JUnitPerf.html might help – Icky

Why not using a dedicated stressing tool? something like jMeter or Gatling? – Manducate

Why not simply reusing existing integration tests? I know that it´s possible to run JUnit tests in JMeter, but it´s painful. The integration tests are Eclipse Plugin unit tests and it would be possible to build the application with the eclipse test framework included and use this in JMeter. – Berneta

Are you aware of the method org.junit.runners.ParentRunner.setScheduler() in JUnit 4.10? According to the Javadoc, it is "highly experimental", but you may be able to use it to modify the scheduling behavior in your test suite. Currently the default "scheduler" simply executes whatever it gets passed - in your case I guess it should instead "truly schedule" all test methods annotated with @Parallel and then run the futures when RunnerScheduler.finished() is called. – Lacquer

Yes, I fiddled around with that one in conjunction with org.junit.experimental.ParallelComputer, but that may be the way to go - extend ParallelComputer to make it aware of my annotations and use these for scheduling. Maybe you want to formulate that as an answer, the bounty is ending soon ;-) – Berneta

I wouldn't recommend using a rule to run a test multiple times. JUnit assumes (as do many test authors) that a test class is instantiated once per test method. – Geisel

Answer 1

If I'm understanding you correctly, you are trying to control the execution behaviour of the test runner. A TestRule might be to limited to achieve that goal. But how about writing your own Runner which controls the execution flow of your unit test? Just have a look at the ParentRunner of JUnit. From there, you should get a pretty good idea how the basic scheduling works, and maybe implement your own version of org.junit.experimental.ParallelComputer.

Setaceous