How to Run a JMH Benchmark in Maven Using exec:java Instead of exec:exec?

Question

This post on javapapers.com shows how to run a JMH benchmark in Maven by typing mvn exec:exec. Running JMH within Maven is pretty handy, since you can easily run it from an Eclipse Run Configuration or even in a Maven phase.

However, there are two problems with this setup:

1. When you kill Maven, JMH will continue running in the background, as exec:exec starts it in a separate VM.

2. Usually, JMH will start yet another VM to run the benchmarks, so you will end up with at least 3 VMs running at the same time.

Fortunately, the Exec Maven Plugin comes with a second goal, exec:java, which executes a main class directly in the VM Maven runs. However, when I tried to configure Maven to run JMH using exec:java, the benchmark crashes because of missing classes:

# JMH 1.11.3 (released 40 days ago)
# VM version: Error: Could not find or load main class org.openjdk.jmh.runner.VersionMain
# VM invoker: C:\Program Files\Java\jdk1.7.0\jre\bin\java.exe
[...]
# Run progress: 0.00% complete, ETA 00:02:40
# Fork: 1 of 1
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
<forked VM failed with exit code 1>

Here is the relevant part of the pom.xml:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <version>1.4.0</version>
    <configuration>
        <mainClass>my.Benchmark</mainClass>
    </configuration>
</plugin>

And here is how I run JMH from my.Benchmark:

public static void main(String[] args) throws RunnerException {
    Options options = new OptionsBuilder().include(my.Benchmark.class.getSimpleName())
            .forks(1).build();
    new Runner(options).run();
}

I realize that JMH uses the java.class.path system property to determine the classpath for the forked VMs and that this property does not contain Maven's project dependencies. But what is the preferred way to deal with this?

Trek

Answer 1

While my previous answer requires modifying the benchmark program, here is a POM-only solution that sets the java.class.path system property to the runtime classpath with the help of the Dependency Plugin:

<plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>build-classpath</id>
            <goals>
                <goal>build-classpath</goal>
            </goals>
            <configuration>
                <includeScope>runtime</includeScope>
                <outputProperty>depClasspath</outputProperty>
            </configuration>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <configuration>
        <mainClass>my.Benchmark</mainClass>
        <systemProperties>
            <systemProperty>
                <key>java.class.path</key>
                <value>${project.build.outputDirectory}${path.separator}${depClasspath}</value>
            </systemProperty>
        </systemProperties>
    </configuration>
</plugin>

Trek

Answer 2

One way to work around this problem is to extract the "effective" classpath from the class loader of the my.Benchmark class before calling JMH from my main method:

URLClassLoader classLoader = (URLClassLoader) my.Benchmark.class.getClassLoader();
StringBuilder classpath = new StringBuilder();
for(URL url : classLoader.getURLs())
    classpath.append(url.getPath()).append(File.pathSeparator);
System.setProperty("java.class.path", classpath.toString());

This seems to work, but it feels a lot like a hack that shouldn't be necessary...

Trek