Using jmh to benchmark code without creating separate maven project

Question

I am working on a Maven project and I wish to use jmh to benchmark my code. I want to organize my project so that it contains the source code, unit tests, and benchmarks. There seems to be a way in gradle to benchmark your code without creating a separate gradle project (see link). Is there a way to do this in Maven?

Freesia

Answer 1

Short answer is yes.

I came across to this directory layout in my projects (but you definitely can change it)

+- src/
   +- main/java   - sources
   +- test/
      +- java        - test sources
      +- perf        - benchmarks

You need a couple of plugins to achieve that.

1. build-helper-maven-plugin to attach custom test sources location
   

<execution>
    <id>add-test-source</id>
    <phase>generate-test-sources</phase>
    <goals>
        <goal>add-test-source</goal>
    </goals>
    <configuration>
        <sources>
            <source>src/test/perf</source>
        </sources>
    </configuration>
</execution>

2. maven-compiler-plugin to run jmh-generator-annprocess annotation processor on test-compile phase
   

<execution>
    <goals>
        <goal>testCompile</goal>
    </goals>

    <configuration>
        <annotationProcessorPaths>
            <path>
                <groupId>org.openjdk.jmh</groupId>
                <artifactId>jmh-generator-annprocess</artifactId>
                <version>${jmh.version}</version>
            </path>
        </annotationProcessorPaths>
    </configuration>
</execution>

3. maven-assembly-plugin to create runnable jar with benchmarks
   

<execution>
    <id>make-assembly</id>
    <phase>package</phase>
    <goals>
        <goal>single</goal>
    </goals>
    <configuration>
        <attach>true</attach>
        <archive>
            <manifest>
                <mainClass>org.openjdk.jmh.Main</mainClass>
            </manifest>
        </archive>
    </configuration>
</execution>

<assembly>
    <id>perf-tests</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <unpack>true</unpack>
            <scope>test</scope>
        </dependencySet>
    </dependencySets>
    <fileSets>
        <fileSet>
            <directory>${project.build.directory}/test-classes</directory>
            <outputDirectory>/</outputDirectory>
            <includes>
                <include>**/*</include>
            </includes>
            <useDefaultExcludes>true</useDefaultExcludes>
        </fileSet>
    </fileSets>
</assembly>

After that you'll get an executable jar with benchmarks which could be run as usual

java -jar target/your-project-version-perf-tests.jar

You can see working example here.

NOTE

The only drawback of this solution is that all test classes and test dependencies will also be included in jar with benchmarks which definitely bloats it up. But you can avoid it by compiling benchmarks in separate (other than ${project.build.directory}/test-classes) directory.

Moureaux

In your working example (i.e., doSomethingTest in ExampleBenchmark.java), it's better to avoid the for loop, as it leads to unintended performance results due to loop optimizations in the JVM. – Model

Having src/test/perf besides src/test/java is a bit inconsistent, since the latter distinguishes the artifacts by technology (java in this case) and the former distinguishes it by purpose (perf). Since the benchmarks are probably written in Java too, I would prefer src/perf/java, as suggested in this article about setting up JMH with Gradle. – Chickenhearted

Where does the <assembly> element go? I don't see it in your working example's pom.xml – Dogcart

Oh, I see it in src/main/assembly/perf-tests.xml – Dogcart