How to use JMH with gradle?

Question

I want to use JMH, an OpenJDK microbenchmark tool, with gradle. However, Im getting the NPE on compilation. On the other hand, JMH works when using from maven.

I am not posting any build.gradle as it is basic - apply java plugin and add dependency on JHM tool (org.openjdk.jmh:jmh-core:0.2).

I have tried whats written here without success.

What else I have to do? I think something with setting the agent, but I still didnt figure it out.

Exception:

:compileJava
java.lang.NullPointerException
at org.openjdk.jmh.processor.internal.GenerateMicroBenchmarkProcessor.validMethodSignature(GenerateMicroBenchmarkProcessor.java:502)

Otway

Answer 1

Just finished my "masterpiece". No uber-jars, no plugins, code base separated from main & test, benchmarks compilation hooked to main, but does not run automatically in the mainstream lifecycle. Simple, explicit, and hackable, vanilla gradle.

I run it directly from IntelliJ, to run on a box you probably will need the uber-jar back :-)

Before doing it I have spent a fair amount of time trying to get that plugin work, but it's way too clunky for my taste.

Step-by-step breakdown below.

Define a new sourceSet called jmh with classpath hooked to that of the main sourceSet

sourceSets {
    jmh {
        java.srcDirs = ['src/jmh/java']
        scala.srcDirs = ['src/jmh/scala']
        resources.srcDirs = ['src/jmh/resources']
        compileClasspath += sourceSets.main.runtimeClasspath
    }
}

Define dependencies for it (at minimum JMH and its annotation processor).

dependencies {
    ...
    jmhImplementation 'org.openjdk.jmh:jmh-core:1.35'
    jmhImplementation 'org.openjdk.jmh:jmh-generator-annprocess:1.35'
}

Define a task jmh of type JavaExec

task jmh(type: JavaExec, dependsOn: jmhClasses) {
    main = 'org.openjdk.jmh.Main'
    classpath = sourceSets.jmh.compileClasspath + sourceSets.jmh.runtimeClasspath
}

Hook jmhClasses task to run after classes to make sure benchmarks are compiled with the rest of the code

classes.finalizedBy(jmhClasses)

Nonperformance

So simple. Perfect! One thing that I ran across: you need to ensure that you've configured IntelliJ to use separate module per source set, otherwise it will not find your JMH source sets. – Jaffa

I would only add the requirement of jmhAnnotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.35' in the dependencies. – Ventricular

Answer 2

Currently you can just use dedicated plugin jmh-gradle-plugin

It requires minimal configuration and allows you to run JMH benchmarks as well as build benchmarks artifact

Ataxia

no possibility to run individual tests, no possibility to use separate jvm_args per test, etc. If you just want to "play" with JMH - sure, if you plan to be serious about it - this is a big NOGO. – Gluttonous

It looks like the plugin now can be configured to run individual tests, use separate JVM args etc. See the configuration options. – Edmondo

In addition to the configuration options (described above), it's also possible to tweak a lot of these things through JMH's annotations. You can also run ./gradlew jmhJar in your project and then use the JMH cli as you would with a JMH jar file built with maven. e.g. java -jar build/libs/my-project-0.0.1-SNAPSHOT-jmh.jar MySpecificBenchmarkClass $insertYourFavoriteOptionsHere. Seems to me the gradle plugin has all the bells & whistles a maven project would. – Eboh

Answer 3

My bad, I was trying to benchmark a method that has an argument - of course JMH will not know what to pass :) Once when I created a void method with no arguments, everything worked.

My build.gradle:

defaultTasks 'build'

apply plugin: 'java'
apply plugin: 'shadow'

buildscript {
    repositories {
        mavenCentral()
        maven {
            name 'Shadow'
            url 'http://dl.bintray.com/content/johnrengelman/gradle-plugins'
        }
    }
    dependencies {
        classpath 'org.gradle.plugins:shadow:0.7.4'
    }
}

jar {
    manifest {
        attributes 'Main-Class': 'org.openjdk.jmh.Main'
    }
}

repositories {
    mavenCentral()
}


build.doLast {
    tasks.shadow.execute()
}

shadow {
    outputFile = new File('build/libs/microbenchmarks.jar')
}

ext {
    lib = [
        ... other dependencies...
        jmh:            'org.openjdk.jmh:jmh-core:0.2'
    ]
}

dependencies {
    compile lib... other dependencies...
    compile lib.jmh
}

sourceCompatibility = 1.7

Build tests and jar:

gw clean build

and then run them with:

java -jar build/libs/microbenchmarks.jar ".*" -wi 2 -i 10 -f 2 -t 16

UPDATE

From recent versions of JMH, you would also need to add dependency to:

org.openjdk.jmh:jmh-generator-annprocess:0.5.4

and you can use shadow 0.8.

Otway

Yeah, but also JMH should have printed more sane error message. Can you try to feed the erroneous benchmark you had before to 1.0-SNAPSHOT version of jmh (requires building from source)? I have rebuilt the significant part of the validation code there. – Albric

Yeah, NPE was confusing to me, I thought I did something wrong. Sure, I will try 1.0-SNAPSHOT but have first to prepare everything with online repo, for other ppl. After that I can play with local. – Otway

Answer 4

I made a very small example project to clone and modify as you like. It's a full working example:
https://gitlab.com/barfuin/jmh-gradle-example

It requires no shadow Jars and no plugins, while still running the benchmark in a dedicated JVM. The project also includes some extra Gradle tasks for printing the classpath, the JMH options, etc., stuff that may help to understand what's going on.

Leninist

It's a lot of boilerplate that can be avoided by using the JMH Gradle plugin. – Teddman

It's quite lean, no plugin, not much code either. Take a look! – Leninist

I’ve already looked at it, and have come to the conclusion posted above. IMO, there’s no need to reinvent the wheel because “no plugin” isn’t necessarily an advantage here. – Teddman

Well, 9 lines of code achieve the result, without a plugin. – Leninist

Answer 5

If you're an IntelliJ user, perhaps the easiest way to make it work, without all that workarounds, is to use the IDE plugin:

https://github.com/artyushov/idea-jmh-plugin

• Add the dependencies
• Create your benchmark
• be happy :)

Sherrilsherrill