JMH Unable to find the resource: /META-INF/BenchmarkList

Question

I'm not able to run simple JMH benchmark inside eclipse. Maven dependencies:

            <dependency>
                <groupId>org.openjdk.jmh</groupId>
                <artifactId>jmh-core</artifactId>
                <version>1.12</version>
            </dependency>
            <dependency>
                <groupId>org.openjdk.jmh</groupId>
                <artifactId>jmh-generator-annprocess</artifactId>
                <version>1.12</version>
            </dependency>

Java code:

    public class BTest {
        @Benchmark
        public void test() {
            // todo
        }
    
        public static void main(String[] args) throws RunnerException {
            Options opt = new OptionsBuilder()
                     .include(BTest.class.getSimpleName())
                      .build();
    
            new Runner(opt).run();
        }
    }

Result of run:

> Exception in thread "main" java.lang.RuntimeException: ERROR: Unable
> to find the resource: /META-INF/BenchmarkList     at
> org.openjdk.jmh.runner.AbstractResourceReader.getReaders(AbstractResourceReader.java:96)
>   at org.openjdk.jmh.runner.BenchmarkList.find(BenchmarkList.java:104)
>   at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:256)   at
> org.openjdk.jmh.runner.Runner.run(Runner.java:206)    at
> com.test.BTest.main(BTest.java:24)

Maybe the problem is, that I'm running it from eclipse.

Lacasse

Have you tried not running it in eclipse? Since you mentioned it, it would make sense to try it – Tichon

@WillBarnwell not yet... – Lacasse

here may be an answer for idea – Slipcover

@Slipcover not working – Emelia

Does anyone know if I can benchmark a springboot controller and get the benchmarks by firing the API that hits the controller and getting the benchmarks in the output? – Halliday

Answer 1

Finally found it out. There was a problem with missing exec-maven-plugin plugin

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>run-benchmarks</id>
            <phase>integration-test</phase>
            <goals>
                <goal>exec</goal>
            </goals>
            <configuration>
                <classpathScope>test</classpathScope>
                <executable>java</executable>
                <arguments>
                    <argument>-classpath</argument>
                    <classpath />
                    <argument>org.openjdk.jmh.Main</argument>
                    <argument>.*</argument>
                </arguments>
            </configuration>
        </execution>
    </executions>
</plugin>

Lacasse

Weirdly, for Scala + Maven, I had to add this in to even get the error you have above ... : github.com/bbarker/scala-maven-jmh-bench-demo/tree/… – Orvie

Answer 2

If anyone is using Gradle, add the jmh-gradle-plugin to your plugins block:

plugins {
    id 'java'
    id 'me.champeau.jmh' version '0.6.8'
}

Then add all of the following in your dependencies block (check the latest version of JMH on Maven here):

dependencies {
    jmh 'org.openjdk.jmh:jmh-core:1.36'
    jmh 'org.openjdk.jmh:jmh-generator-annprocess:1.36'

    // this is the line that solves the missing /META-INF/BenchmarkList error
    jmhAnnotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.36'
}

Then just use the following to run your benchmarks through Gradle:

./gradlew jmh

Running through your IDE

If you also want to run the benchmarks from within your IDE instead of through Gradle, you can do either of the following:

Option 1 - main method

Just use a main method with no additional configuration and your IDE will respect your annotation configuration:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
@Fork(value = 1)
@Warmup(iterations = 5, timeUnit = TimeUnit.MILLISECONDS, time = 5000)
@Measurement(iterations = 5, timeUnit = TimeUnit.MILLISECONDS, time = 5000)
public class MyBenchmark {

    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
            .include(MyBenchmark.class.getSimpleName())
            .build();
        new Runner(options).run();
    }

    // benchmarks omitted
}

Option 2 - Install JMH Plugin

If you're using IntelliJ, install the JMH Java Microharness Benchmark Plugin from the Preferences > Plugins section, then you can omit your main method altogether and IntelliJ will give you a run button next to your class name:

Windblown

Why doesn't the plugin just set these things for you? – Belittle

jmhAnnotationProcessor is not required. If fact, none of the things shown in the dependencies block are required unless you want to change the version. – Crack

As opposed to previous comment, without those dependencies, the JMH plugin for intelliJ produces the errors listed in the question – Item

Answer 3

pom.xml must have the below dependencies and configurations to Java Micro-benchmark Harness (JMH) Framework

<properties>
    <jmh.version>1.21</jmh.version>
</properties>
<dependencies>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>${jmh.version}</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>${jmh.version}</version>
</dependency>
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>4.11</version>
  <scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>java-jmh</finalName>
<plugins>
    <plugin>    
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
            <source>1.8</source>
            <target>1.8</target>
            <annotationProcessorPaths>
                <path>
                    <groupId>org.openjdk.jmh</groupId>
                    <artifactId>jmh-generator-annprocess</artifactId>
                    <version>${jmh.version}</version>
                </path>
            </annotationProcessorPaths>
        </configuration>
    </plugin>
    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <executions>
            <execution>
                <id>run-benchmarks</id>
                <phase>integration-test</phase>
                <goals>
                    <goal>exec</goal>
                </goals>
                <configuration>
                    <classpathScope>test</classpathScope>
                    <executable>java</executable>
                    <arguments>
                        <argument>-classpath</argument>
                        <classpath />
                        <argument>org.openjdk.jmh.Main</argument>
                        <argument>.*</argument>
                    </arguments>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

After this go to command line and run the command $mvn clean install

Rothenberg

Answer 4

I realized that I already had exec-maven-plugin in my parent pom as mentioned in expected answer but I had to run mvn clean install as mentioned in https://stackoverflow.com/a/40748670 to fix the error

Tarrasa

Answer 5

This could happen when your compiler plugin has not processed the JMH related annotations. For me, Gill's answer with the maven-compiler-plugin's <annotationProcessorPaths> update worked.

Martinmartina

Answer 6

Having had the same error; and running the tests from maven or intellij didn't work. I realised that the problem was that I wrote the benchmark in Kotlin. Changing the code to java sorted the issue.

Goldi

I banged my head against this error for hours trying to write a @Benchmark in Kotlin to no avail. Once I flipped to Java it all worked. – Mooring

Answer 7

Add version as well. This works for me

        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.0.0</version>

Winifield

Answer 8

If you are getting this error using the Intelij Idea's JMH Java Microharness Benchmark Plugin and you also using Gradle try changing the default running task from Gradle to Intelij Idea. To do this:

Settings -> Build, Execution, Deployment -> Build Tools -> Gradle

Change Build and Run using to Intelij IDEA

Landrum

Answer 9

I tried all the above solution. I ended using jmh-maven-plugin:

<plugin>
    <groupId>pw.krejci</groupId>
    <artifactId>jmh-maven-plugin</artifactId>
    <version>0.2.2</version>
</plugin>

and I run it with:

mvn clean install jmh:benchmark

It seems that it does not require exec-maven-plugin, maven-compiler-plugin or maven-shade-plugin.

Notice that the benchmarks must be in the test package.

Hendrix

Answer 10

Here's everything I had to add to pom.xml fix mine.
This snippet ended up pretty close to Ishaq's answer minus a few extras and name-specifics.
I hope this is just a cleaner and easier to readily-use answer.

  <properties>
    <jmh.version>1.36</jmh.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.openjdk.jmh</groupId>
      <artifactId>jmh-core</artifactId>
      <version>${jmh.version}</version>
    </dependency>
    <dependency>
      <groupId>org.openjdk.jmh</groupId>
      <artifactId>jmh-generator-annprocess</artifactId>
      <version>${jmh.version}</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <finalName>benchmarks</finalName>
              <transformers>
                <transformer
                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>org.openjdk.jmh.Main</mainClass>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Abrahan

Answer 11

In my case the reason for the Unable to find the resource: /META-INF/BenchmarkList exception was that annotation processing was generally disabled in a parent pom of my project. Without annotation processing, adding the jmh-generator-annprocess dependency doesn't do anything:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <configuration>
    <!-- The line below disables annotation processing: -->
    <compilerArgument>-proc:none</compilerArgument>
  </configuration>
</plugin>

Weinstein