How do you specify the JMH microbenchmarks test to run WITHOUT using a resource file?

Question

jmh 0.6. I have jmh-core, jmh-generator-annprocess, jmh-generator-reflection as dependencies.

First, documentation is poor, unfortunately. For one, I use gradle, not maven, so using the maven archetype is not an option.

Second, I want to use the Java API, not the command line.

My very simple code is that:

public final class TestBenchmark
{
    private static final int COUNT = 100_000;
    private static final List<Integer> LIST = new ArrayList<>();

    static {
        for (int i = 0; i < COUNT; i++)
            LIST.add(i);
    }

    @GenerateMicroBenchmark
    public void foreachLoop()
    {
        int dummy;
        for (final int i: LIST)
            dummy = i;
    }

    @GenerateMicroBenchmark
    public void forLoop()
    {
        int dummy;
        final int size = LIST.size();
        for (int i = 0; i < size; i++)
            dummy = LIST.get(i);
    }

    public static void main(final String... args)
        throws RunnerException
    {
        final Options options = new OptionsBuilder()
            .forks(1)
            .warmupIterations(1)
            .measurementIterations(20)
            .verbosity(VerboseMode.EXTRA)
            .build();

        new Runner(options).run();
    }
}

As I have no .include(), it means .* as a regex therefore all benchmarks. This is the only class I have in my project.

But no: "no benchmarks found".

So, as a last resort I tried and created the META-INF/MicroBenchmarks file as suggested in other places; content, the name of the class:

com.github.parboiled1.grappa.TestBenchmark

but it doesn't work either:

Exception in thread "main" java.lang.IllegalStateException: Mismatched format for the line: com.github.parboiled1.grappa.TestBenchmark

and the format of this file is, of course, not documented.

But I don't want to use this file to start with; I want to specify the list of classes to run.

How do I do that?

Sabir

Answer 1

Your problem is not about mismatched benchmarks, but rather the build misconfiguration which prevented JMH to generate and compile the synthetic benchmark code for you. You have to let annotation processors to run, and let the rest of the build to pack the generated source and resources into the complete bundle, even if you are to use API afterwards. The project produced by Maven archetype does just that. There is also the Ant sample which does specify exactly what should happen with other build systems. If you want to to that with Gradle, try this.

P.S. In JMH, META-INF/MicroBenchmarks is an internal API, and it may change without notice. No surprises you are not able to use it.

Kristiankristiansand

But what I really want is to run main()s with benchmarks in them; I don't want to have to generate a jar and run the benchmarks from it... – Sabir

@fge: You still have to compile the benchmark and generate the resources with JMH. IDEs are able to run main() methods like yours with a properly set up project. (E.g. opening up the Maven project generated from archetype and running the main() from there works at least for IDEA and NetBeans). – Kristiankristiansand

Well, precisely; I want to do that the programmatical way but there is zero documentation on how to do that! – Sabir

What do you mean by "zero documentation"? JMH page says that archetypes provide the golden build configuration, plus there is an Ant build sample which describes what processes are happening during the benchmark build -- if you choose to ignore that, it's not the documentation fault :) Have you tried generating the simple JMH project and doing the main() from there? – Kristiankristiansand

I didn't ignore that, I tried and read as much as I could, as deep as I could to no avail: I don't see how to do it without ant and maven, since I use neither; and as I said, I want to do that without having to issue a task build using x, where x is a build system. – Sabir

You can't do this without a build system, because you need several build steps to succeed! This is why you have to have either Maven, or Ant, or Gradle to do this for you. You can lure the IDE into doing the same job silently, but you have to show IDE the build script beforehand. Maybe I am missing what exactly you want: is that "put the vanilla Java source in my IDE, [configure the build], and go to running main"? – Kristiankristiansand

I just used the single command from JMH page to generate the project from the archetype, fired up IntelliJ IDEA, pointed to my project (well, on to pom.xml actually), put your benchmark into the project, right-clicked main() and said Run. Worked flawlessly (modulo the compiler errors since your original code is not compilable under JMH). – Kristiankristiansand

These build steps I want to do in pure Java! There are all the needed tools for that -- the compiler API etc. You can run annotation processors programmatically as well. So, it's far from being impossible. But even though I have dug deep into the source I cannot begin to find where to start! – Sabir

Okay. And who exactly will invoke those "pure Java" tools? And when? You should probably eliminate JMH from the equation for a time being, and try a simple project which compiles the source code and runs your own annotation processors over it. I think that would be an enlightening exercise. After you are done with that, get the Ant sample from JMH, and redo it in "plain Java". Not sure if that is the problem needing the solution though -- it smells like winding up an ad hoc build system when standard ones are available. – Kristiankristiansand

I already did source code generation and compilation in Java and even class loading, so that is not news to me. I have links on how to run annotation processors as well. – Sabir

Good, carry on. I assume you have a good reason for winding up the adhoc solution. If you have specific JMH internals questions, you are welcome at mail.openjdk.java.net/mailman/listinfo/jmh-dev – Kristiankristiansand

Answer 2

I am using Maven so this is a Maven solution but I am sure you can adapt it to Gradle if you follow the steps. I have to admit that the greatness of JMH is overshadowed by the lack of documentation.

I couldn't make it run properly at first so at the end I created a JUnit test cases that starts the benchmark. No main for me as it is easier for Jenkins and I don't have a jar project created.

1. You have execute the APT processor either by code or from a build script. In Maven it is working out of the box.
2. You can verify the success if JMH source code is generated.
3. Now the JMH source code needs to be compiled. Maven is doing out of the box. Not sure about Gradle.
4. Verify that your class files exist.
5. Now you can execute the main method.

This profile tells maven to add *Benchmark.java to surefire execution.

<profile>
  <id>benchmark</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <includes>
            <include>**/*Benchmark.java</include>
          </includes>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>

when I run Maven with profile benchmark. Then Benchmarks are executed.

Flatways