How can I use JMH for Scala benchmarks together with sbt?

Question

I have tried to use jmh together with sbt, but so far I have not managed to set it up properly so that .scala based benchmarks work.

As the combination sbt + .java based benchmarks works, I tried to start from that base. I am using sbt 0.13.1.

---

.java based benchmarks using sbt

build.sbt

import AssemblyKeys._

name := "scala-benchmark"

version := "1.0"

scalaVersion := "2.10.3"

scalacOptions += "-deprecation"

libraryDependencies += "org.openjdk.jmh" % "jmh-core" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-java-benchmark-archetype" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-generator-annprocess" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-generator-bytecode" % "0.5.5"

assemblySettings

jarName in assembly := "microbenchmarks.jar"

test in assembly := {}

mainClass in assembly := Some("org.openjdk.jmh.Main")

To get a single "fat" jar at the end, the sbt-assembly plugin is required:

project/assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

A simple benchmark:

src/main/java/app/benchmark/java/benchmark2/Benchmark2.java

package app.benchmark.java.benchmark2;

import org.openjdk.jmh.annotations.GenerateMicroBenchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class Benchmark2 {
    @GenerateMicroBenchmark
    public int run() {
        int result = 0;
        for (int i = 0; i < 10; i++) {
            result += i * i;
        }
        return result;
    }
}

Running sbt assembly gives this output:

$ sbt assembly
[...]
[info] Compiling 2 Scala sources and 2 Java sources to ...
[warn] warning: Supported source version 'RELEASE_6' from annotation processor 'org.openjdk.jmh.generators.GenerateMicroBenchmarkProcessor' less than -source '1.8'
[warn] 1 warning
[info] Including: jmh-java-benchmark-archetype-0.5.5.jar
[info] Including: jmh-generator-bytecode-0.5.5.jar
[info] Including: jopt-simple-4.6.jar
[info] Including: jmh-generator-reflection-0.5.5.jar
[info] Including: jmh-generator-annprocess-0.5.5.jar
[info] Including: asm-4.2.jar
[info] Including: commons-math3-3.2.jar
[info] Including: jmh-core-0.5.5.jar
[info] Including: scala-library.jar
[...]
[info] Packaging /home/scala-2.10/vc/rhaag/scala/scala-benchmark/target/scala-2.10/microbenchmarks.jar ...

and the resulting microbenchmarks.jar contains everything required to run the benchmarks:

$ java -jar target/scala-2.10/microbenchmarks.jar -wi 3 -i 3 -f 1 app.benchmark.java.benchmark2.Benchmark2.run 

[...] 

Benchmark                  Mode   Samples         Mean   Mean error    Units 

a.b.j.b.Benchmark2.run    thrpt         3   607555,968    70243,275   ops/ms 

So far so good.

---

Scala benchmarks using sbt

From that base I tried to switch to .scala based benchmarks:

build.sbt

Replacing the Java archetype with the Scala one

libraryDependencies += "org.openjdk.jmh" % "jmh-scala-benchmark-archetype" % "0.5.5"

does not work, as the download fails.

This works:

libraryDependencies += "org.openjdk.jmh" % "jmh-scala-benchmark-archetype" % "0.5.5" from "http://repo1.maven.org/maven2/org/openjdk/jmh/jmh-scala-benchmark-archetype/0.5.5/jmh-scala-benchmark-archetype-0.5.5.jar"

Another simple benchmark:

src/main/scala/app/benchmark/scala/benchmark2/Benchmark2.scala

package app.benchmark.scala.benchmark2

import org.openjdk.jmh.annotations.GenerateMicroBenchmark
import org.openjdk.jmh.runner.Runner
import org.openjdk.jmh.runner.RunnerException
import org.openjdk.jmh.runner.options.Options
import org.openjdk.jmh.runner.options.OptionsBuilder

class Benchmark2 {
  @GenerateMicroBenchmark
  def run() = {
    Seq.range(0, 10).map(i => i * i).sum
  }
}

Now sbt assembly creates the jar file, but target/scala-2.10/microbenchmarks.jar#META-INF/MicroBenchmarks does not list the Scala benchmarks, and these are not shown by java -jar target/scala-2.10/microbenchmarks.jar -l either.

---

Resources:

• JMH with maven (there is a link to an ant based approach as well)
• A Japanese page where I got the inital sbt setup from
• JMH with gradle

How can I integrate the (bytecode based) JMH processor for Scala? Or from another perspective: Why is the (annotation based) JMH processor picked up automatically and produces Java based benchmarks?

Weese

Including archetype dependencies is futile, those are just the project templates. – Roily

I don't know how SBT works internally, but here's how Maven compiles JMH benchmarks with Scala: hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-archetypes/…. You might want to replicate it in SBT :) – Roily

hmmm, @GenerateMicroBenchmark seems to be ignored in .scala files. I looked at the Japanese page and the work around he did was to put the code to be benchmarked in a separate scala file and call it from a Java method which has the annotation assigned. No idea other than that why SBT is not picking it up. – Sparke

We discussed that with Jamie Allen before, and figured SBT does not process Java annotations. That is when JMH bytecode generator was born, and JMH Scala archetypes use that generator to parse out the bytecode from scalac, and generate the JMH benchmarks. The link above outlines what it takes in Maven. – Roily

@AlekseyShipilev, thanks for that info. I will try to get around to writing a task (maybe even a plugin?) to generate the JMH benchmarks via sbt. – Sparke

Answer 1

I have implemented an sbt-jmh plugin that actually works: https://github.com/ktoso/sbt-jmh

Currently building benchmarks.jar is not supported, but you can simply type run -i 10 .*MyBenchmark.* and it will work as expected (doing all the multi-step compilation for you).

I hope this helps, cheers!

Foyer

Thanks a lot. I've tried it, but even in the example project, run does not find any benchmarks (sbt 0.13.2, Scala 2.11.3). During run the line Processing 0 classes from /tmp/example-project/target/classes looks odd, and the directory target/generated-sources/jmh is empty (= it does not exist). – Weese

Right, sorry - I'm working on updating the project. But it's working as part of the akka build if you want to have a look: github.com/akka/akka/commit/… – Yeanling

No problem at all, thanks for your effort, there's no hurry :-) Just drop another comment, if it's ready for prime time on your plugin page. – Weese

I've added scripted tests and updated the example project over the weekend. Go ahead and give it a shot (0.1.3)! – Yeanling

Sorry for the late answer, I was on vacation. I have tested it now using 0.1.4, and it works. The only minor issue is that it fails if there is a Java-based benchmark in the same sbt project with an IllegalArgumentException in org.objectweb.asm.ClassReader/org.openjdk.jmh.generators.bytecode.ASMGeneratorSource.processClass(ASMGeneratorSource.java:65). Thanks for your work – Weese

I am quite interested in this.. trying it out. I will update here after my attempt. – Tauten

Thanks @Weese for letting me know! I'll open an issue and try solving this. Would be really interesting because of scala vs. java benchmarks (though these tend to go horribly "wrong" in terms of apples vs. oranges). – Yeanling

@Konrad'ktoso'Malawski Exactly, that's very often my use case as well – Weese