Is it possible to run a JMH benchmark under an external profiler?

Question

I'm currently performance hunting. In order to measure the throughput, and to enforce that we don't regress, I'm using the wonderful JMH.

When I come across something that is slow though, I want to start profiling to see what is going on, according to this link the author of JMH writes:

While JMH profilers can provide the aid in analyzing, I don't think they are the substitute for proper profiling. E.g. the "stack" profiler is good to glance over the profiles, but not for the serious work.

Run the workload for longer, and attach your profiler of choice to the running VM.

I was secretly hoping I could tweak the JMH test from the command line, then attach something like visualvm to that... but so far I've been able to get that to work. I guess that is a bad idea? It would be ideal if I could share the code I use to profile, as well as the code I use to enforce.

Moniliform

Although the stack profiler isn't very good, the perf (or perfasm) profiler is much better for finding out where the hotspots are. Though in microbenchmarks you are going to see the effects of a small part of your code, and in any case, the perf/asm profiler is subject to skid (the CPU will report that the timings are perhaps a few instructions out from the real hotspot). Note that perfm/asm is valid for Linux only. – Mancuso

Answer 1

Yes it is possible, it is simple to do, and I recommend it as a best practice to do it all the time! It is always good to have data available to explain what is going on without the need to spend extra time to reproduce the issue.

I wrote my own JMH profiler to use JFR (Java Flight Recorder): JmhFlightRecorderProfiler which will start JMH profiling and save the data to a file when your benchmark ends. You can open this file with Java Mission control shipped with the JDK.

Here is how you can start your benchmarks and use it (complete example):

Options opt = new OptionsBuilder()
        .include(".*")
        .addProfiler(JmhFlightRecorderProfiler.class)
        .jvmArgs("-Xmx256m", "-Xms256m", "-XX:+UnlockCommercialFeatures",
                "-Djmh.stack.profiles=" + destinationFolder,
                "-Djmh.executor=FJP",
                "-Djmh.fr.options=defaultrecording=true,settings=" + profile)
        .result(destinationFolder + "/" + "benchmarkResults.csv")
        .resultFormat(ResultFormatType.CSV)
        .warmupIterations(10)
        .measurementIterations(10)
        .forks(1)
        .build();
 new Runner(opt).run();

I run benchmarks during the standard build, I use Jenkins JMH plugin to monitor/trend benchmark result/build and if any degradation is observed I have the Benchmark profiles available to investigate the potential cause.

Neighbor

Is there any way to not run flight recorder on warmup iterations? – Penelope

@Neighbor the link to (complete example) is broken – Byram

@Byram thanks for reporting the broken link, fixed. – Neighbor

@Penelope it might be possible I haven't investigated this, however you can use a stack sampling profiler to get profile information broken down by iteration. (see for how that can help in a perf investigation) – Neighbor

@Penelope take a look at a new JMH profiler integration I just wrote at, it will save separate jfr profiles for warmup and measurements. See example on how to use it, – Neighbor

Answer 2

It is not a bad idea. The actual way to attach depends on what the external profiler supports.

• If profiler can launch the Java process under profiling (e.g. like Solaris Studio Performance Analyzer's collect), then executing my-awesome-profiler java -jar benchmarks.jar would get you half-way there. Note, however, that JMH forks out the new process where the workload is run. Make sure the profiler is able to catch the child processes.

• If profiler attaches to a running process, then find a child process and attach to it. The process would have a telling name, like ...ForkedMain. VisualVM should be able to list it in running JVMs list, and attach to it without problems. You can jps to have a PID to attach to for other profilers. The caveat is to make the run longer, so you will have more time to attach. Increase iteration time/count to do that.

Embedded JMH profilers are good in the way they know exactly when to attach (plus, what warmup data to ignore).

Duodenum

Answer 3

You can run with YourKit profiling agent by adding "-agentpath" JVM option. Please find details here https://www.yourkit.com/docs/java/help/agent.jsp

Environmentalist