JMH measurement iterations

Question

I'm using JMH and I find something hard to understand: I have one method annotated with @Benchmark and I set measurementIterations(3). The method is called 3 times, but within each iteration call, the function runs a rather big and random number of times.

My question is: is that number completely random? Is there a way to control it and determine how many times should the function run within an iteration? And what is the importance with set up the measurementIterations if each way or another, the function will run a random number of times?

Velasquez

Answer 1

measurementIterations defines how many measured iterations you want to measure of the benchmark. I don't know which parameters you have specified but by default JMH runs the benchmark time-based (default I guess 1 second). This means the benchmark method is invoked in that time frame as often as possible. There are possibilities to specify how often the method should be called in one iteration (-> batching).

I would recommend to study the JMH Samples provided by JMH: http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/ They are a very good introduction into JMH and cover pitfalls you easily make within benchmarks.

Pierson

Answer 2

The number of iteration depends on the various JMH modes I think you must be using Avgtime mode it will perform various iterations. /////////////////////////////////////////////////////////////////////////////////

Mode.Throughput:    Calculate number of operations in a time unit.

Mode.AverageTime:   Calculate an average running time.

Mode.SampleTime:    Calculate how long does it take for a method to run
(including percentiles). 

Mode.SingleShotTime:    Just runs a method
once (useful for cold-testing mode).

////////////////////////////////////////////////////////////////////////////////
For example Use mode "Mode.SingleShotTime", it will perform iteration exactly the number of times you mentioned in the run(see below). // Example runner class

public static void main(String[] args) throws RunnerException {
Options opt = new OptionsBuilder()
                .include(JMHSample_01_HelloWorld.class.getSimpleName())
                .warmupIterations(1)// number of times the warmup iteration should take place
                .measurementIterations(1)//number of times the actual iteration should take place
                .forks(1)
                .shouldDoGC(true) 
                .build();


        new Runner(opt).run(); 
}

Bragi

Answer 3

Yes, in every iteration, the times for method running is random(it's the max number of times the method can run). The times is not important. What is important is the average time used each time.

Besides, you can control how many iterations to run with measurementIterations() and the duration of every iteration with measurementTime().

For example, if you want to run you method with only 1 iteration and it's duration to 1ms, without warmup, just set warmupIterations to 0, measurementTime to 1ms, measurementIterations to 1. Like below:

    Options opt = new OptionsBuilder()
            .include(xxx.class.getSimpleName())
            .warmupIterations(0)
            .measurementTime(TimeValue.milliseconds(1))
            .measurementIterations(1)
            .forks(1)
            .build();

Significance for mutiple iterations: Run more, the results should be more reliable.

Wordsmith

Answer 4

JMH is doing warm-up iterations that are not measured but necessary for valid results.

measurementIterations defines how many iterations should be measured. This does not include warm-up, because warm-up is not measured.

Glomeration