How to generate methods in jmh benchmarks?

Question

I use jmh(http://openjdk.java.net/projects/code-tools/jmh/ ) to benchmark some method. Also I have the set of parameters that I want to use as arguments to run this method. Is it possible to generate a method for the each particular parameter value (with @GenerateMicroBenchmark annotation)?

Now I use the similar implementation, but it isn't so convenient, because of I have to write a lot of uniform code by hand:

interface State {
    int action();
    void prepare();
}

class Method {
    ...;
    toString() { return "State1_" + param; }
}

{
    Method[] states;
    curState = -1;
    count = 0;
    int[] params = {1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000, 12000};
    for (int param: params) {
      states[count++] = new Method(param);
    }
}

@Setup(Level.Invocation)
public void prepareState() {
  if (curState != -1) {
    states[curState].prepare();
  }
}

@GenerateMicroBenchmark
public int param_1000() {
    curState = 0;
    return states[curState].action();
}

@GenerateMicroBenchmark
public int param_2000() {
    curState = 1;
    return states[curState].action();
}

@GenerateMicroBenchmark
public int param_3000() {
    curState = 2;
    return states[curState].action();
}
...
@GenerateMicroBenchmark
public int param_12000() {
    curState = 11;
    return states[curState].action();
}

Seaver

Answer 1

The benchmarks usually don't lend themselves for reuse without some serious breakage. Most attempts I've seen people trying to simplify their benchmarks broke them beyound belief. For example, using arrays here might offset the results for nano-benchmarks (e.g. if action() is small). Level.Invocation is usually a bad idea as well, as stated in its Javadoc.

The bottom line is, just because some usage is allowed by API, it does not necessarily means you should use it. Here is what you should do instead:

@State(Scope.Thread)
class MyBenchmark {
    Method[] states;

    @Setup
    public void setup() {
        int[] params = {1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000, 12000};
        int count = 0;
        for (int param: params) {
            states[count++] = new Method(param);
        }
    }

    int doWork(int idx) {
        states[idx].prepare();
        return states[idx].action();
    }

    @GenerateMicroBenchmark
    public int param_1000() {
       doWork(0);
    }

    ...

    @GenerateMicroBenchmark
    public int param_12000() {
       doWork(11);
    }
}

...or even:

@State(Scope.Thread)
class MyBenchmark {
    Method p1000, p2000, ..., p12000;

    @Setup
    public void setup() {
        p1000 = new Method(p1000);
        p2000 = new Method(p2000);
        ...
        p12000 = new Method(p12000);
    }

    @GenerateMicroBenchmark
    public int param_1000() {
       p1000.prepare();
       return p1000.action();
    }

    ...

    @GenerateMicroBenchmark
    public int param_12000() {
       p12000.prepare();
       return p12000.action();
    }
}

The alternative would be accepting the parameter from outside, and using Java API to juggle the parameter. For the sake of example:

@State(Scope.Thread)
class MyBenchmark {
    final int param = Integer.getInteger("param", 1000); 

    Method m;

    @Setup
    public void setup() {
        m = new Method(param);
    }

    @GenerateMicroBenchmark
    public int work() {
       m.prepare();
       return m.action();
    }

     public static void main(String[] args) throws RunnerException {
         Options opts = new OptionsBuilder()
                 .include(".*")
                 .jvmArgs("-Dparam=2000")
                 .build();

         RunResult runResult = new Runner(opts).runSingle();
         Result result = runResult.getPrimaryResult();

         System.out.println();
         System.out.println("API replied benchmark score: " + result.getScore() + " " + result.getScoreUnit() + " over " + result.getStatistics().getN() + " iterations");
     }
}

Brushoff