Benchmarking spring boot application with JMH

Question

I have a spring boot application which I want to benchmark using JMH. Any reference for this integration will be useful.

Lieutenant

IMHO jmh isn't suitable for benchmarking applications but only for particular methods. – Sculptor

I manged to do bechmarking for spring MVC application.They should be way to do it for Spring-boot apps – Lieutenant

spring-cloud-sleuth benchmark samples – Connotation

Answer 1

The solution was quite than easy than I thought. The important part is to start the spring-boot application when the benchmark is getting initialized. Define a class level variable for configuration context and give a reference to it during setup of the benchmark. Make a call to the bean method inside the benchmark.

import java.io.File;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import org.springframework.boot.SpringApplication;
import org.springframework.context.ConfigurableApplicationContext;

@BenchmarkMode(Mode.Throughput) @OutputTimeUnit(TimeUnit.MINUTES)
@State(Scope.Thread)
public class ProcessFeedBenchMark   {

    public static void main(String args[]) throws Exception {
        URLClassLoader classLoader = (URLClassLoader) ProcessFeedBenchMark.class.getClassLoader();
        StringBuilder classpath = new StringBuilder();
        for(URL url : classLoader.getURLs())
            classpath.append(url.getPath()).append(File.pathSeparator);
        classpath.append("/D:/work/zymespace/benchmark/src/main/resources/").append(File.pathSeparator);
        System.out.print(classpath.toString());
        System.setProperty("java.class.path", classpath.toString());

        Options opt = new OptionsBuilder()
        .include(ProcessFeedBenchMark.class.getName() + ".*")
        .timeUnit(TimeUnit.MILLISECONDS)
        .threads(1)

        .shouldFailOnError(true)
        .shouldDoGC(true)
        .build();

        new Runner(opt).run();
    }
    static ConfigurableApplicationContext context;

    private BenchmarkTestService service;

    @Setup (Level.Trial) 
    public synchronized void  initialize() {
        try {
            String args = "";
            if(context == null) {
                context = SpringApplication.run(BenchmarkSpringBootStater.class, args );
            }
            service = context.getBean(BenchmarkTestService.class);
            System.out.println(service);
        } catch(Exception e) {
            e.printStackTrace();
        }
    }

    @Benchmark 
    public void benchmark1 (ProcessFeedBenchMark state, Blackhole bh) {
        try {
            service.li();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Lieutenant

I would not rely on a number that you get out of this benchmark. You probably rely on the service configuration and any proxy mechanism. Also, most of the time spend will be for printing the result. Rather, you should return the service value from the method and let JMH handle the escape. This is not how you should use JMH. – Dismast

The System.out.println statement is removed to get the right benchmark for li() method. – Lieutenant

@SoumyajitSwain could you please tell me how you managed to include spring boot fat jar as dependency of JMH project? – Lutestring

@VolodymyrBakhmatiuk - This line does the trick. System.setProperty("java.class.path", classpath.toString()); – Lieutenant

Answer 2

There is no need to spin up the Spring context to benchmark a method. In fact, I'm not sure what are you really trying to benchmark here as the method implementation isn't changing. If all you want is to time the method, it's a whole lot simpler and less error-prone to call the method directly. If you want to benchmark the startup time under varying circumstances, see my answer here.

Headcheese

If you have references to spring beans within your method, it is not possible to directly call the method, otherwise there is no need as you mentioned. – Tasset