How to use Caliper for benchmarking?

Question

I am trying to figure out how to use Caliper to do benchmark testing in Eclipse and I am getting nowhere. I tried following the 26 minute tutorial found here: https://code.google.com/p/caliper/ but I get lost quickly. I have downloaded the Caliper jar file but I'm not sure what folder it should be in. I've also downloaded Maven for Eclipse plugin but I'm not even sure if that is necessary. Is it possible to install Caliper from the 'Install New Software..' option in the Help menu in Eclipse? I just want to be able to do very simple speed tests for some of the algorithms I've created for a Data Structures and Algorithms class I am taking.

Troxler

Are you using Windows or Linux? What version of caliper are you using? – Hargreaves

I am using Windows and I downloaded caliper-1.0-beta-1-all.jar – Troxler

I downloaded the file to my Downloads folder because I wasn't sure where to download it to. – Troxler

Answer 1

This answer is now obsolete. Caliper has worked in Windows for more than a year, at least: https://code.google.com/p/caliper/issues/detail?id=167

---

Caliper doesn't work in Windows. See this case. You need to use version 0.5-rc1, which has other issues but is still pretty okay and is missing a lot of features, but it does work in Windows.

• If you know how to use Maven, add this pom snippet to your pom.xml.

  <dependency>
      <groupId>com.google.caliper</groupId>
      <artifactId>caliper</artifactId>
      <version>0.5-rc1</version>
  </dependency>
  

  • If you want to learn maven, first read this: http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html
  • Convert your project to a maven project (Right click on project -> Configure -> Convert to Maven Project)
• If you don't know how to use Maven (here is a guide to how to do this with pictures):
  • Download the 0.5-rc1 jar
  • Right click on the project you want to use and choose Build Path -> Configure Build Path
  • Add it to your libraries tab using Add External Jar

Once you've done that, you can start writing benchmarks. Here is an example of a benchmark I wrote for a different Stack Overflow question.

Hargreaves

How can I add the pom snippet to my pom.xml ? I have Maven installed in Eclipse, just not sure how to use it. If this is an easy procedure I would like to use it. – Troxler

@Troxler I added a little bit about Maven. Maven is very very very good once you learn it but it's rather complicated and outside the scope of this question. I suggest searching here or googling about how to use Maven and m2eclipse. – Hargreaves

@Hargreaves What about running it? Would you mind adding the section from your pom that does that? I've been using a different question as a basis but haven't had much luck: stackoverflow.com/questions/18405740 – Luxemburg

My mistake wasn't with the above maven pom, it was because I guava was already a dependency and 0.5-rc1 won't work with guava above version 14.0.1 – Luxemburg

@porciesphino they claim the new version works with Windows but I haven't tried it yet code.google.com/p/caliper/issues/detail?id=167 – Hargreaves

Answer 2

Here is how you set up a working Caliper class using the latest version of Caliper as of this writing, caliper-1.0-beta2 . As far as I can tell this procedure isn't documented anywhere outside of inline comments in the Caliper code files.

First install caliper-1.0-beta2 in pom.xml or by downloading the jar file. Then make a file like this:

import com.google.caliper.Benchmark;
import com.google.caliper.runner.CaliperMain;

public class DemoBenchmark {
  public static void main(String[] args) {
    CaliperMain.main(DemoBenchmark.class, args);
  }

  @Benchmark
  void timeStringBuilder(int reps) {
    StringBuilder sb = new StringBuilder();
    for (int i = 0; i < reps; i++) {
      sb.setLength(0);
      sb.append("hello world");
    }
  }
}

Run that file, and Caliper will do the benchmark for you.

Pyrargyrite

Answer 3

I have tried all the suggested solutions on Windows, however without success.

I always encountered the following error:

This selection yields 4 experiments.
ERROR: Trial failed to complete (its results will not be included in the run):
  The worker exited without producing data. It has likely crashed. Inspect C:\Users\Piotr\AppData\Local\Temp\1583760443321-0\trial-1.log to see any worker output.

I was able to solve it by adding the @VmOptions annotation to a custom benchmark class.

Here is the full configuration:

@VmOptions("-XX:-TieredCompilation")
public class CaliperBenchmark {

    @BeforeExperiment
    void setUp() {
        // set up
    }

    @Benchmark
    void boxed() {
       // test
    }
}

Maven: com.google.caliper caliper 1.0-beta-2 compile

Main class:

public class Main {

    public static void main(String[] args) {
        CaliperMain.main(CaliperBenchmark.class, args);
    }
}

Command line: mvn exec:java -Dexec.mainClass="com.google.caliper.runner.CaliperMain" -Dexec.args="org.CaliperBenchmark"

Exoenzyme