Google Caliper: Trouble getting started (Java Benchmarking)

Question

I am trying to use Google Caliper to benchmark some simple code. I am using the examples from their websites. Here's what I've done so far:

1. Downloaded the Caliper JAR and added it to my Netbeans project
2. After having difficulties, I downloaded JUnit.jar and hamcrest.jar. Still not working.

Here's my code:

import com.google.caliper.Benchmark;

public class Benchmark1 extends Benchmark {

    public void timeNanoTime(int reps) {
      for (int i = 0; i < reps; i++) {
        System.nanoTime();
      }
    }

}

I am extending Benchmark because when I try to extend "SimpleBenchmark" like on their website it tells me it cannot find SimpleBenchmark. I then, in my main method, create a new Benchmark1() hoping something will happen. Nothing does. This is the code inside my main class.

Benchmark1 test = new Benchmark1();
test.timeNanoTime(10);

I know this is no doubt a simple error but I cannot, despite much Googling, figure out where I'm going wrong. The code compiles but does not run.

EDIT: I should say I'm running Netbeans on Windows 7 with Caliper 1.0

Overweary

Answer 1

It's true; the documentation is woefully outdated and incomplete. I'm working on it. In the meantime, here's what will get your benchmark running.

Your main method should delegate to CaliperMain, not directly to the benchmark. Try

public static void main(String[] args) {
    CaliperMain.main(Benchmark1.class, args);
}

Windows will be a problem. Particularly, issue 215 will be the biggest blocker.

Hydrolytic

Windows is not supported? WTF Please post this as a big red notice on the project homepage. – Simms

The current, pre-release version Caliper doesn't support Windows due to a bug. Support will return as we work through bugs, but none of the developers have Windows machines, so the turn-around has been slow. Moreover, Windows has been a lower priority as it is a less common production environment and somewhat poor benchmarking environment due to imprecision in the timers (similar issue with OS X). That said, since people develop on Windows it is still a goal to get it working. Please star the issue (and any others) as we consider those a factor in prioritizing work. – Hydrolytic

Nice to hear that the team is working on it. Didn't realize that caliper is in it's pre-release. It was a little bit frustrating to hear that windows does not work with caliper after spending time on building benchmarks and getting things to work. I share the opinion that Linux is smarter for production machines, but sometimes you code on your laptop. Most laptops come with Windows and special software from the manufacturerer relying on it. Thus it's more comfortable to use a ready to go System. Perhaps your developers can get a Windows from dreamspark or elsewhere. – Simms

FWIW, the version @HEAD may work with Windows now if somebody wants to try it. – Hydrolytic

May I suggest that you add a notice to the homepage that the published jars don't support Windows, as suggested by Thorben? It's great that it's being worked on and/or fixed in head, but that won't help anyone who develops on Windows, sees a jar there and expects it to work. – Fanciful

Hey how about working with Annotations? How can i load a SNAPSHOT to use Annotations? – Treat

I have been struggling a lot to find some documentation. Some tutorial on caliper as to how to start would be extremely helpful. – Disarming

Answer 2

You could switch over to Perfidix http://perfidix.org/

Perfidix has eclipse Integration and can be used like JUnit.

Another option would be JUnitbenchmarks http://labs.carrotsearch.com/junit-benchmarks.html

It's a really great framework for Junit 4+. It can even build html charts to compare results.

Simms