What does OpenJDK JMH "score error" exactly mean?

Question

I am using http://openjdk.java.net/projects/code-tools/jmh/ for benchmarking and i get a result like:

Benchmark                         Mode   Samples        Score  Score error    Units
o.a.f.c.j.b.TestClass.test1       avgt         5  2372870,600   210897,743    us/op
o.a.f.c.j.b.TestClass.test2       avgt         5  2079931,850   394727,671    us/op
o.a.f.c.j.b.TestClass.test3       avgt         5    26585,818    21105,739    us/op
o.a.f.c.j.b.TestClass.test4       avgt         5    19113,230     8012,852    us/op
o.a.f.c.j.b.TestClass.test5       avgt         5     2586,413     1949,487    us/op
o.a.f.c.j.b.TestClass.test6       avgt         5     1942,963     1619,967    us/op
o.a.f.c.j.b.TestClass.test7       avgt         5      233,902       73,861    us/op
o.a.f.c.j.b.TestClass.test8       avgt         5      191,970      126,682    us/op

What does the column "Score error" exactly mean and how to interpret it?

Gurkha

See also some insights in JMH Benchmark Metrics Evaluation – Overawe

Answer 1

This is the margin of error for the score. In most cases, that is a half of confidence interval. Think about it as if there is a "±" sign between "Score" and "Score error". In fact, the human-readable log will show that:

Result: 1.986 ±(99.9%) 0.009 ops/ns [Average]
  Statistics: (min, avg, max) = (1.984, 1.986, 1.990), stdev = 0.002
  Confidence interval (99.9%): [1.977, 1.995]


# Run complete. Total time: 00:00:12

Benchmark                  Mode  Samples   Score  Score error   Units
o.o.j.s.HelloWorld.hello  thrpt        5   1.986        0.009  ops/ns

Myrick

Btw, I think there's room for improvement here. jmh suffers from "false precision" (en.wikipedia.org/wiki/False_precision) as you can see from the OP's data. The displayed significant digits of the measurement falsely imply higher precision than the error does. For example, it does not make sense to say "1.776361±0.154321". Best case it would be 1.78±0.15. Any further digit is false precision because of the magnitude of the error. So be careful with precision in jmh. – Jetpropelled

In version 1.21 instead of "Score error" the column label is "Error" but the answer above still applies, right? – Dripstone

@JamesFreitas Yes. – Myrick