How does jsPerf determine which of the code snippets is fastest?

Question

Today I visited jsPerf and now I am wondering…

1. What is "ops/sec"?
2. How many iterations does it do?
3. On what basis does it calculate which is faster? What is the formula behind these calculations?

Example: http://jsperf.com/concatenation-vs-join

Can anyone tell me?

Intransigence

2. It runs each given code for a few seconds (~4 seconds) – Lido

3. The code that has more OPS/Sec is faster (obviously) :) – Lido

Also, read the FAQ: jsperf.com/faq – Lido

Answer 1

I wrote Benchmark.js, which jsPerf uses.

1. "ops/sec" stands for operations per second. That is how many times a test is projected to execute in a second.

2. A test is repeatedly executed until it reaches the minimum time needed to get a percentage uncertainty for the measurement of less than or equal to 1%. The number of iterations will vary depending on the resolution of the environment’s timer and how many times a test can execute in the minimum run time. We collect completed test runs for 5 seconds (configurable), or at least 5 runs (also configurable), and then perform statistical analysis on the sample. So, a test may be repeated 100,000 times in 50 ms (the minimum run time for most environments), and then repeated 100 times more (5 seconds). A larger sample size (in this example, 100), leads to a smaller margin of error.

3. We base the decision of which test is faster on more than just ops/sec by also accounting for margin of error. For example, a test with a lower ops/sec but higher margin of error may be statistically indistinguishable from a test with higher ops/sec and lower margin of error.

   We used a welch t-test, similar to what SunSpider uses, but switched to an unpaired 2-sample t-test for equal variance (the variance is extremely small) because the welch t-test had problems comparing lower ops/sec and higher ops/sec with small variances which caused the degrees of freedom to be computed as less than 1. We also add a 5.5% allowance on tests with similar ops/sec because real world testing showed that identical tests can swing ~5% from test to re-test. T-tests are used to check that differences between tests are statistically significant.

Labialize

I've always wondered if when doing tests that manipulate i.e. HTML form elements, do you need to "reset" them back to defaults in order for the other tests to be fair/accurate? – Bissonnette

@GaryHole Yes, you would need to reset everything back to the default state outside of the timed code region for it to be as accurate as possible. – Cantlon

What is the ±percentage inside the Ops/sec column? – Immobility

Answer 2

You can read Bulletproof JavaScript benchmarks article from the authors. It uses Benchmark.js btw, which is Open Source.

Neufer