Which JS benchmark site is correct?

Question

I created a benchmark on both jsperf.com and jsben.ch, however, they're giving substantially different results.

JSPerf: https://jsperf.com/join-vs-template-venryx
JSBench: http://jsben.ch/9DaxR

Note that the code blocks are exactly the same.

On jsperf, block 1 is "61% slower" than the fastest:

On jsbench, block 1 is only 32% slower than the fastest: ((99 - 75) / 75)

What gives? I would expect benchmark sites to give the same results, at least within a few percent.

As it stands, I'm unable to make a conclusion on which option is fastest because of the inconsistency.

EDIT

Extended list of benchmarks:

• https://jsperf.com/join-vs-template-venryx
• https://jsbench.me/f3k3g71sg9
• http://jsbench.github.io/#7f03c3d3fdc9ae3a399d0f2d6de3d69f
• https://run.perf.zone/view/Join-vs-Template-Venryx-1512492228976
• http://jsben.ch/9DaxR

Not sure which is the best, but I'd skip jsben.ch (the last one) for the reasons Job mentions: it doesn't display the number of runs, the error margin, or the number of operations per second -- which is important for estimating absolute performance impact, and enabling stable comparison between benchmark sites and/or browsers and browser versions.

(At the moment http://jsbench.me is my favorite.)

Mandate

Answer 1

March 2019 Update: results are inconsistent between Firefox and Chrome - perf.zone behave anomalously on Chrome, jsben.ch behaves anomalously on Firefox. Until we know exactly why, the best you can do is benchmark on multiple websites (but I'd still skip jsben.ch, the others give you a least some error margin and stats on how many runs were taken, and so on)

TL;DR: running your code on perf.zone and on jsbench.github.io (see here and here), the results closely match jsperf. Personally, and for other reasons than just these results, I trust these three websites more than jsben.ch.

Recently, I tried benchmarking the performance of string concatenation too, but in my case it's building one string out of 1000000+ single character strings (join('') wins for numbers this large and up, btw). On my machine the jsben.ch timed out instead of giving a result at all. Perhaps it works better on yours, but for me that's a big warning sign:

http://jsben.ch/mYaJk

http://jsbench.github.io/#26d1f3705b3340ace36cbad7b24055fb

https://run.perf.zone/view/join-vs-concat-when-dealing-with-very-long-lists-of-single-character-strings-1512490506658

(I can't be bothered to ever have to deal with jsperf's not all tests inserted again, sorry)

At the moment I suspect but can't prove that perf.zone has slightly more reliable benchmark numbers:

• when optimising lz-string I used jsbench.github.io for a very long time, but at some point I noticed there were impossibly large error margins for certain types of code, over 100%.

• running benchmarks on mobile is fine with jsperf.com and perf.zone, but jsbench.github.io is kinda janky and the CSS breaks while running tests.

Perhaps these two things are related: perhaps the method that jsbench.github.io uses to update the DOM introduces some kind of overhead that affects the benchmarks (they should meta-benchmark that...).

Note: perf.zone is not without its flaws. It sometimes times out when trying to save a benchmark (the worst time to do so...) and you can only fork your own code, not edit it. But the output still seems to be more in line with jsperf, and it has a really nice "quick" mode for throwaway benchmarking

Priestly

In my latest test on jsbench.ch an empty block of code gave a result of more than 40%. it looks like the timer is timing more things than just your code. – Nicolis

Yeah, and perf.zone got taken off-line, sadly. Lost a lot of benchmarks because of that :( – Priestly

Yes I noticed :( Maybe it was abused for pirating/hacking purposes? jsperf also had problems with this in the past: github.com/jsperf/jsperf.com/issues/436 github.com/jsperf/jsperf.com/issues/232 – Nicolis

Well, it was a single guy's passion project and nobody came onboard to build on it, so I guess he decided to stop paying for it. I can't blame him. The source still exists on GitHub though – Priestly

It looks like perf.zone is back online! :) – Nicolis

Answer 2

Sorry for a bump but might be interesting for others running into this in search results.

I can't speak for others but jsbench.me just uses benchmark.js for testing. Its a single-page React app meaning it runs completely on your browser and your engine of choice, so results should be consistent within single browser. You can run it in Firefox or mobile and results will be different of course. But absolutely nothing related to testing is on the server, other than AWS DynamoDB to store results.

P.S. I'm the author, so only passion project of individual. Currently doesnt cost me anything as its optimized for serverless and it fits AWS free tier. Anount of work on it is proportional to number of users :)

Homothallic

thank you for making jsbench.me Unfortunately the size limits are too small for many types of tests. In particular the setup's 2k limit just isn't remotely enough. – Befriend

@gman, yes, I'm aware of that. Technically, it's very easy to fix it, config is there and ready, but financially, I have all kinds of limits to keep this within AWS Free Tier limit. I'm looking into some way of minimal monetization to lift those limits. Probably donations. Keep an eye on it, I'm hoping to change it within next few months. Or just subscribe to this issue and you'll get notified: github.com/psiho/jsbench-me/issues/24 – Agbogla

@MirkoVukušić The problem with your site is that we can't choose the number of iterations, – Phane

Answer 3

AFAIK one issue is that various JavaScript engines optimize vastly differently based on the environment.

I have a test of the exact same function that produces different results based on where the function is created. In other words, for example, in one test it's

const lib = {}
lib.testFn = function() {
   ....
}

And in other it's

const lib = {
 testFn: function() {
   ....
 },
};

and in another it's

function testFn() {
   ....
}

const lib = {}
lib.testFn = testFn

and there's a >10% difference in results for a non-trivial function in the same browser and different results across browsers.

What this means is no JavaScript benchmark is correct because how that benchmark runs it's tests, as in the test harness itself, affects the results. The harness for example might XHR the test script. Might call eval. Might run the test in a worker. Might run the test in an iframe. And the JS engine might optimize all of those differently.

Befriend