How to make microbenchmark with console.time, to measure small differences in compiler optimization?

Question

This code is an adaptation of this other one... It is an ugly code but the question is about "how to do a benchmark".

1. The new console.time function measure the "real execution time" or is it not so reliable? Is it really reliable for all browsers or can it change a bit?
   PS: is better to use Date.now() (or something like Unix terminal time), external time reference?

2. How to check "compiler optimizations" as ignoring a loop that do nothing?

The benchmark below is supposed to measure the cost of operations == and === that is near the same. The problem is that each browser (Firefox and Chorme) results in different average time... So,

3. Make sense to use an "average time for all browsers" when optimizations are oposite? There are another way to estimate execution times?

---

Test by yourself in different browsers, or also with NodeJs... And remember that it is not an usual performance ("which is faster?"), but a measure of the reality, a way to check what the compilers are doing (in average) to implement the ECMA 262 prescriptions.

var testString = "444442";
var testNumber = 444442;
var testString2 = "444443";
var testObject = {};
var testObject2 = {x:1};

const MAX = 1000000;
var name;
var result;

name='===';
result = 0;
console.time('Operation '+name);
for(var i = 0; i < MAX; i++){
    result += Number(
       (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
       &&      (testString2 === testString || testNumber === testNumber) && (testObject2 === testObject || testString === testString) && (result === testNumber || testObject === testObject)
      )
      testNumber++
}
console.timeEnd('Operation '+name);

var result0 = result;

name='=='
result = 0;
console.time('Operation '+name);
var result = null;
for(var i = 0; i < MAX; i++){
  result += Number(
    (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
    &&      (testString2 == testString || testNumber == testNumber) && (testObject2 == testObject || testString == testString) && (result == testNumber || testObject == testObject)
  )
}
console.timeEnd('Operation '+name);

if (result0 != result)
console.log("OOPS BUG, result0 differing result: ", result0, result)

name='NoOp'
result = 0;
console.time('Operation '+name);
for(var i = 0; i < MAX; i++){
    result =  true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true &&
    true && true && true
}
console.timeEnd('Operation '+name);

Sample results (cutting zeros of ms)

On Firefox:

  Operation ===: 80 ms   Operation ==: 200 ms   Operation NoOp: 8 ms

On Chrome:

  Operation ===: 21 ms   Operation ==: 27 ms   Operation NoOp: 22 ms

On NodeJS:

  Operation ===: 108 ms   Operation ==: 152 ms   Operation NoOp: 9 ms

Conclusion: === is a little bit faster in all compilers.

---

The problem: running the other cited code, of this answer ("Rick test"),

On Firefox:

  Operation ===: 1089 u   Operation ==: 280 u   Operation NoOp: 281 u

On Chrome:

  Operation ===: 352 u   Operation ==: 349 u   Operation NoOp: 349 u

On NodeJS:

  Operation ===: 201 u   Operation ==: 387 u   Operation NoOp: 195 u

Conclusion: == (not ===) is faster in browsers, but inverse for Node.

---

Final conclusions: no convergence of benchmark results, even when using more loops, etc. Seems that Rick's test is subjected to optimizations (that changes over compilers), and the above here not.

Final conclusions: there is no convergence of benchmark results, even when using more loops, etc. It seems that Rick's test is subject to (compiler dependent) optimizations, and the above test (in this page) is not.

PS: of course, ideally (by specification) the operators == and === have same time when comparing equal datatypes, so is difficult to measure difference... But this is the question, I want to check this little difference with console.time.

Rosellaroselle

1. console.time is neither new nor reliable 2. you write code that cannot be optimised away (the result += Number(…) is a good idea, the &&-repetition of the expressions less so) 3. No it doesn't make sense to average them – Neurasthenic

There are a number of benchmarking libraries out there; is there a reason you're trying to roll your own? Also, you're measuring the performance of == and === in the specific situation of your setup, not its performance in an actual application (where hopefully you're not testing equality repeatedly like you are in your tests). Eric Lippert's rant about performance, while somewhat off-topic for this question, holds some pearls of wisdom. – Dincolo

Hi @HereticMonkey, can you cite some of these "benchmarking libraries out there" that will work with this kind of problem? Or cite on-line-benchmark that is reliable for it? PS: yes, I agree that is lost of my time to check this kind of performance "in an actual application" context, but this is another discussion... I need to test first the limits of the simplest benchmark procedures... And not lost my time with "== vs ===" analysis; sometimes, some basic stylistic behaviors can be reinforced by some benchmark demonstrations, avoiding aleatory stylistcs decisions. – Rosellaroselle

Hi @Bergi, thanks for all items (!). Thanks to review my code (item 2), with your confirmation I feel more confident to measure something real. About item 1, you assert that console.time not reliable... Important information for me (!), what library-test or online-test you suggest to use as better/reliable? About item 3, I agee, but perhaps an alternative to make sense is to elaborate better, expressing normalizations first. – Rosellaroselle

FYI, I have updated my test to deal with optimizations that have been implemented in recent years. To comment on what you are doing here, I considered myself lucky when I was able to see === performing better than == 75% of the time. Trying to see convergence at the microsecond level may be asking for too much. There are all kinds of things that can throw off results at the hardware level, like SpeedStep, Turbo Boost, XFR, thermal throttling, and who knows what else. – Poche