What is the meaning of Google Benchmark Iteration?

Question

I am working with Google Benchmark to measure the execution time of some code. For example, I wrote the following code to measure its execution time performance.

#include <benchmark/benchmark.h>

// Alternatively, can add libraries using linker options.
#ifdef _WIN32
#pragma comment ( lib, "Shlwapi.lib" )
#ifdef _DEBUG
#pragma comment ( lib, "benchmarkd.lib" )
#else
#pragma comment ( lib, "benchmark.lib" )
#endif
#endif

static void BenchmarkTestOne(benchmark::State& state) {
    int Sum = 0;
    while (state.KeepRunning())
    {
        for (size_t i = 0; i < 100000; i++)
        {
            Sum += i;
        }
    }
}

static void BenchmarkTestTwo(benchmark::State& state) {
    int Sum = 0;
    while (state.KeepRunning())
    {
        for (size_t i = 0; i < 10000000; i++)
        {
            Sum += i;
        }
    }
}

// Register the function as a benchmark
BENCHMARK(BenchmarkTestOne);
BENCHMARK(BenchmarkTestTwo);


// Run the benchmark
BENCHMARK_MAIN();

When the above code has run, it shows me the following results:

Benchmark                 Time             CPU   Iterations
-----------------------------------------------------------
BenchmarkTestOne     271667 ns       272770 ns         2635
BenchmarkTestTwo   27130981 ns     27644231 ns           26

But I couldn't figure out what is the meaning of Iterations here? And also why Time and CPU are different from each other?

Jovian

Iterations is the number of iterations that were needed to get a stable result. – Nagano

@JesperJuhl So why in the second test, it is 26 but in the first test it is 2635? – Jovian

"So why in the second test, it is 26 but in the first test it is 2635" - because, for some reason (that I can't tell you), the test suite decided that the second test gave stable results after 26 runs, but the first required more than two and a half thousand before it could nail down a sensible value. – Nagano

Answer 1

Google Benchmark tries to benchmark each candidate for a similar amount of time, and/or for long enough to get stable results.

The benchmark counts how many iterations it actually did, along with the exact time. A much slower per-iteration benchmark will do far fewer iterations.

The printout is (calculated) per-iteration time, and (counted) iterations of the benchmark function.

It might actually be a count of calls to state.KeepRunning(), but I don't know that level of detail.

---

Just FYI, your benchmark loops don't return any result or store it to a volatile after the loop, so a compiler could easily optimize away the loop. Also note that signed overflow is UB in C, and your int will pretty definitely overflow.

(Or clang could still optimize those sum loops into a closed form formula based on Gauss's n * (n+1) / 2 but avoiding overflow.)

Benchmarking with optimization disabled is useless; don't do it.

Burton

Answer 2

as the user guide write: When the benchmark binary is executed, each benchmark function is run serially. The number of iterations to run is determined dynamically by running the benchmark a few times and measuring the time taken and ensuring that the ultimate result will be statistically stable. As such, faster benchmark functions will be run for more iterations than slower benchmark functions, and the number of iterations is thus reported.

so the iteration is in the same limit time, the benchmark function can iteration time. the iteration times is larger the faster the program is.

Chrysotile