Benchmarking with googletest?

Question

Background (skip to Question below if not interested)

I have a simulator that runs through three states:

1. Single threaded startup (I/O ok)
2. Multi-threaded in-memory CPU-bound simulation stage (I/O not ok)
3. Post-simulation, post-join single threaded stage (I/O ok)

What the heck! During standard testing, CPU usage dropped from 100% down to 20%, and the total run took about 30 times longer than normal (130secs vs 4.2secs).

When Callgrind revealed nothing suspicious, my head buzzed as I was on the precipice of rolling back to the last commit, losing all bug-fixes.

Discouraged, I walked into the server room during a run and noticed nasty grinding sounds, verified later to be caused by writes to Mysql sockets in /proc/PID/fd!!! It turned out that Mysql code, several layers deep in Stage 2., was causing problems.

Lessons Learned

1. Accidental I/O can be lethal to a real-time application
2. Unit testing is not enough: I need benchmarking, too

Fix I will introduce thread-local-storage IOSentinels and asserts() on ReadAllowed() and WriteAllowed() to ensure that Stage 2 threads will never do any IO.

Question

Anyone have any luck with attaching/writing a benchmarking framework with googletest?

Unfortunately, all my googletests passed this time. Had I stepped away for a bit and come back without noticing the run-time, this would have been a disastrous commit, and possibly much harder to fix.

I would like googletest to fail if a run takes >2 or 3 times the last runtime: this last part is tricky because for very quick runs, system state can cause something to take twice as long but still be ok. But for a long simulation run/test, I don't expect runtimes to change by a great deal (>50% would be unusual).

I am open to suggestions here, but it would be nice to have a low-maintenance check that would work with automated testing so it will be obvious if the system suddenly got slow, even if all the outputs appear to be ok.

Savoirvivre

Answer 1

I do this in v1.5.0:

BENCHMARK(SomeBenchmark);
BENCHMARK(AnotherBenchmark);

TEST(MyTest, Benchmarks)
{
    ::benchmark::RunSpecifiedBenchmarks();
}

That is, I just call RunSpecifiedBenchmarks directly in one of the gtest tests.

Finsen

Answer 2

Google Test framework proposes by default a measure of elapsed time. It is commanded by an environment variable, GTEST_PRINT_TIME. This variable defaults to 1.

So, why not monitor elapsed time using this feature of Google Test platform?

Here is a word on elapsed time variable in Google Test:

By default, googletest prints the time it takes to run each test. To disable that, run the test program with the --gtest_print_time=0 command line flag, or set the GTEST_PRINT_TIME environment variable to 0.

Ahlers

+1 this is almost what I'm looking for - if I could actually access the result per test as part of some validation, that would be perfect: eg EXPECT_LT( TEST_RUNTIME, 1000) for less than a thousand milliseconds... – Savoirvivre

Answer 3

Some updates on this question (in 2016):

1. Here is a nice blog-post of Nick Brunn about his Hayai benchmarking framework. (2012)

   • It does not provide the possibility of specifying running time requirements.
   • It is very similar to Google Test. Syntax, etc.
   • It provides the benchmarking results to the user or a Continous Integration framework. Also have a look at MojaveWastelander's fork for active development and MSVC support.

2. Google published 'Benchmark' in 2014. This provides similar behaviour then Hayai above. As far as I understand, defining requirements is not possible. Again, the syntax is inspired by GoogleTest.

   • There are even advanced features such as measuring complexity (big-O).

3. GoogleTest has this as an open feature on Github. There is a rudimentary implementation but it is not part of GoogleTest yet.

Welter

Answer 4

Isn't it just as simple as this?

const clock_t t0 = clock(); // or gettimeofday or whatever
int res = yourFunction();
const clock_t t1 = clock();
const double elapsedSec = (t1 - t0) / (double)CLOCKS_PER_SEC;
EXPECT_EQ(EXPECTED, res);
EXPECT_GT(10.0, elapsedSec);

Here, you need to manually change 10.0 depending on your task.

Of course, you can go further by something like:

double prev = -1;
{
  ifstream ifs("/var/tmp/time_record.txt");
  ifs >> prev;
}
if (prev < 0) prev = DEFAULT_VALUE;
// ...
EXPECT_GT(2 * prev, elapsedSec);
{
  ofstream ofs("/var/tmp/time_record.txt");
  ofs << elapsedSec << endl;
}

But I wonder this additional complexity can really be justified.

Chauffeur

Or, if possible, use the std::chrono libraries from C++11. The googletest library works fine when compiling for C++11 as well. – Singlecross