(Google Test) Automatically retry a test if it failed the first time

Question

Our team uses Google Test for automated testing. Most of our tests pass consistently, but a few seem to fail ~5% of the time due to race conditions, network time-outs, etc.

We would like the ability to mark certain tests as "flaky". A flaky test would be automatically re-run if it fails the first time, and will only fail the test suite if it fails both times.

Is this something Google Test offers out-of-the-box? If not, is it something that can be built on top of Google Test?

Marcum

Wouldn't a better approach be to actually fix the tests? – Hipster

I mean...obviously. But in the meantime, we'd still like to run these tests to ensure good code coverage. The current "solution" is to disable the tests while we improve robustness, but we'd like something better. – Marcum

What good is code coverage for code that doesn't work? "Not that it works, but we have tested it all". – Stamen

I am also confused. Why would "good code coverage" have any value if the tests fail? – Mitchellmitchem

You guys do raise a good point. To clarify, these tests do work...most of the time. In my opinion, there is still value in running a test that passes ~95% of the time. If a test that was previously passing ~95% of the time suddenly starts passing 0% of the time, that would indicate a regression. – Marcum

To support my use case, I want to point out that unit test frameworks for other languages do support this. For example, pytest-rerunfailures. – Marcum

You can run all tests multiple times (to have a check "test passes ~95% of times"). I don't there is an option to automatically rerurn failed tests only. – Leningrad

Do you mean "fluky"? "flaky" can be corn, but not a test case. – Fusible

Answer 1

You have several options:

1. Use --gtest_repeat for the test executable:

The --gtest_repeat flag allows you to repeat all (or selected) test methods in a program many times. Hopefully, a flaky test will eventually fail and give you a chance to debug.

You can mimic tagging your tests by adding "flaky" somewhere in their names, and then use the gtest_filter option to repeat them. Below are some examples from Google documentation:

$ foo_test --gtest_repeat=1000
Repeat foo_test 1000 times and don't stop at failures.

$ foo_test --gtest_repeat=-1
A negative count means repeating forever.

$ foo_test --gtest_repeat=1000 --gtest_break_on_failure
Repeat foo_test 1000 times, stopping at the first failure.  This
is especially useful when running under a debugger: when the test
fails, it will drop into the debugger and you can then inspect
variables and stacks.

$ foo_test --gtest_repeat=1000 --gtest_filter=Flaky.*
Repeat the tests whose name matches the filter 1000 times.

See here for more info.

2. Use bazel to build and run your tests:

Rather than tagging your tests in the test files, you can tag them in the bazel BUILD files.

• You can tag each test individually using cc_test rule.
• You can also define a set of tests (using test_suite) in the BUILD file and tag them together (e.g. "small", "large", "flaky", etc). See here for an example.

Once you tag your tests, you can use simple commands like this:

  % bazel test --test_tag_filters=performance,stress,-flaky //myproject:all

The above command will test all tests in myproject that are tagged as performance,stress, and are not flaky.

See here for documentation.

Using Bazel is probably cleaner because you don't have to modify your test files, and you can quickly modify your tests tags if things change.

See this repo and this video for examples of running tests using bazel.

Snowslide

Thank you for the detailed comment! Unfortunately, our project is heavily integrated with CMake, so the Bazel solution is off the table (cool to see Bazel supports this, though). Your first solution is clever, I hadn't considered using --gtest_filter in that way. --gtest_repeat seems to do the opposite of what we're trying to accomplish, but I think we can easily write a custom script to operate on just the flaky tests. – Marcum