How to catch segmentation fault with Google Test?

Question

How do I test that a function won't produce a segmentation fault?

Here what I know right now, I can do:

EXPECT_DEATH(foo(nullParameter))

In side the function, a segmentation fault is produce which is the behavior that I want to make fail. The snippet above will make the test pass because that is what is expected, the death of the process.

Now, how can I make it fail?

Bernard

Other than not dying, I assume? You want what, a segfault that the tester misses? – Watermelon

It's too late once a segfault happens. It's easier to prevent getting hit by a bus than it is to revive the dead. – Voodoo

The idea behind this is that I know that the function is causing a segfault and there will be a patch to fix this. Right now my goal is to make a test that fails before the path is made. – Bernard

Answer 1

Here's a function that will segfault if passed a null pointer argument and otherwise not:

int deref(int * pint)
{
    return *pint;
}

And here is a googletest program that tests that behaviour:

main.cpp

#include <gtest/gtest.h>

int deref(int * pint)
{
    return *pint;
}


TEST(test_deref_1,will_segfault)
{
    ASSERT_EXIT((deref(nullptr),exit(0)),::testing::KilledBySignal(SIGSEGV),".*");
}


TEST(test_dref_2,will_not_segfault)
{
    int i = 42;
    ASSERT_EXIT((deref(&i),exit(0)),::testing::ExitedWithCode(0),".*");
}


int main(int argc, char **argv) {
    ::testing::InitGoogleTest(&argc, argv);
    return RUN_ALL_TESTS();
}

Compile and link:

$ g++ -Wall -Wextra -pedantic -o tester main.cpp -pthread -lgtest

Run:

$ ./tester 
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from test_deref_1
[ RUN      ] test_deref_1.will_segfault
[       OK ] test_deref_1.will_segfault (168 ms)
[----------] 1 test from test_deref_1 (168 ms total)

[----------] 1 test from test_dref_2
[ RUN      ] test_dref_2.will_not_segfault
[       OK ] test_dref_2.will_not_segfault (1 ms)
[----------] 1 test from test_dref_2 (1 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (169 ms total)
[  PASSED  ] 2 tests.

As far as I can imagine, TEST(test_deref_1,will_segfault) is a pointless test, because I cannot think of any circumstances in which I would want to assure myself that a program will segfault as a result of making a certain call to a function I have written.

TEST(test_dref_2,will_not_segfault) is possibly a useful kind of test. In effect, it is a test that the program:

int main()
{
    int i = 42;
    defref(&i);
    exit(0);
}

will terminate by exit(0) rather than in any premature abnormal way. A better name for this test would probably be TEST(test_dref,does_not_crash), or similar.

It is a possibly useful kind of test because there could be a significant risk of it failing, if defref was some sufficiently complicated code, and the test suite could report that failure without crashing itself. We can force a failure by rewriting it:

TEST(test_dref_2,will_not_segfault)
{
    ASSERT_EXIT((deref(nullptr),exit(0)),::testing::ExitedWithCode(0),".*");
}

and then test test report is:

$ ./tester
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from test_deref_1
[ RUN      ] test_deref_1.will_segfault
[       OK ] test_deref_1.will_segfault (147 ms)
[----------] 1 test from test_deref_1 (147 ms total)

[----------] 1 test from test_dref_2
[ RUN      ] test_dref_2.will_not_segfault
main.cpp:25: Failure
Death test: (deref(nullptr),exit(0))
    Result: died but not with expected exit code:
            Terminated by signal 11 (core dumped)
Actual msg:
[  DEATH   ] 
[  FAILED  ] test_dref_2.will_not_segfault (90 ms)
[----------] 1 test from test_dref_2 (90 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (237 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] test_dref_2.will_not_segfault

 1 FAILED TEST

See the documentation of {ASSERT|EXPECT}_EXIT to understand these macros.

Septate

The documentation also puts a lot of emphasis about not using underscores in test names. You should probably refactor your code. – Ribera

@Mike Kinghan But in case of a failure a core file still gets created, right? – Calices

Answer 2

Tests which crash are already failures (presumably you don't want any of your code to segfault). Just test for the behavior you expect, as with any other test.

Discourteous

What I want is to test that the segment fault doesn't happen. For example I want to say something EXPECT_NO_DEATH, which implies that the code successfully execute. – Bernard

I second that test should continue running if one test segfaults. I think the answer by @marcin wrochna comes closest to this expectation. If the testrunner just crashes you get no info which test caused the failure and therefore you're facing a worse debugging experience. – Edraedrea

@WörDuSchnaffzig If your test runner doesn't tell you which test was running at the point of the crash (or even give you a callstack) then that's more the test runner's fault. The normal way for a test runner to do this (as described in marcin's answer) is to delegate the actual running to child processes. – Discourteous

Answer 3

GoogleTest crashes when the tested program segfault, without producing a useful test report, so in general each test needs to be run in a subprocess.

One way to do that is to wrap all your tests to exit(0) and ASSERT_EXIT as described in the top answer.

However, a much more convenient way to do that is to use gtest-parallel. With gtest-parallel --serialize_test_cases --worker=1, it just calls gtest as a subprocess, which solves the issue: segfaults get reported as test failures without any changes in tests. It's a wrapper for gtest, so it has mostly the same command line options as gtest.

Taught