Interleaving EXPECT_CALL()s and calls to the mock functions

Question

The Google Mock documentation says:

Important note: Google Mock requires expectations to be set before the mock functions are called, otherwise the behavior is undefined. In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions.

Does anyone know any details behind this restriction? I have a number of unit tests which definitely violate this rule but seem to function properly, however.

Dialectology

Answer 1

Interleaving EXPECT_CALL()s and calls to the mock functions is undefined behavior

(as confirmed by Google, the makers of gmock, here)

I have to disagree with @Marko Popovic's assessment, and believe what he's done is undefined behavior that happens to work. I have personally seen what he is doing, interleaving calls, to appear to work just fine. But I think it's undefined behavior.

Nevertheless, I need clarity from Google, so I've opened this issue here: https://github.com/google/googletest/issues/2828. Please go upvote it to get it attention, as I'd like the Googletest team to clarify this themselves.

Update: Google has responded, and declared that @Marko Popovic's answer relies on undefined behavior. It is a very common trap to fall in, however, because, as Marko has pointed out, and as I have seen too, it does indeed work (most of the time at least). The problem is that it is relying on undefined gmock behavior.

The problem with undefined behavior is it works oftentimes, but isn't technically correct, may be buggy, can lead to flaky tests, and can break for unknown reasons in the future when gmock gets updated in the future. In short: undefined behavior is not future-proof and not cross-platform. It may also result in race conditions or otherwise not always work. Therefore, don't do it. Listen to Google. Their statement where they state the following is in fact correct:

In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions (https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md#using-mocks-in-tests)

---

Back to my original answer:

In this other answer of mine where I talk a lot about how to properly use multiple EXPECT_CALL()s, I state interleaving is not okay: google mock - can I call EXPECT_CALL multiple times on same mock object?

Here is a quote from my own writing:

Question 3: Can I call EXPECT_CALL to set some expectations on a mock method, call the mock method, then call EXPECT_CALL on the method again to change the expectations, then call the mock method again?

This question wasn't even explicitly asked by the OP, but the only reason I found this page is because I was searching for this answer for many hours and couldn't find it. My Google search was "[gmock multiple expect_call][10]." Therefore, others asking this question will also fall on this page and need a conclusive answer.

A: NO, you can NOT do this! Although it may seem to work in testing, according to Google, it produces undefined behavior. See general rule #2 above!

"Important note: gMock requires expectations to be set before the mock functions are called, otherwise the behavior is undefined. In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions" (https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md#using-mocks-in-tests)

Therefore, this is NOT ALLOWED!

The following code sample is also undefined behavior.

From: https://google.github.io/googletest/gmock_cheat_sheet.html#verifying-and-resetting-a-mock (emphasis added):

Do not set new expectations after verifying and clearing a mock after its use. Setting expectations after code that exercises the mock has undefined behavior. See Using Mocks in Tests for more information.

(Thanks to @nnovich-OK for pointing the above quote out in the comments).

And also, from here: https://google.github.io/googletest/gmock_for_dummies.html#using-mocks-in-tests (with original emphasis from the original quote):

Important note: gMock requires expectations to be set before the mock functions are called, otherwise the behavior is undefined. Do not alternate between calls to EXPECT_CALL() and calls to the mock functions, and do not set any expectations on a mock after passing the mock to an API.

This means EXPECT_CALL() should be read as expecting that a call will occur in the future, not that a call has occurred. Why does gMock work like that? Well, specifying the expectation beforehand allows gMock to report a violation as soon as it rises, when the context (stack trace, etc) is still available. This makes debugging much easier.

So, the following is also undefined gmock behavior:

Perhaps this is NOT undefined behavior though!? All I added was Mock::VerifyAndClearExpectations(&myMockObj).

TEST(FooTest, testCaseName)
{
    MyMock myMockObj;
    ...
    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(true));
    testMethod();
    ASSERT_THAT(...);

    Mock::VerifyAndClearExpectations(&myMockObj); // <== NOTICE THIS ADDED LINE!
    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(false));
    testMethod();
    ASSERT_THAT(...);
}

References:

1. @Marko Popovic's assessment
2. Googletest issue where I ask for clarity about this: https://github.com/google/googletest/issues/2828
3. [my own answer] google mock - can I call EXPECT_CALL multiple times on same mock object?

Plesiosaur

Thanks for driving the clarification further. Just a remark on the 3rd question: changing expectations is definitely possible, see github.com/google/googletest/blob/master/googlemock/docs/… – Dialectology

Ah...I see, but NOT by another call to EXPECT_CALL() on that mocked method. You must call Mock::VerifyAndClearExpectations(&mock_obj); or Mock::VerifyAndClear(&mock_obj); instead. – Plesiosaur

Ah...interesting! This also implies that if you simply put a Mock::VerifyAndClearExpectations(&mock_obj) before each new call to EXPECT_CALL() for that mocked object, then the interleaving behavior is no longer undefined! I'll add this to my answer. – Plesiosaur

Current wording in cheat sheet: Do not set new expectations after verifying and clearing a mock after its use. Setting expectations after code that exercises the mock has undefined behavior. It looks like Mock::VerifyAndClearExpectations() doesn't help. – Rubberneck

Answer 2

Just as the documentation says, what is important is that expectations are set before the method is called. However, this does not mean that it is not possible to interleave EXPECT_CALL and invocation of mock method. It's just that you have to clear the expectations on the mock object yourself. For example, assume that we have the following mock class

class MyMock : public bravo::IRealClass
{
public:
    ...
    MOCK_METHOD1(myMethod, bool(int));
    ...
}

Now, assuming that a call to method testMethod calls myMethod once, you can write something like this in a test:

TEST(FooTest, testCaseName)
{
    MyMock myMockObj;
    ...
    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(true));
    testMethod();
    ASSERT_THAT(...);

    // Explicitly clear expectations and set new ones.
    Mock::VerifyAndClearExpectations(&myMockObj); 
    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(false));

    testMethod();
    ASSERT_THAT(...);
}

This will be fine since the mock object can be reliably reused again. However, if you omit to explicitly clear the expectations, you are entering the realm of undefined behavior. As is always the case with undefined behavior, it is possible that it won't crash and it might even work in some cases, but if you have have something like that in your code you should definitely fix it.

Jot

Why would the last set of expect's be a problem? It's not like the EXPECT_CALL expires after the first testMethod(). Otherwise it wouldn't be able to check if indeed the method was called once and once only. – Cade

@DBendrenko, it seems to me that according to @Marko Popovic, the problem with the 2nd one isn't really that testMethod() is called before the 2nd EXPECT_CALL(), as that would simply test the first EXPECT_CALL() yet again, but rather, it is that there is NOT another testMethod() AFTER the 2nd EXPECT_CALL(). In other words, @Marko Popovic is saying that for each EXPECT_CALL(), you must have at least one call to the mock method. This certainly isn't true though because you can do an EXPECT_CALL() with .Times(0) and have no issues. Nevertheless, in this case he may be right... – Plesiosaur

...but it goes exactly against what Google said: "In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions", so I have a hard time believing he is correct even though I have seen constructs like this to work too. I think @Marko is wrong and what he is showing in the first case is undefined behavior that does happen to work, and in the 2nd case, undefined behavior that doesn't happen to work. I'd like an authoritative answer from Google. Let's open an issue on the Googletest repo. – Plesiosaur

Issue opened: github.com/google/googletest/issues/2828 – Plesiosaur

@Marko Popovic, Google has responded, and declared what you have done to be undefined behavior. It is a very common trap to fall in, however, because, as you have pointed out, it does indeed work. The problem is that it is relying on undefined behavior, and therefore may result in race conditions, be buggy, cause flaky tests, or otherwise not always work. Therefore I feel compelled to downvote your answer. – Plesiosaur