How does google test make test sequence

P

4

21

How google-test makes test sequence (or order of test case execution) for testing test cases?

Suppose I have 5 test cases.

TEST(First, first)
TEST(Secnd, secnd)
TEST(Third, third)
...
TEST(Fifth, fifth)

How google-test test above test cases? I mean in what sequence? Or can we provide any test sequence?

Publication answered 5/10, 2012 at 9:12 Comment(0)

E

5

The advanced reference pages for googletest in the chapter Shuffling the Tests tells :

By default, Google Test uses a random seed calculated from the current time. Therefore you'll get a different order every time.

This is actually a good way of unit testing, since tests should not depend on the order of execution.

As far as I know, there are no ways of setting the order of tests execution. The only parameter you can set is the seed, used to set the same order of execution.

European answered 5/10, 2012 at 9:25 Comment(5)

That (random order of execution) is true only if you specify the --gtest_shuffle flag. – Worthington 7/3, 2013 at 10:22

"tests should not depend on the order of execution" What if one small test failing means many other tests fail? In that case it's faster to run the small test first and abort when it fails. – Snowshed 15/10, 2019 at 5:26

@Snowshed no, it is better not to abort, because in that case you wouldn't see what other tests are failing. You can fix more tests at once. Executing unit tests should be very very fast, therefore it doesn't matter if a test is big or small. Time it takes me to execute 600 unit tests for one of my libraries takes me 0.24 seconds. My PC is middle class. – Philosophy 15/10, 2019 at 7:33

There are times where it is useful to have an order of execution. My current project had a handful of "bootstrap" tests which test the underlying functionality, along with dozens of higher-level tests which unit test higher level functionality. If something goes wrong in the bootstrap, it's nice to know then, rather than having to wade through dozens of higher level tests that all failed. – Eldwen 7/5, 2020 at 15:26

On Jul 17, 2018, they updated the doc so now (2020-08-26) it is "By default, googletest uses a random seed calculated from the current time. Therefore you'll get a different order every time." The random order has become a default behavior, although --gtest_shuffle surely still works. – Lowrie 26/8, 2020 at 19:10

L

19

By default it will test them in the order it finds them at link time, which will depend upon your tools.

You can select which tests to run, such as a subset, or a single test.

There is also an option to run them in a random order.

Liguria answered 5/10, 2012 at 9:25 Comment(0)

E

5

The advanced reference pages for googletest in the chapter Shuffling the Tests tells :

By default, Google Test uses a random seed calculated from the current time. Therefore you'll get a different order every time.

This is actually a good way of unit testing, since tests should not depend on the order of execution.

As far as I know, there are no ways of setting the order of tests execution. The only parameter you can set is the seed, used to set the same order of execution.

European answered 5/10, 2012 at 9:25 Comment(5)

That (random order of execution) is true only if you specify the --gtest_shuffle flag. – Worthington 7/3, 2013 at 10:22

"tests should not depend on the order of execution" What if one small test failing means many other tests fail? In that case it's faster to run the small test first and abort when it fails. – Snowshed 15/10, 2019 at 5:26

@Snowshed no, it is better not to abort, because in that case you wouldn't see what other tests are failing. You can fix more tests at once. Executing unit tests should be very very fast, therefore it doesn't matter if a test is big or small. Time it takes me to execute 600 unit tests for one of my libraries takes me 0.24 seconds. My PC is middle class. – Philosophy 15/10, 2019 at 7:33

There are times where it is useful to have an order of execution. My current project had a handful of "bootstrap" tests which test the underlying functionality, along with dozens of higher-level tests which unit test higher level functionality. If something goes wrong in the bootstrap, it's nice to know then, rather than having to wade through dozens of higher level tests that all failed. – Eldwen 7/5, 2020 at 15:26

On Jul 17, 2018, they updated the doc so now (2020-08-26) it is "By default, googletest uses a random seed calculated from the current time. Therefore you'll get a different order every time." The random order has become a default behavior, although --gtest_shuffle surely still works. – Lowrie 26/8, 2020 at 19:10

N

4

Even if you can guess some pattern for the order of execution (as written, or linked) you shouldn't depend on that.

It would be repeated on different executions, though. If you don't want it to happen, you can use --gtest_shuffle. That runs the test on a random order, according to a random seed.

In case of failure, you can use --gtest_random_seed= with that number and repeat the exact sequence (to investigate why it failed).

That said, the randomness is not complete:

Test suites will be run in a random order
Tests within a test suite will be run in a random order

If not run in this way, SetUpTestSuite and TearDownTestSuite methods would be mixed. You don't need a fixture for this grouping to happen, though.

Nicholenicholl answered 14/4, 2020 at 9:1 Comment(0)

W

3

By default they run in the declaration order. As said by other, you have to provide the flag --gtest_shuffle to shuffle them.

Wacky answered 7/12, 2016 at 15:9 Comment(2)

it states, they don't: Remember that the test order is undefined, so your code can't depend on a test preceding or following another. – Downdraft 22/11, 2018 at 7:25

The given reference doesn't appear to say that tests are run in declaration order. More likely it is dependent on tool order, as per another answer. – Corkboard 23/11, 2018 at 10:53

Recommended topics

Hot tags