Create tests at run-time (google test)

Question

I have a Parser that has to be tested. There are lots of test input files to this Parser. The expected behaviour of the Parser is tested by comparing output of the Parser with corresponding pregenerated files.

Currently I'm processing YAML file in the test to get input files, expected files and their's description (in case of failure this description will be printed).

There are also parameters of the Parser that should be tested on the same input.

So, I need to form the following code in the test:

TEST(General, GeneralTestCase)
{
   test_parameters = yaml_conf.get_parameters("General", "GeneralTestCase");
   g_parser.parse(test_parameters);

   ASSERT_TRUE(g_env.parsed_as_expected()) << g_env.get_description("General", "GeneralTestCase");
}

TEST(Special, SpecialTestCase1)
{
   test_parameters = yaml_conf.get_parameters("Special", "SpecialTestCase1");
   g_parser.parse(test_parameters);

   ASSERT_TRUE(g_env.parsed_as_expected()) << g_env.get_description("Special", "SpecialTestCase1");
}

TEST(Special, SpecialTestCase2)
{
   test_parameters = yaml_conf.get_parameters("Special", "SpecialTestCase2");
   g_parser.parse(test_parameters);

   ASSERT_TRUE(g_env.parsed_as_expected()) << g_env.get_description("Special", "SpecialTestCase2");
}

It is easy to see the repetition of the code. So I feel there is a way to automate these tests.

Thanks in advance.

Millrace

Answer 1

Use value-parameterized tests:

typedef std::pair<std::string, std::string> TestParam;

class ParserTest : public testing::TestWithParam<TestParam> {};

TEST_P(ParserTest, ParsesAsExpected) {
   test_parameters = yaml_conf.get_parameters(GetParam().first,
                                              GetParam().second);
   g_parser.parse(test_parameters);
   ASSERT_TRUE(g_env.parsed_as_expected());
}

INSTANTIATE_TEST_CASE_P(
    GeneralAndSpecial,
    ParserTest,
    testing::Values(
        TestParam("General", "GeneralTestCase")
        TestParam("Special", "SpecialTestCase1")
        TestParam("Special", "SpecialTestCase2")));

You can read the list of test cases from disk and return it as a vector:

std::vector<TestParam> ReadTestCasesFromDisk() { ... }

INSTANTIATE_TEST_CASE_P(
    GeneralAndSpecial,  // Instantiation name can be chosen arbitrarily.
    ParserTest,
    testing::ValuesIn(ReadTestCasesFromDisk()));

Concuss

It would be good to indicate some additional information here. The registration of the tests need to happen before the call to ::testing::InitGoogleTest();. Since ReadTestCasesFromDisk() will probably need to access some argument passed by cmd arguments, the call to ::testing::InitGoogleTest(); will need probably to be moved to a later point in the main function. – Georgetown

Answer 2

I have added two classes DynamicTestInfo & ScriptBasedTestInfo as well as RegisterDynamicTest function into google unit test. It requires C++17 at least (haven't analyzed backporting to C++11 or C++14) - it allow to create google unit tests dynamically / at run-time slightly simpler than current google API's.

For example usage could be something like this:

https://github.com/tapika/cppscriptcore/blob/f6823b76a3bbc0ed41b4f3cf05bc89fe32697277/SolutionProjectModel/cppexec/cppexec.cpp#L156

But this requires modified google test, see this file for example:

https://github.com/tapika/cppscriptcore/blob/f6823b76a3bbc0ed41b4f3cf05bc89fe32697277/SolutionProjectModel/logTesting/gtest/gtest.h#L819

I will try to merge changes to official google test repository.

I have changed also how tests are reported to user application (using <loc> tag) but that one requires modified google test adapter for Visual studio, for more information see following youtube video for more explanation how things works.

https://www.youtube.com/watch?v=-miGEb7M3V8

Using newer GTA you can get file system listing in test explorer, for example like this:

Lemming

This is exactly what I'm looking for. Did you ever try to get it merged? – Fulgor

I've got both pull request rejected, in both - google test adapter and in google test as well for a different reasons. Meanwhile I'm also working on next phase of unit testing - that's recoding / playbacking & verifying same unit test. Some ideas are written down in here: github.com/csoltenborn/GoogleTestAdapter/issues/102 Unfortunately not yet available as open source code, if you really want, can get them into github as demo test project. – Lemming