Comparison of C++ unit test frameworks [duplicate]

Question

I know there are already a few questions regarding recommendations for C++ unit test frameworks, but all the answers did not help as they just recommend one of the frameworks but do not provide any information about a (feature) comparison.

I think the most interesting frameworks are CppUnit, Boost and the new Google testing framework. Has anybody done any comparison yet?

Purveyor

I have my own IOC based testing framework which I like more because it isn't just a clone of what all the others do but addresses what I find all the problems of the others to be. You write test cases by deriving from a class, not by using macros. Macros only used for assertions as they give you reflection. Customised output of testing statistics. Run from IOC scripting so you choose what you test, how often and with what parameters. – Thing

and it's brilliant from a development point of view as when I add my own test I can run it without having to run everyone else's at the same time. So I know that my code is working. – Thing

Answer 1

See this question for some discussion.

They recommend the articles: Exploring the C++ Unit Testing Framework Jungle, By Noel Llopis. And the more recent: C++ Test Unit Frameworks

I have not found an article that compares googletest to the other frameworks yet.

Rolling

As I wrote: all the answers just recomend one of the frameworks but do not compare the framework to another. – Purveyor

You're not happy with the article either ? – Randarandal

One criticism: the article, while good, is from 2004 and doesn't include Google Test. – Desist

In the first link you'll see two comparisons. Except the new framework from google, most information is (are?) still relevant. (And CppUnit is not the most interesting, it's too clumsy to use) – Lister

fixed up links and expanded the answer with a more recent comparison – Rolling

The second link is broken. Could you please fix it? – Stirring

Answers shouldn't include just a link to see something. it's now common in the community to include full info, since all links will become invalid at sometime throughout the great eternal test of time, where at some time all things end, and all things begin, including us and our descendants. – Atalya

The link to the more recent link resource is outdated. It states that boost does not allow to select tests from the command line which is no longer true. Not only can you select tests to run from the command line, but you can also do so using patterns. – Sigurd

Answer 2

A new player is Google Test (also known as Google C++ Testing Framework) which is pretty nice though.

#include <gtest/gtest.h>

TEST(MyTestSuitName, MyTestCaseName) {
    int actual = 1;
    EXPECT_GT(actual, 0);
    EXPECT_EQ(1, actual) << "Should be equal to one";
}

Main features:

• Portable
• Fatal and non-fatal assertions
• Easy assertions informative messages: ASSERT_EQ(5, Foo(i)) << " where i = " << i;
• Google Test automatically detects your tests and doesn't require you to enumerate them in order to run them
• Make it easy to extend your assertion vocabulary
• Death tests (see advanced guide)
• SCOPED_TRACE for subroutine loops
• You can decide which tests to run
• XML test report generation
• Fixtures / Mock / Templates...

Kirkland

I really enjoy using google test over some of the other frameworks especially with its mocking capabilities that can be found in the googlemock framework. – Vaientina

I provide all these features (although some are not yet public) and more in my new test framework, CATCH. See my answer for link. – Leucas

combining it together with Google C++ Mocking framework makes it really powerful xUnit test framework for unit test C++ code. – Screed

@Thing Running with the build is something different from test detection. Running with build depends on your build system. Test detection means you don't have to list all tests in another class, just create the tests methods and that's it. – Kirkland

I dislike their overuse of macros though, and the fact that use common words like TEST which might clash with something. GTEST would be better, less likely to clash. – Thing

@Thing I would keep tastes outside SO (and you can rename if you really wanted to). Macros are ugly I admit but probably a necessary evil in C++ to support all the features without scaffolding code. – Kirkland

By Scaffolding code you mean deriving from their classes by actually typing their names? What's wrong with that. That's how you normally use an API and it is a clear indication of what you are doing. Macros are useful for reflection in the asserts and to add the FILE and LINE for you. It is however totally non-intuitive to know what TEST and TEST_F mean and these things don't work if you do them wrong. For example, can I use their asserts in free-functions called from my test functions? – Thing

@Thing Remember C++ doesn't have the kind of reflection most language have. I suggest you reach me directly by email/IM so we can discuss more in details if you like. – Kirkland

I know C++ doesn't have reflection and therefore macros are useful for the asserts. I have written a testing library. It uses dependency injection like GTEST does. It has macros for asserts but not for anything else, and there's a Predicate class in the API. The macros are there for convenience. You have to #include their header to use them, they are not included for you. – Thing

I second this. Currently using Boost.Test and suffering from lack of more assert operations and a way to properly output xUnit conforming xml files to nicely integrate with TeamCity. – Sasin

Answer 3

I've just pushed my own framework, Catch2, out there. It's still under development but I believe it already surpasses most other frameworks. Different people have different criteria but I've tried to cover most ground without too many trade-offs. Take a look at my linked blog entry for a taster. My top five features are:

• Header only
• Auto registration of function and method based tests
• Decomposes standard C++ expressions into LHS and RHS (so you don't need a whole family of assert macros).
• Support for nested sections within a function based fixture
• Name tests using natural language - function/method names are generated

It also has Objective-C bindings. The project is hosted on Github

Leucas

doctest is my reimplementation of Catch with a huge focus on compilation speed - checkout the FAQ to see how they differ – Inglis

I'm really at a loss when comparing all the testing frameworks (one of which I now have to choose). Would you write your own answer comparing and contrasting doctest with Catch and other offerings? – Quintillion

@Quintillion I would if I could - but the question is closed as being not constructive and I can write only a comment... – Inglis

You should add: no fall through on failures, meaning the tests will stop immediately. Google test proudly declare to be exceptions free, but they fall through on failures if you don't check return codes (at least this is the default behavior), meaning that the test continues if you don't do it. I found this laughable. – Koeninger

Answer 4

Boost Test Library is a very good choice especially if you're already using Boost.

// TODO: Include your class to test here.
#define BOOST_TEST_MODULE MyTest
#include <boost/test/unit_test.hpp>

BOOST_AUTO_TEST_CASE(MyTestCase)
{
    // To simplify this example test, let's suppose we'll test 'float'.
    // Some test are stupid, but all should pass.
    float x = 9.5f;

    BOOST_CHECK(x != 0.0f);
    BOOST_CHECK_EQUAL((int)x, 9);
    BOOST_CHECK_CLOSE(x, 9.5f, 0.0001f); // Checks differ no more then 0.0001%
}

It supports:

• Automatic or manual tests registration
• Many assertions
• Automatic comparison of collections
• Various output formats (including XML)
• Fixtures / Templates...

PS: I wrote an article about it that may help you getting started: C++ Unit Testing Framework: A Boost Test Tutorial

Kirkland

I used to use Boost test and liked it except that it seemed to change significantly between release. It was difficult enough selling unit testing to my client without having to spend more of my time (and their money) fixing the tests when the API changed, than fixing the code it was meant to be testing. In the end I ditched it and wrote my own - this was about 5 years ago though. – Cystocele

Answer 5

Wikipedia has a comprehensive list of unit testing frameworks, with tables that identify features supported or not.

Unsnarl

Answer 6

I've recently released xUnit++, specifically as an alternative to Google Test and the Boost Test Library (view the comparisons). If you're familiar with xUnit.Net, you're ready for xUnit++.

#include "xUnit++/xUnit++.h"

FACT("Foo and Blah should always return the same value")
{
    Check.Equal("0", Foo()) << "Calling Foo() with no parameters should always return \"0\".";
    Assert.Equal(Foo(), Blah());
}

THEORY("Foo should return the same value it was given, converted to string", (int input, std::string expected),
    std::make_tuple(0, "0"),
    std::make_tuple(1, "1"),
    std::make_tuple(2, "2"))
{
    Assert.Equal(expected, Foo(input));
}

Main features:

• Incredibly fast: tests run concurrently.
• Portable
• Automatic test registration
• Many assertion types (Boost has nothing on xUnit++)
• Compares collections natively.
• Assertions come in three levels:
  • fatal errors
  • non-fatal errors
  • warnings
• Easy assert logging: Assert.Equal(-1, foo(i)) << "Failed with i = " << i;
• Test logging: Log.Debug << "Starting test"; Log.Warn << "Here's a warning";
• Fixtures
• Data-driven tests (Theories)
• Select which tests to run based on:
  • Attribute matching
  • Name substring matchin
  • Test Suites

Renelle

The question is asking for comparison. IMO, it is vital to present what are the differences between your framework and, at least, the two popular ones: googletest and Boost. Especially, if you advertise xUnit++ as alternative to those two. Will be +1 if updated :) – Ardell

Fair enough. :) I've already got a comparison table on the wiki, but I will try to sum up a few of the differences directly in my answer. – Renelle

I decided to just link the wiki table directly, it was cluttering up the summary to list it all out. – Renelle

Also written in macro language.. What's wrong with using C++ instead of FACT and THEORY macros? – Thing

Personally, I really hate macros. I've considered writing v2.0 using method calls and lambdas, I just don't write any C++ these days and haven't gotten around to it. – Renelle

has your project been discontinued? Last commit dates back to 09/2015... Anyway, great answer. Thanks. – Maraca

Answer 7

CppUTest - very nice, light weight framework with mock libraries. Worthwhile taking a closer look.

Screed

Answer 8

CPUnit (http://cpunit.sourceforge.net) is a framework that is similar to Google Test, but which relies on less macos (asserts are functions), and where the macros are prefixed to avoid the usual macro pitfall. Tests look like:

#include <cpunit>

namespace MyAssetTest {
    using namespace cpunit;

    CPUNIT_FUNC(MyAssetTest, test_stuff) {
        int some_value = 42;
        assert_equals("Wrong value!", 666, some_value);
    }

    // Fixtures go as follows:
    CPUNIT_SET_UP(MyAssetTest) {
        // Setting up suite here...
        // And the same goes for tear-down.
    }

}

They auto-register, so you need not more than this. Then it is just compile and run. I find using this framework very much like using JUnit, for those who have had to spend some time programming Java. Very nice!

Penalty

Answer 9

API Sanity Checker — test framework for C/C++ libraries:

An automatic generator of basic unit tests for a shared C/C++ library. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files.

The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code and program hanging.

Unique features in comparison with CppUnit, Boost and Google Test:

• Automatic generation of test data and input arguments (even for complex data types)
• Modern and highly reusable specialized types instead of fixtures and templates

Aggrieved