How to run multiple QTest classes?

Question

I have a subproject where I put all my QTest unit tests and build a stand-alone test application that runs the tests (i.e. I run it from within Qt Creator). I have multiple test classes that I can execute with qExec(). However I don't know what is the proper way to execute multiple test classes.

Currently I do it in this way (MVCE):

tests.pro

QT -= gui
QT += core \
    testlib

CONFIG += console
CONFIG -= app_bundle
TEMPLATE = app
TARGET = testrunner

HEADERS += test_foo.h
SOURCES += main.cpp

main.cpp

#include <QtTest>
#include <QCoreApplication>
#include "test_foo.h"

int main(int argc, char** argv) {
    QCoreApplication app(argc, argv);

    TestFooClass testFoo;
    TestBarClass testBar;
    // NOTE THIS LINE IN PARTICULAR.
    return QTest::qExec(&testFoo, argc, argv) || QTest::qExec(&testBar, argc, argv);
}

test_foo.h

#include <QtTest>

class TestFooClass: public QObject
{
    Q_OBJECT
private slots:
    void test_func_foo() {};
};

class TestBarClass: public QObject
{
    Q_OBJECT
private slots:
    void test_func_bar() {};
};

However the documentation for qExec() says this is the wrong way:

For stand-alone test applications, this function should not be called more than once, as command-line options for logging test output to files and executing individual test functions will not behave correctly.

The other major downside is that there is no single summary for all the test classes, only for individual classes. This is a problem when I have dozens of classes that each have dozens of tests. To check if all tests passed I have to scroll up to see all the "Totals" of what passed/failed of each class, e.g.:

********* Start testing of TestFooClass *********
PASS   : TestFooClass::initTestCase()
PASS   : TestFooClass::test_func_foo()
PASS   : TestFooClass::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of TestFooClass *********
********* Start testing of TestBarClass *********
PASS   : TestBarClass::initTestCase()
PASS   : TestBarClass::test_func_bar()
PASS   : TestBarClass::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of TestBarClass *********

I'm also surprised my qExec() || qExec() works considering that the documentation says if a test failed qExec() returns a non-zero value, which should mean all the following qExec() calls wouldn't happen, but this seems not to be the case.

What is the proper way to run multiple test classes? And so that I can see at a glance if any of the hundreds of unit tests I have have failed.

Voleta

Have you made any progress regarding this problem? The best I could find so far was the answer to a similar question. – Saturnian

@Saturnian The solutions are either 1) The guys in #qt told me the proper way is to have 1 test project per test class (you can put them in a TEMPLATE = subdirs). Then the testrunner combines the test results. 2) Use GoogleTest, which is superior in many ways. – Voleta

Hi, I have the same issue. Why should I create one project per test class. This is ridiculous. I don't want to switch to CMake :(. Is there still no solution with Qt 5 or 6? I actually prefer your solution to the one posted as answer. – Range

The qExec() || qExec() works because all tests in your example pass, hence all qExec() calls return 0 (i.e. false), hence the || operator continues to evaluate the right-hand side. Regarding the issue with the single summary, see my answer below. – Abacus

Does this answer your question? Qt: How to organize Unit Test with more than one class? – Greige

Answer 1

I once found a nice solution using a plain Qt project (no TEMPLATE = subdirs) which uses a macro approach for creating the main function and automatic registering of all test classes (macro, too) with only a simple helper header file.

Here is a sample test class (only the relevant header file):

#ifndef FOOTESTS_H
#define FOOTESTS_H

#include "AutoTest.h"

class FooTests : public QObject
{
    Q_OBJECT
    private slots:
        void initTestCase();
        void test1();
        void test2();
        void cleanupTestCase();
};

DECLARE_TEST(FooTests)

#endif // FOOTESTS_H

and the main, which consumes every test class created this way:

#include "AutoTest.h"

TEST_MAIN

The code of AutoTest.h:

#ifndef AUTOTEST_H
#define AUTOTEST_H

#include <QTest>
#include <QList>
#include <QString>
#include <QSharedPointer>

namespace AutoTest
{
 typedef QList<QObject*> TestList;

 inline TestList& testList()
 {
  static TestList list;
  return list;
 }

 inline bool findObject(QObject* object)
 {
  TestList& list = testList();
  if (list.contains(object))
  {
   return true;
  }
  foreach (QObject* test, list)
  {
   if (test->objectName() == object->objectName())
   {
    return true;
   }
  }
  return false;
 }

 inline void addTest(QObject* object)
 {
  TestList& list = testList();
  if (!findObject(object))
  {
   list.append(object);
  }
 }

 inline int run(int argc, char *argv[])
 {
  int ret = 0;

  foreach (QObject* test, testList())
  {
   ret += QTest::qExec(test, argc, argv);
  }

  return ret;
 }
}

template <class T>
class Test
{
public:
 QSharedPointer<T> child;

 Test(const QString& name) : child(new T)
 {
  child->setObjectName(name);
  AutoTest::addTest(child.data());
 }
};

#define DECLARE_TEST(className) static Test<className> t(#className);

#define TEST_MAIN \
 int main(int argc, char *argv[]) \
 { \
  return AutoTest::run(argc, argv); \
 }

#endif // AUTOTEST_H

All credits goes to Rob Caldecott.

Cato

I had to add QApplication app(argc, argv); to the TEST_MAIN makro (like in the original QTEST_MAIN makro line 263) so the target does not show the command line help – Marcelina

It is better to put the macro DECLARE_TEST(FooTests) into a cpp file instead of header to prevent creation of two instances of the class although there is a protection from the double registration in AutoTest.h. If there is a cpp implementation of FooTests the header is included into two different cpp files (there is always one auto generated moc file). – Neuroblast

Documentation of QTest::qExec says: "this function should not be called more than once". – Solana

Answer 2

To execute multiple test classes contained in a single test project, including auto-detection of test classes (by simply deriving from TestClass) and having a nice summary printed at the end, use the following code:

FooTestClass.h

#include "TestClass.h"

class FooTestClass : public TestClass {
    Q_OBJECT
private:
    Q_SLOT void fooTest1() { }
    Q_SLOT void fooTest2() { }
};

// Note: C++17 style inline variable. In C++11, use a static variable instead.
inline FooTestClass fooTests;

BarTestClass.h

#include "TestClass.h"

class BarTestClass : public TestClass {
    Q_OBJECT
private:
    Q_SLOT void barTest1() { }
    Q_SLOT void barTest2() { }
};

// Note: C++17 style inline variable. In C++11, use a static variable instead.
inline BarTestClass barTests;

main.cpp

#include "TestClass.h"

int main(int argc, char **argv) {
    return TestClass::runAllTests(argc, argv);
}

TestClass.h

#include <QDebug>
#include <QObject>
#include <QString>
#include <QTest>

class TestClass : public QObject {
    Q_OBJECT

private:

    static QObjectList &testObjects() { static QObjectList testObjects; return testObjects; }

public:

    TestClass() { testObjects().append(this); }

    static int runAllTests(int argc, char **argv) {
        // Sort test objects by class name.
        std::sort(testObjects().begin(), testObjects().end(), [] (const QObject *a, const QObject *b) {
            return strcmp(a->metaObject()->className(), b->metaObject()->className()) < 0;
        });

        // Run all tests.
        QStringList results;
        int passed = 0;
        for (QObject *testObject : testObjects()) {
            bool success = false;
            try {
                success = QTest::qExec(testObject, argc, argv) == EXIT_SUCCESS;
            } catch (...) { }
            qDebug() << "";
            results << QString("%1  : %2").arg(success ? "PASS " : "FAIL!").arg(testObject->metaObject()->className());
            passed += success ? 1 : 0;
        }

        // Print summary.
        int tested = testObjects().size();
        results << QString("Totals: %1 tested, %2 passed, %3 failed").arg(tested).arg(passed).arg(tested - passed);
        results << QString("Result: %1").arg(tested == passed ? "All tests PASSED." : "Some tests FAILED!");
        qDebug() << "********* Start of summary *********";
        for (const QString &line : results) {
            qDebug() << line.toUtf8().data();
        }
        qDebug() << "********* End of summary *********";
        qDebug() << "";

        return tested == passed ? EXIT_SUCCESS : EXIT_FAILURE;
    }
};

Output

********* Start testing of BarTestClass *********
Config: Using QtTest library 5.15.12, Qt 5.15.12 (x86_64-little_endian-llp64 shared (dynamic) release build; by GCC 13.2.0), windows 10
PASS   : BarTestClass::initTestCase()
PASS   : BarTestClass::barTest1()
PASS   : BarTestClass::barTest2()
PASS   : BarTestClass::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped, 0 blacklisted, 2ms
********* Finished testing of BarTestClass *********

********* Start testing of FooTestClass *********
Config: Using QtTest library 5.15.12, Qt 5.15.12 (x86_64-little_endian-llp64 shared (dynamic) release build; by GCC 13.2.0), windows 10
PASS   : FooTestClass::initTestCase()
PASS   : FooTestClass::fooTest1()
PASS   : FooTestClass::fooTest2()
PASS   : FooTestClass::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped, 0 blacklisted, 1ms
********* Finished testing of FooTestClass *********

********* Start of summary *********
PASS   : BarTestClass
PASS   : FooTestClass
Totals: 2 tested, 2 passed, 0 failed
Result: All tests PASSED.
********* End of summary *********

Limitations of this solution

The documentation of QTest::qExec says that this function shouldn't be called more than once because "command-line options for logging test output to files and executing individual test functions will not behave correctly".

That's true, but it's only a minor issue. Just keep in mind that some of the QtTest command line options won't work as intended. If you don't pass any command line options at all, then you're fine in any case.

Abacus