What unit-testing framework should I use for Qt? [closed]

Question

I am just starting up a new project that needs some cross-platform GUI, and we have chosen Qt as the GUI-framework.

We need a unit-testing framework, too. Until about a year ago we used an in-house developed unit-testing framework for C++-projects, but we are now transitioning to using Google Test for new projects.

Does anyone have any experience with using Google Test for Qt-applications? Is QtTest/QTestLib a better alternative?

I am still not sure how much we want to use Qt in the non-GUI parts of the project - we would probably prefer to just use STL/Boost in the core-code with a small interface to the Qt-based GUI.

EDIT: It looks like many are leaning towards QtTest. Is there anybody who has any experience with integrating this with a continous integration server? Also, it would seem to me that having to handle a separate application for each new test case would cause a lot of friction. Is there any good way to solve that? Does Qt Creator have a good way of handling such test cases or would you need to have a project per test case?

Dorcia

Would be great to have an update on this topic (2021). Seems that QSignalSpy is still available doc-snapshots.qt.io/qt6-dev/qsignalspy.html How are the mocking support in Qtest, which I have the most concerns in and therefore I would rather go with Gtest. – Radionuclide

Answer 1

I don't know that QTestLib is "better" than one framework for another in such general terms. There is one thing that it does well, and that's provide a good way to test Qt based applications.

You could integrate QTest into your new Google Test based setup. I haven't tried it, but based on how QTestLib is architected, it seems like it would not be too complicated.

Tests written with pure QTestLib have an -xml option that you could use, along with some XSLT transformations to convert to the needed format for a continuous integration server. However, a lot of that depends on which CI server you go with. I would imagine the same applies to GTest.

A single test app per test case never caused a lot of friction for me, but that depends on having a build system that would do a decent job of managing the building and execution of the test cases.

I don't know of anything in Qt Creator that would require a seperate project per test case but it could have changed since the last time I looked at Qt Creator.

I would also suggest sticking with QtCore and staying away from the STL. Using QtCore throughout will make dealing with the GUI bits that require the Qt data types easier. You won't have to worry about converting from one data type to another in that case.

Joettejoey

I'd just like to point out for posterity that much has changed in the 8+ years since the original question was posted. In 2017, choosing googletest/mock over QTestLib seems to be a much more viable choice than previously. As Exhibit A, here's a webinar from ICS entitled Qt Test-Driven Development Using Google Test & Google Mock. For me, using googletest/mock together with QSignalSpy has been really effective, I can't see any reason to ever go back. – Advancement

Hey, so I was looking for unit testing in QT and the vod on given link is not working for me or it has been taken down. Do you perhaps have any available resources in form of anything I could go through to get even a general idea of how to start about it? – Sevigny

Answer 2

You don't have to create separate tests applications. Just use qExec in an independent main() function similar to this one:

int main(int argc, char *argv[])
{
    TestClass1 test1;
    QTest::qExec(&test1, argc, argv);

    TestClass2 test2;
    QTest::qExec(&test2, argc, argv);

    // ...

    return 0;
}

This will execute all test methods in each class in one batch.

Your testclass .h files would look as follows:

class TestClass1 : public QObject
{
Q_OBJECT

private slots:
    void testMethod1();
    // ...
}

Unfortunately this setup isn't really described well in the Qt documentation even though it would seem to be quite useful for a lot of people.

Emilyemina

I like this approach, but Im getting the error Unknown test function: 'test()'. Possible matches: firstTest(), any tip? – Palaeozoology

qExec should not be called more than once: because it breaks command line support. When you write a test with QtTest, you can list all functions of the test with the -functions option, and run an individual function by passing it on the command line. This breaks if qExec is called more than once since only the first call will handle command line options. – Cherlynchernow

Answer 3

I started off using QtTest for my app and very, very quickly started running into limitations with it. The two main problems were:

1) My tests run very fast - sufficiently quickly that the overhead of loading an executable, setting up a Q(Core)Application (if needed) etc often dwarfs the running time of the tests themselves! Linking each executable takes up a lot of time, too.

The overhead just kept on increasing as more and more classes were added, and it soon became a problem - one of the goals of unit tests are to have a safety net that runs so fast that it is not a burden at all, and this was rapidly becoming not the case. The solution is to glob multiple test suites into one executable, and while (as shown above) this is mostly do-able, it is not supported and has important limitations.

2) No fixture support - a deal-breaker for me.

So after a while, I switched to Google Test - it is a far more featureful and sophisticated unit testing framework (especially when used with Google Mock) and solves 1) and 2), and moreover, you can still easily use the handy QTestLib features such as QSignalSpy and simulation of GUI events, etc. It was a bit of a pain to switch, but thankfully the project had not advanced too far and many of the changes could be automated.

Personally, I will not be using QtTest over Google Test for future projects - if offers no real advantages that I can see, and has important drawbacks.

Sialagogue

Any advice for how you combine Qt and gtest? IE: do you still have a QApplication or QMainWindow? Do you embed your tests directly in main, or in a member function of some descendant of QObject? – Chisel

@Chisel Any end-to-end/ integration tests that require QWidgets have a QApplication; this QApplication is created (but not exec()'d) in main(), before I RUN_ALL_TESTS(). QMainWindow can be used, but I mainly only use it in my end-to-end tests. The tests themselves follow the standard Google Test scheme and I generally have e.g. all unit tests for class X in a file called Xtests.cpp. So it's essentially a standard gtest project, with a few concession to Qt (creation of a QApplication before running the tests in the normal way). – Sialagogue

I don't know if it's recent but Qt Test allow adding two private slots init and cleanup which are called before and after each test function respectively. – Choreography

Answer 4

To append to Joe's answer.

Here's a small header I use (testrunner.h), containing an utility class spawning an event loop (which is, for example, needed to test queued signal-slot connections and databases) and "running" QTest-compatible classes:

#ifndef TESTRUNNER_H
#define TESTRUNNER_H

#include <QList>
#include <QTimer>
#include <QCoreApplication>
#include <QtTest>

class TestRunner: public QObject
{
    Q_OBJECT

public:
    TestRunner()
        : m_overallResult(0)
    {}

    void addTest(QObject * test) {
        test->setParent(this);
        m_tests.append(test);
    }

    bool runTests() {
        int argc =0;
        char * argv[] = {0};
        QCoreApplication app(argc, argv);
        QTimer::singleShot(0, this, SLOT(run()) );
        app.exec();

        return m_overallResult == 0;
    }
private slots:
    void run() {
        doRunTests();
        QCoreApplication::instance()->quit();
    }
private:
    void doRunTests() {
        foreach (QObject * test, m_tests) {
            m_overallResult|= QTest::qExec(test);
        }
    }

    QList<QObject *> m_tests;
    int m_overallResult;
};

#endif // TESTRUNNER_H

Use it like this:

#include "testrunner.h"
#include "..." // header for your QTest compatible class here

#include <QDebug>

int main() {
    TestRunner testRunner;
    testRunner.addTest(new ...()); //your QTest compatible class here

    qDebug() << "Overall result: " << (testRunner.runTests()?"PASS":"FAIL");

    return 0;
}

Ebon

good work! should be QCoreApplication again in private slot run... – Drill

Thanks, fixed! Not tested yet, but will be soon (as I am using a variation of this code whenever I need tests :)) – Ebon

... One question: Is there a reason why you used QTimer instead of QMetaObject::invokeMethod with Qt::QueuedConnection? I think that would be more legible IMHO. – Revkah

@Revkah Why queue the connections, if the object will have affinity to the same thread as the QTimer anyways (which should be the GUI thread, as creating qApp-s outside that is probably prohibited)? :) – Ebon

I believe the slot is queued into the main event loop either way, in which case explicit queuing is more direct (and legible). – Revkah

@Revkah Check out the docs on how queued and auto (default) connections work, you may be quite wrong in your assumptions :) – Ebon

Answer 5

Why not using the unit-testing framework included in Qt? An example : QtTestLib Tutorial.

Betti

"Is QtTest/QTestLib a better alternative?" ... I think that is the question :-P – Revkah

Link is dead. This is why link-only answers are bad. – Austriahungary

Link is fixed with standard QT doc website instead of previous Nokia one – Betti

Answer 6

I unit tested our libraries using gtest and QSignalSpy. Use QSignalSpy to catch signals. You can call slots directly (like normal methods) to test them.

Onomatopoeia

Answer 7

QtTest is mostly useful for testing parts that require the Qt event loop/signal dispatching. It's designed in a way that each test case requires a separate executable, so it should not conflict with any existing test framework used for the rest of the application.

(Btw, I highly recommend using QtCore even for non-GUI parts of the applications. It's much nicer to work with.)

Thacker

Answer 8

To extend mlvljr's and Joe's solution we can even support complete QtTest options per one test class and still run all in a batch plus logging:

usage: 
  help:                                        "TestSuite.exe -help"
  run all test classes (with logging):         "TestSuite.exe"
  print all test classes:                      "TestSuite.exe -classes"
  run one test class with QtTest parameters:   "TestSuite.exe testClass [options] [testfunctions[:testdata]]...

Header

#ifndef TESTRUNNER_H
#define TESTRUNNER_H

#include <QList>
#include <QTimer>
#include <QCoreApplication>
#include <QtTest>
#include <QStringBuilder>

/*
Taken from https://mcmap.net/q/345234/-what-unit-testing-framework-should-i-use-for-qt-closed
BEWARE: there are some concerns doing so, see  https://bugreports.qt.io/browse/QTBUG-23067
*/
class TestRunner : public QObject
{
   Q_OBJECT

public:
   TestRunner() : m_overallResult(0) 
   {
      QDir dir;
      if (!dir.exists(mTestLogFolder))
      {
         if (!dir.mkdir(mTestLogFolder))
            qFatal("Cannot create folder %s", mTestLogFolder);
      }
   }

   void addTest(QObject * test)
   {
      test->setParent(this);
      m_tests.append(test);
   }

   bool runTests(int argc, char * argv[]) 
   {
      QCoreApplication app(argc, argv);
      QTimer::singleShot(0, this, SLOT(run()));
      app.exec();

      return m_overallResult == 0;
   }

   private slots:
   void run() 
   {
      doRunTests();
      QCoreApplication::instance()->quit();
   }

private:
   void doRunTests() 
   {
      // BEWARE: we assume either no command line parameters or evaluate first parameter ourselves
      // usage: 
      //    help:                                        "TestSuite.exe -help"
      //    run all test classes (with logging):         "TestSuite.exe"
      //    print all test classes:                      "TestSuite.exe -classes"
      //    run one test class with QtTest parameters:   "TestSuite.exe testClass [options] [testfunctions[:testdata]]...
      if (QCoreApplication::arguments().size() > 1 && QCoreApplication::arguments()[1] == "-help")
      {
         qDebug() << "Usage:";
         qDebug().noquote() << "run all test classes (with logging):\t\t" << qAppName();
         qDebug().noquote() << "print all test classes:\t\t\t\t" << qAppName() << "-classes";
         qDebug().noquote() << "run one test class with QtTest parameters:\t" << qAppName() << "testClass [options][testfunctions[:testdata]]...";
         qDebug().noquote() << "get more help for running one test class:\t" << qAppName() << "testClass -help";
         exit(0);
      }

      foreach(QObject * test, m_tests)
      {
         QStringList arguments;
         QString testName = test->metaObject()->className();

         if (QCoreApplication::arguments().size() > 1)
         {
            if (QCoreApplication::arguments()[1] == "-classes")
            {
               // only print test classes
               qDebug().noquote() << testName;
               continue;
            }
            else
               if (QCoreApplication::arguments()[1] != testName)
               {
                  continue;
               }
               else
               {
                  arguments = QCoreApplication::arguments();
                  arguments.removeAt(1);
               }
         }
         else
         {
            arguments.append(QCoreApplication::arguments()[0]);
            // log to console
            arguments.append("-o"); arguments.append("-,txt");
            // log to file as TXT
            arguments.append("-o"); arguments.append(mTestLogFolder % "/" % testName % ".log,txt");
            // log to file as XML
            arguments.append("-o"); arguments.append(mTestLogFolder % "/" % testName % ".xml,xunitxml");
         }
         m_overallResult |= QTest::qExec(test, arguments);
      }
   }

   QList<QObject *> m_tests;
   int m_overallResult;
   const QString mTestLogFolder = "testLogs";
};

#endif // TESTRUNNER_H

own code

#include "testrunner.h"
#include "test1" 
...

#include <QDebug>

int main(int argc, char * argv[]) 
{
    TestRunner testRunner;

    //your QTest compatible class here
    testRunner.addTest(new Test1);
    testRunner.addTest(new Test2);
    ...

    bool pass = testRunner.runTests(argc, argv);
    qDebug() << "Overall result: " << (pass ? "PASS" : "FAIL");

    return pass?0:1;
}

Sera

Answer 9

If you are using Qt, I would recommend using QtTest, because is has facilities to test the UI and is simple to use.

If you use QtCore, you can probably do without STL. I frequently find the Qt classes easier to use than the STL counterparts.

Frayda

Answer 10

I've just been playing around with this. The main advantage of using Google Test over QtTest for us is that we do all our UI development in Visual Studio. If you use Visual Studio 2012 and install the Google Test Adapter you can get VS to recognise the tests and include them in its Test Explorer. This is great for developers to be able to use as they write code, and because Google Test is portable we can also add the tests to the end of our Linux build.

I'm hoping in the future that someone will add support for C++ to one of the concurrent testing tools that C# have, like NCrunch, Giles and ContinuousTests.

Of course, you might find someone writes another adapter for VS2012 that adds QtTest support to Test Adapter in which case this advantage goes away! If anyone is interested in this there's a good blog post Authoring a new Visual studio unit test adapter.

Luxury

Answer 11

For Visual Studio test adapter tool support with the QtTest framework use this Visual Studio extension: https://visualstudiogallery.msdn.microsoft.com/cc1fcd27-4e58-4663-951f-fb02d9ff3653

Thermoluminescent

Welcome to Stack Overflow! While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. See How do I write a good answer. – Adopt