Regression testing in C++

Question

I can't use unit tests for some parts of code so I'm falling back to regression tests. I would like to check whether my program behaves in the same way after some modifications. And by behaviour I mean mostly a state of data structures. So far I was serializing them into human readable text format and dumped to some files in the first run. Then in the next dumps I could compare whether the state changed or not. And update it if the change comes from a new feature and not from a bug.

I could use a library (C++) to organize all that. Do you know any? Together with dump files it would provide a cheap, massive unit-test.

The most cumbersome thing are the serialization procedures. Sometimes I just dump memory state, but when it is different it's hard to reverse engineer. So I moved to another method. Now, during the compare phase I read a memory dump into a "phantom" object and run a specialized diff method (operator== with rich error reporting), which sometimes is easier to write than serializing to human readable text format.

Basically I feel like reinventing the wheel, so my questions are quite general:
How do you perform regression testing (if you do)?
Do you use any library/toolkit?
Have you ever implemented one for your own needs?

And just out of curiosity:
Have you ever wanted to do regression testing, but something stopped you?

Deconsecrate

Just out of curiosity... why can't you use unit testing? – Myxoma

Mostly because the system works on massive amount of data and it is difficult to write fixtures to test subroutines. – Rhizocarpous

Answer 1

You might take a look at the Boost Test Library . I have never used it, but it might fulfill your desires.

When i am working i usually write simple test cases in our test-tracker to make regression tests as easy as possible (especially for not-me people :).

Capri

Answer 2

Check out the boost serialization library. This will allow you to dump your files to xml which you can then diff for each version.

In order to create your test cases, if you're not already doing so, use gcov to work out the coverage though your functions. This will at least ensure that you've got all statements covered in your function.

Unit Testing

When we're testing legacy code that is missing fixtures, we sometimes use a middle ground approach. What we have is a slightly modified version of our application, which performs the usual tasks, but can also perform tests on functions.

It's not unit testing in the strict sense as you have to assume that the rest of the library is correct, however, it does allow you at least to do more testing than may be possible with regression/system tests.

Harmony

Answer 3

OK, there's three things being discussed here: fitting tests onto legacy code, unit testing and acceptance/regression testing, all in C++.

First, for taking legacy code and fitting tests to it, I recommend purchasing a copy of "Working Effectively with Legacy Code" by Michael Feathers. Its an awesome book and will teach you that any legacy code can be unit tested! I've used techniques from that book to fit tests onto things that everyone told me couldn't be unit tested, but I did it anyway :-).

Second, for unit testing in C++, I just wrote a 5-part series of blog posts that describe how to do this in detail with Visual Studio: C++ Unit Testing With Boost.Test.

Finally, for acceptance/regression testing I've had success useing Fitnesse. This is basically an acceptance test framework that uses a wiki to organize and write the tests. The wiki page is traversed and parsed to turn into calls into a test fixture that you write. The test fixture then mediates between the test, as described by a wiki page, and your production code. I've used this mechanism to perform regression tests on the whole application end-to-end. Combine that with unit tests for the classes you're changing and its a very powerful bug detector mechanism. The regression tests squeeze from above and the unit tests squeeze from below and the bugs are caught in the middle! Its worked great for me.

Get the main fitnesse distribution from fitnesse.org You can get C++ test runner for FitNesse from sourceforge. (I'm a developer on that project.) We haven't added the SLIM support yet that is in the main fitnesse wiki, but we do have good support for several fixtures. We're hoping to add SLIM support soon. I have a rudimentary implementation that needs to be finished off.

Laresa