How can I measure CppUnit test coverage (on win32 and Unix)?

Question

I have a very large code base that contains extensive unit tests (using CppUnit). I need to work out what percentage of the code is exercised by these tests, and (ideally) generate some sort of report that tells me on a per-library or per-file basis, how much of the code was exercised.

Here's the kicker: this has to run completely unnatended (eventually inside a continuous integration build), and has to be cross platform (well, WIN32 and *nix at least).

Can anyone suggest a tool, or set of tools that can help me do this? I can't change away from CppUnit (nor would I want to - it kicks ass), but otherwise I'm eager to hear any recommendations you might have.

Cheers,

Shanly

Answer 1

Which tool should I use?

This article describes another developers frustrations searching for C++ code coverage tools. The author's final solution was Bullseye Coverage.

Bullseye Coverage features:

• Cross Platform Support (win32, unix, and embedded), (supports linux gcc compilers and MSVC6)
• Easy to use (up and running in a few hours).
• Provides "best" metrics: Function Coverage and Condition/Decision Coverage.
• Uses source code instrumentation.

As for hooking into your continuous integration, it depends on which CI solution you use, but you can likely hook the instrumentation / coverage measurement steps into the make file you use for automated testing.

---

Testing Linux vs Windows?

So long as all your tests run correctly in both environments, you should be fine measuring coverage on one or the other. (Though Bullseye appears to support both platforms). But why aren't you doing continuous integration builds in both environments?? If you deliver to clients in both environments then you need to be testing in both.

For that reason, it sounds like you might need to have two continuous build servers set up, one for a linux build and one for a windows build. Perhaps this can be easily accomplished with some virtualization software like vmware or virtualbox. You may not need to run code coverage metrics on both OSs, but you should definitely be running your unit tests on both.

Dealer

Answer 2

If you can use GNU GCC as your complier, then the gcov tool works well. It's very easy to fully automate the whole process.

Seasonseasonable

Answer 3

If you are using the GCC toolchain, gcov is going to get you source, functional, and branch coverage statistics. gcov works fine for MinGW and Cygwin. This will allow you to get coverage statistics as well as emitting instrumented source code that allows you to visualize unexecuted code.

However, if you really want to hit it out of the park with pretty reports, using gcov in conjunction with lcov is the way to go. lcov will give you bar reports scoped to files and directories, functional coverage statistics, and color coded source file browsing to show coverage (green means executed, red means not...).

lcov is easy on Linux, but may require some perl hacking on Cygwin. I personally have had some problems executing the scripts (lcov is implemented in perl) on Windows. I've gotten a hacked up version to work, but be forewarned.

Another approach is doing the gcov emit on windows, and doing the lcov post processing on Linux, where it will surely work out of the box.

Edric

Today LCOV is really easy to install on Windows if you use MSYS2 and pacman package manager. The PKGBUILD from Archlinux works just fine. I put a step-by-step guide on how to get it running on my blog: txt.arboreus.com/2015/05/29/… – Ornstead

Answer 4

Check out our SD C++ Test Coverage tool. It can be obtained for GCC, and for MSVC6.

It has low overhead probe data collection, a nice display of coverage data overlayed on your code, and complete report generation with rollups on coverage across the method/class/file/directory levels.

EDIT: Aug 2015: Now supports GCC5 and various MS dialects through Visual Studio 2015. To use these tools under Linux, you need Wine, but there the tools provide Linux-native sh scripting and a Linux/Java based UI, so the tool feels like a native Linux tool there.

Ortega

Answer 5

I guess I should have specified the compiler - we're using gcc for Linux, and MSVC 6 (yeah I know, it's old, but it works (mostly) for us) for WIn32.

For that reasons, gcov won't work for our Win32 builds, and Bullseye won't work for our Linux builds.

Then again maybe I only need coverage in one OS...

Shanly

Unless you have a lot of OS specific code, measuring the coverage on Linux ought to be enough. – Czar

This answer is really a comment, but it is so old I'm not sure comments even worked then :-). Anyway, in case someone else wanders by like me, Bullseye works on Linux, and it looks like the "use Bullseye" was the accepted answer, so either this comment is just wrong about that, or there is another reason it didn't work for you. – Aimeeaimil