Getting DUnit Test Coverage stats using FinalBuilder and AQTime

Question

We have a large Delphi project (1.5 million lines of code), and we're moving to using agile processes.

We already have a continous integration environment (FinalBuilder) which I've updated to include unit tests (dUnit) and code metrics (CodeHealer) in the e-mails to everyone in our development team. Our unit test coverage isn't great, so I'm now trying to get AQtime into the mix for some test coverage results on every build.

I'm using the "Execute Program" task to run the unit test executable, log the results and parse the file afterwards. I intend to use the "Run Script" task to run AQtime (via COM) and export the results to XML so I can parse through those results.

The issue I have is with AQtime running the unit test executable, I lose the ability to monitor the unit test executable directly. I'd like to get FinalBuilder to parse the results of both tasks. Does anybody know how to get access to the dUnit results when it's called from AQtime?

Immaculate

Answer 1

We ran down this same path, and we don't run DUnit test from AQTime,

Instead we build and run our Dunit Tests using FinalBuilder.

Our unit tests use XmlTestRunner, then we can know if the test fails or not very easily using an XPath Query on the resulting XML File.

Update: An recent update to FinalBuilder 7.0 now supports DUnit. It's through its own XMLTestrunner.pas, which is under the FinalBuilder Directory after install. This runner outputs the test results in the same format as NUnit. It then integrates really well into FinalBuilder Server.

Embroil

Thanks Robert. The version of dUnit that was in our build didn't have XMLTestRunner. I downloaded dUnit 9.3 from sourceforge and grabbed XMLTestRunner from the Contrib folder. Does everything I wanted it to do. – Immaculate

Catharz, you should accept an answer if it has been answered for you. – Embroil

Answer 2

You are kind of describing the setup we are slowly evolving to.

• DUnit tests are compiled as console applications using the TTextTestListener defined in TextTestRunner unit.
• CI server is a cmd script that builds all projects and executes all tests.
• The output of the tests are piped to a file.

A solution might be to have AQTime profile these console applications while still be able to pipe the results to a file that can be parsed afterwards?!

Another solution might be to implement your own TestListener object and have that object write the testresults to the eventlog, directly to a logfile, a database or wherever you like and have this picked up by FinalBuilder.

Instead of having something like this in your project file

  Application.Initialize;
  if System.IsConsole then TextTestRunner.RunRegisteredTests
  else GUITestRunner.RunRegisteredTests;

it would become something like this

  Application.Initialize;
  if System.IsConsole then OurEventLogTestRunner.RunRegisteredTests
  else GUITestRunner.RunRegisteredTests;

Extensive

Thanks for the response Lieven. I tried getting AQtime to pipe the results of the dUnit executable to a file, but that didn't work. I also had a look at implementing my own TestListener, which looked pretty easy. But after looking at the XMLTestRunner from dUnit 9.3, that looked even easier. – Immaculate

Also, I would suggest you change your code to: TextTestRunner.RunRegisteredTests(rxbHaltOnFailures) so that you get an error code <> 0 when some test fails. Excellent for continuous integration. – Armenian

@marius: thx, I didn't know that. Right now, we do a grep search in the test results to find the amount of errors/failures. – Extensive