Java code coverage in Hudson

Question

I'm migrating a couple of projects from an ant build to a maven one. The build server is , and will remain, Hudson.

I've been having trouble recording code coverage in hudson with cobertura due to the tests run and recorded twice problem.

The project is multi-module and it would be nice, although not required, to have an aggregated output of the code coverage data.

All in all, the solution I'm looking for must:

• run automated tests for all modules and record the results once;
• display the individual module code coverage in Hudson;
• be easily configured once for the whole project, not in every module.

The solution can be based on Cobertura, or Emma, or any other java code coverage tool.

---

Update: Running the tests with Emma still duplicates the results and there's no merge capability, so it's not really usable with multi-module builds.

Membranophone

Answer 1

Sonar is a very cool tool that is easily integrated with Hudson, I really like its organization with multi-module projects. You should give it a try

alt text http://sonar.codehaus.org/wp-content/uploads/2009/08/dashboard.png

Downbow

Answer 2

It's a bit hackish, but the approach I'm using is to use a modified version of the Maven cobertura plugin (which is available from their repo). It provides a cobertura:generate-report target, so that you can insert cobertura:instrument and cobertura:generate-report into your lifecycle before and after tests run, respectively. That'll get you the coverage data you want without the duplicate test execution/recording.

The underlying problem is that all the non-clover Maven coverage plugins I've run into are built around the idea of running the tests with coverage separately from the main test execution in the Maven lifecycle. This, obviously, results in two sets of test executions. If you're using a freestyle project, you'll only get one set of tests recorded (since, even with two test executions, there's only one copy of the test output), but the Maven project type actually intercepts the Maven mojo executions and records test output/results at test execution time, rather than all at once at the end of the build as freestyle projects do. This has plenty of advantages, but it also has the rather glaring disadvantage that a single test getting executed twice gets counted as two tests.

That said, while I've seen strong arguments for running tests against both non-instrumented and instrumented code, I prefer to only run the tests once, against instrumented code - not just because of the Maven/Hudson issues, but because when you've got tests that take 45 minutes, it seems quite frankly silly to run them twice to generate the same result.

Yearling

Links not working any more, maybe you could update them? Thanks! – Woodsman

Answer 3

Robert,

I had this problem as well and found that Hudson does not double report if you setup the project as a freestyle project rather than a Maven2 project. You do lose some of the niceness of having a maven2 project, but for us, it was a trade we had to make.

Jeff

Abney

Thanks for the answer. It's good to know and I'll have this as the fallback option. – Membranophone

Sure. Also, I'm not 100% sure, but I believe what happens is that the instrumented code ends up on the classpath before the uninstrumented code and then when unit tests are run by the surefire plugin, it runs the instrumented code which results in the results being calculated twice. If you have other plugins that run unit tests, you will see the number of tests multiplied again. You can run your build with mvn -X to get debug info which should give the classpath. – Abney

Answer 4

We use free style projects and don't have this problem, so as indicated, this may be the source of your problem.

To provide the merging features, we created our own artifact repository (we aren't using Maven). At the end of each build, we copy the cobertura.ser file to a network share, renaming it in the process. We have a consolidated view job that copies all of the cobertura files and the source code files (another build artifact copied to the network share) into the local build directory and generate the Cobertura report.

The lack of a standard artifact repository within Hudson is a bit frustrating, but it makes sense give the authors typically use Maven for those needs. Our build process runs over multiple servers so we can't just use relative paths into the other job directories.

Note, we do the same thing for other metrics: test results, JavaNCSS, ect. and joined either using the correct tools or some custom code.

We use the same repository for traditional build artifacts: DLLs, JARs, installation scripts.

Armandinaarmando

Answer 5

Have you considered Atlassian's Clover?

The maven-clover2-plugin has a new goal: clover2:setup which will simply instrument your tests without forking the lifecycle, or running the tests twice.

You would define the goals to run in Hudson like so:

mvn clover2:setup verify clover2:aggregate clover2:clover

The maven-clover2-plugin is absolutely free to try for 30 days.

Henley

Thanks for the reply. I did consider Clover, but AFAIK it does not work with the M2 Hudson job, only with the freestyle one. – Membranophone

Hi Robert, I will look into that. It definitely works if you use a freestyle maven job though. Cheers, Nick – Henley

Answer 6

See SD Java Test Coverage for an extremely low overhead tool with a nice GUI. I'm not sure I understand your "run twice" problem, but if you ran (the same deterministic) tests twice with the SD tools, you'll get the the same test coverage data, e.g., its idempotent. If your tests are nondeterministic, you'll get two different test runs, but these tools easily merges the results of several runs into a single overall summary.

They also handle extremely large applications, and handle multiple threaded applications pretty well (small timing splinters can make the answer slightly inaccurate in theory, but practice this simply isn't an issue).

Erme

Thanks for the answer. Is there by any change a maven plugin for this tool as well? I was unable to find one. – Membranophone