Measure Code Coverage only on New Code

Question

We are looking for a creative way to measure code coverage on new code separate from existing code. We have a large legacy project and want to start getting 90+% coverage on any new functionality. We would like a way to easily view a report that filters out any older code to make sure the new functionality is meeting our goal. Obviously still looking a increasing overall coverage on the project, but need a non-manual way to give us feedback on the new code activity. We have this working for Static analysis since we can look at the dates on the source files. Since Cobertura is analyzing the class files they have new dates and this technique doesn't work.

Any Ideas?

Stack:

Java 1.5 JUnit Cobertura Hudson

Towny

Answer 1

We had a similar situation.. wanted new code tested but could not test all old code at once. What we did is not exactly what you asked, but may give you an idea.

We have a file called linecoverage.standard, and a file called branchcoverage.standard that live on the build server (and local copies). They have a number inside with the current line and branch coverage limits. If the checked in code is below the standard, it fails the build. If it is at the standard it passes the build. If it is ABOVE the standard, a new standard is written equal to the current coverage.

This means our code coverage will never get worse, and should slowly go up. If new code is 90%, the coverage will keep creeping up. You could also set a goal like raise the standard by 1 each week until it gets to your final goal (90%). Having to add a few tests a week to old code is not a bad idea, if it is spread out over enough time.

Our current coverage is up to 75%ish... pretty good coming from a 0% rate under a year ago.

Dillion

Answer 2

Have a look on emma.sourceforge and associated Eclipse plugin here (if you are using Eclipse)

I think this tool can answer to your need by selecting exactly what to test for coverage.

Juliannajulianne

I didn't see how EMMA can be set separate coverage measurement on new code from existing code. Could you explain? – Calibre

Answer 3

I did this for a large C++ project by using svn blame combined with the output of gcov. If you zip those two results together you have revision information and coverage information for each line. I actually loaded this all into a database to do queries (e.g. show me all the uncovered lines written by joe since r1234). If you only want an aggregate number you can just avoid counting 'old' uncovered lines in your total.

Ottoman

Answer 4

IMO the best option is to split the codebase into "new" and "legacy" sections. Then either run test coverage analysis only on the "new" section, or ignore the results for the "old" section.

The two best ways to accomplish this are a) split the codebase into two source trees (two projects with a dependency between), or b) maintain two separate package hierarchies in a single project.

Two separate projects is probably preferable, but it might not be possible if there's a cyclical dependency between the legacy codebase and the new codebase (old code depends on new code and new code depends on old code). If you can manage it, a one-way dependency between old and new code will also make the combined codebase easier to understand.

Once you've got this done, either adjust cobertura so that it's only analyzing the bits you want, or at least just focus on the "new" part of the codebase. One additional tip is that in this scheme, it's best to move bits of code from the "legacy" section to the "new" section as you refactor/add tests to them (if code is frequently moving in the other direction, that's not so good :-).

Nitrogenous

Answer 5

We did it as below using sonar.exclusions property: We use Sonar to display the code coverage reports (reported by Cobertura).

a) Identify the classes that you don't want coverage report on (Legacy classes) Use your SCM cmd line client. eg: p4 files //depot/... @2000/01/01,@2013/07/13 git log --until="5 days ago"

Direct this list into a file. You will need to do some parsing based on the SCM tool you use and your destination file should contain one file name per line.

eg. the destination file is excludeFile.list should look like below:
abc.java
xyz.java
...

b) Now..when you integrate with Sonar (from Jenkins Job), use the below property.

    -Dsonar.exclusions=<filename>

And your final coverage report in Sonar contains only your new classes ( added after 07/13 in the above example).

Raylenerayless

Answer 6

We call what you are trying to do Test Gap analysis. The idea is to test all (or at least most of) the changes you make to a large software system during development, because that's where the most bugs will be. There's empirical evidence to back up this intuition as well!

Teamscale is a tool that does what you are looking for and it can handle Cobertura reports. The advantage is, that you just measure coverage as you normally do and then upload the reports to Teamscale, which will perform the Test Gap analysis to highlight new/changed but untested code on a method-by-method basis.

Full disclaimer: I work for CQSE, the company that makes Teamscale.

Gyroscope

Answer 7

In my scenario, we need to have the measureament of new code in our day-to-day process. What we did was install sonarquobe locally where the developers can check their code quality control, such as the code coverage of the new code as the sonar can provide to us, and make actions right away.

For a global metrics, we implemented sonarquobe for only our production code and we gather from there all of the quality metrics (such as new code coverage)

Purposeless