How can I set up code coverage threshold as a high water mark in TeamCity?

Question

I have a TeamCity build that captures code coverage for unit tests. I have also defined an environment variable for the minimum code coverage for the build to succeed, which works fine but I don't like to maintain this threshold manually. The question I have is whether there is a way (outside of publishing code coverage statistics somewhere outside of TeamCity and then reading the results from the last successful build) to automatically adjust the threshold as code coverage improves to ensure that it's a steady improvement without allowing backsliding :)?

For example, suppose the current code coverage is 20% (a legacy application) and as new unit tests are written, code coverage improves to 25%. Then, someone checks in new code without unit tests and code coverage drops to 24%. I would want TeamCity to fail the build because code coverage dropped from 25% to 24%.

Jacy

Answer 1

This is an old question but I wanted to mention that this is now possible in newer versions of TeamCity through the "Failure Conditions" feature. Failure conditions can use constants as well as compare to previously generated metrics. In this case, a failure condition would look like this:

Jacy

Answer 2

I have some pet theories about code coverage which I'd like to explain first before I get on to answering the question.

First some context:

• There are many sorts of Code coverage, but I'm only going to talk about line coverage, but you should be able to substitute a different sort.
• From the question: "...someone checks in new code without unit tests and code coverage drops..." which is related to the similar: "Some one (refactors/eliminates duplication/replaces an algorithm and) removes tested code and coverage drops."
• Coverage should be measured as the result of running a single suite of tests. That is, not by running the application and stimulating it from the outside.
• Coverage as a percentage is very misleading.
  I had a think about this and really you only want to know how many lines of code are NOT covered.
  See my comment to this answer: Ensure minimal coverage on new Subversion commits
• Coverage should be as high as possible. The question talks about "...improvement without allowing backsliding..."
• 100% coverage is possible.
  I've done it, albeit with a library.

I have a theory that you should divide your code into just two divisions as far as code coverage is concerned:

1. A division where all the code is 100% covered.
2. A division where no code is covered.

Either division could be constituted by a number of projects, but members of a division should be files (given that both Java and C# have source files) and preferably whole folders of files. You could have one set of project in the first division and another set in the second division.

Now the report of lack of coverage is just the number of lines in the second division.

The mode of operation should be that you are testing your code as you go and code simply falls into the 100% coverage division. However, if you find a tricky piece of code which your brain just can't find a way to test, you should refactor so that the bits that aren't tested move into the second division. Alternatively you may get a brainwave and be able to find a test which raises the second division above 0%, at which point you refactor the code over to the first division. This means that every check-in maintains my theoretical invariant.

Now, back to the question:
No, I don't know TeamCity at all apart from a brief look at the JetBrains website, so I don't know how to update the coverage, but according to my theory it should be 100% or nothing, so can you set limits per project? If you can, then a fixed limit of 100% works for the first division.
If you can get two divisions you may want to do the automatic update thing with a lines of code metric for the second division, progressively lower is better.

Grummet

+1 good answer - I've similar theories myself and I've acheived 100% coverage for many libraries in our code base. – Yokoyokohama

Answer 3

After you run the tests, see the coverage and update the environment variable to the nearest five not greater than,perhaps?

If your threshold was initially 25 and the coverage jumps to 31, update it to 30 and it will now break if the threshold falls below 30. Of course you can make it 31 when the jump occurs.

So after every test run, you see if the current coverage is greater than the current threshold and set the threshold to the new value?

Centralize

Thanks -- that's exactly what I'm doing currently but it's a manual process and I would like to automate it. – Jacy

I thought I said the automated way to do it! In teamcity, whatever your build scripts are, after you run the tests, also update based on the coverage value. If you can give me more details on your build script, how you call ncover, etc I will tell you the exact automated way to do it! – Centralize

Sorry -- I didn't see the "automated" implication :). The build runner is MSBuild and I don't have a problem exporting the code coverage stats. The threshold is currently set up as an environment variable in TeamCity. Thanks for your help! – Jacy