How to discover which test unit checks which lines of code?

Asked 21/6, 2011 at 6:45 Answered 4/2, 2022 at 7:46

Solved c#testing nunit code-coverage ncover

I was fooling around the NUint, hoping to discover a way to realize which line of code passes in which test.

Imagine I have a method for which I have 3 tests. Is there any way to find out which test checks which line of code?

Having used NCover, I know you can find out which lines have been tested and which have not. However, you really can't see which unit checked that code.

It can be really useful when dealing with tons of tests...

Mallorymallow answered 21/6, 2011 at 6:45 Comment(5)

Can you explain why it is useful to you to see which test covers which lines of code? – Puklich 21/6, 2011 at 7:8

plz have a look at GeertvdC answer comments – Mallorymallow 21/6, 2011 at 8:1

I don't think there's anything that does that... Does it matter where a piece of code is tested so long as it's tested? :) If you want to find out if it's actually tested or which tests exercise it, comment it out and run your tests. The tests that fail are the ones you're looking for. – Allen 22/6, 2011 at 2:48

Actually it does matter. If two tests substantially test the same code, you can run just one of them. When you have thousands of tests, eliminating redundant tests can materially reduce the total test execution time. If you can trace coverage of an area you changed to a particular set of test, then when you change that area, you can run just those tests. This means an individual contributor can run the tests relevant to him when he makes a change. – Sourdough 22/6, 2011 at 4:25

I've come to this question because I've discovered a bug which is due to a line which according to NCover is covered by exactly one test. It would really be handy if it could tell me which test that is, so I can fix it... – Minim 15/6, 2012 at 8:33

JetBrains dotCover can help you to get the info you are looking for. It integrates with the ReSharper unit test runner and collects per-test data. You can see which code is covered by the each particular test (with corresponding highlighting in VS). And what is more interesting, for every piece of code you can get list of tests which cover it and easily rerun them.

Additional info is available here: http://blogs.jetbrains.com/dotnet/2010/07/show-covering-test-with-dotcover/

Traject answered 5/9, 2011 at 8:11 Comment(1)

Thanks ;) hope NCover would implement the same thing someday! :) – Mallorymallow 5/9, 2011 at 12:31

Hi as most people have responded the only way to do this at the moment with most coverage tools is to run each test by itself and then view the coverage for each test.

However it is possible, but would require a bit of effort, using OpenCover. If you captured and stored all the sequence points from the tests and the targets in the order they arrive and then analyse these results on a test by test basis, this is one of the original aims of OpenCover - https://github.com/sawilde/opencover/wiki. Currently OpenCover just aggregates these results and then throws the visitation data away but they could be stored. Of course you would have to be aware of any parallel running tests should they exist. The amount of data can be daunting and is on of the reasons it has not yet been implemented.

Hereford answered 27/6, 2011 at 9:40 Comment(2)

Thanks Shaun, any hope to be implemented at all? :-) – Mallorymallow 28/6, 2011 at 9:25

It's in the pipeline - time is my enemy, and may need to use an alternate store which can then be mined later. – Hereford 28/6, 2011 at 12:16

You can debug your testing. If you are using NUnit you can attach the process to your VS.

Jocko answered 21/6, 2011 at 6:49 Comment(1)

in case of existing many throw exceptions in your business layer you can't do that. cause you almost never reach that code! unless you comment all of the throwing exceptions – Mallorymallow 21/6, 2011 at 6:56

The term you are looking for is "Code coverage".

I don't have any experience how to do this with NUnit but there seems to be a codeplex project making this available for NUnit: http://codecoveragerunner.codeplex.com/

if you want to see which test is testing which line of code you could put a breakpoint on the line of code and debug your tests? When you hit your breakpoint check which test is

when having other tests throwing exceptions you can disable the break on exception option:

in visual studio in the menu go to "Debug" -> "Exceptions" and uncheck the break on exceptions. then you dont have to stop at every exception.

Innutrition answered 21/6, 2011 at 6:52 Comment(7)

code coverage just tells you which line has been tested! not which test unit checked it – Mallorymallow 21/6, 2011 at 6:58

ok then i didn't understand your question correctly. but why does it really matter which line is tested by which test? i presume you want to run every test every time. Unit tests should be fast otherwise people will stop using the pretty soon. – Innutrition 21/6, 2011 at 7:3

honestly, there is a piece of the code that I can't find out where it's been tested. there is a very very same other piece of code which is not tested. So I'm trying to find that unit test which are it's among like 1000 other tests. And as long as there are quiet many throwing exception, I can't use debug unless I comment all of them which is n't rational – Mallorymallow 21/6, 2011 at 7:27

put a breakpoint on the line of code and debug your tests? When you hit your breakpoint check which test is running? – Innutrition 21/6, 2011 at 7:31

I should run the test altogether. and some of them are catching exceptions. and when you have throwing exception in your code, the debug with stops there! so that you never can get exactly to the point you want! – Mallorymallow 21/6, 2011 at 8:1

in visual studio in the menu go to "Debug" -> "Exceptions" and uncheck the break on exceptions. then you dont have to stop at every exception. – Innutrition 21/6, 2011 at 8:14

@Javid: The method that @Innutrition describes (changing VS settings and setting a breakpoint) is what I do quite often. Very useful. – Puklich 21/6, 2011 at 10:21

NCover is a tool you can use to calculate coverage of your unit tests. See NCover.com.

Roughdry answered 21/6, 2011 at 6:54 Comment(1)

code coverage just tells you which line has been tested! not which test unit checked which lines – Mallorymallow 21/6, 2011 at 7:0

The easiest way is to download and install a personal edition of TestDriven.NET which runs your unit tests, and has an option to run and report the code coverage using NCover. Mind you newer NCover versions are commercial but the one included in TD.NET is a little bit older but free to use.

Witkowski answered 21/6, 2011 at 7:2 Comment(1)

my question is something else! ;-) – Mallorymallow 21/6, 2011 at 7:14

I don't know of anything that directly achieves matches what you are looking for. I recently did a project on automated fault localization where we needed this exact functionality, and the only solution we came up with was to roll our own test runner that also gathered coverage information for each method.

There may be an indirect way to achieve this using Visual Studio 2010 (Premium and Ultimate), which introduced a Test Impact Analyzer. This allows you to determine which tests are affected changes to your source code. However, this only works for MSTest. You can use the technique in this blog post to allow your NUnit tests to run under MSTest.

Manville answered 21/6, 2011 at 20:35 Comment(0)

The simple answer is, "run each test by itself and collect test coverage data for that test".

How you organize that may depend on the specific test coverage tool, and how you choose to run your tests.

For our Test Coverage tools, there is an explicit "TestCoverageDump" method added to your software by the test coverage instrumentation step. Normally a call to this method is inserted in the "main" program of your application so that when it exits, you get test coverage data for whatever tests you have run.

For your task, you want to modify your unit test running code to make an explicit call, after each test, on "TestCoverageDump" followed by "TestCoverageReset" (also inserted), so that each test gets its own vector. How you choose to associate the name of the test with the vector is completely under you control, at the price of a tiny bit of adjustment to the "TestCoverageDump" code supplied in source.

Our test coverage display tool can easily combine all the individual vectors to give you an overall view. Or you can view the coverage for any particular vector. The display tool will also let you compare coverage from different tests (where do they intersect? what does one test, that the other does not?)

Having this per-test coverage data also lets you determine which tests you need to run again. If you modify the code, and re-run the instrumenter, it will tell you which test coverage vectors (e.g., which tests) need to be run again based on what modified code that the vector previously covered.

Sourdough answered 22/6, 2011 at 2:23 Comment(0)

If you are using Visual Studio, you should check out the extension "Run Coverlet Report" by Chris Dexter. https://marketplace.visualstudio.com/items?itemName=ChrisDexter.RunCoverletReport

Apart from the above comprehensive report, it can also do code highlighting where you can see which code lines are fully covered, partially covered and which are not.

Metrology answered 4/2, 2022 at 7:46 Comment(0)

Recommended topics

Hot tags