How do I find all the unit tests that may directly or indirectly call a given method? (.net)

Question

How do I find all the unit tests that may directly or indirectly call a given method? When I change a method, I wish to know the best tests to run; there must be a tool for this!

As we have lots of interfaces, I am interested in all unit tests that calls a method on an interface when there is at least one path var the implantation method on a class that implements the interface.

Or in other words, I want a list of all unit tests when the tool cannot prove the result is not affected by the method I have changed.

(We are using nUnit on .net and have lots of slow unit tests, it will be many year until we have refactored all our unit tests to be fast)

see also:

• How to find Junit tests that are using a given Java method directly or indirectly

Egotist

I know it's not a tool, but when I need to do that, I rename the method and try to compile, then let the debugger tell me what fails. That only works for me when in Visual Studio, though, so the helpfulness is limited to when you're actively debugging rather than on a build server. – Embellishment

@David, that only finds direct callers. – Egotist

To me it sounds like a smell. Unit tests are called Unit because they test one unit separately from the others. So generally you should have a class tested in a way when no other class' implementation will affect the test. It also means that when you change anything in a class you should run only unit-tests for that class, because other tests should use mocks instead of this class. – Galyak

@Snowbear, sorry I live in the real world working on a large old code base in a big team – Egotist

I have created a tool that does that for Python: github.com/jbochi/bazinga – Pindling

Answer 1

Visual Studio 2010 has this exact feature: http://blogs.msdn.com/b/phuene/archive/2009/12/07/test-impact-analysis-in-visual-studio-2010.aspx

Since you're using nunit, you could try this technique to run MSTest as well: http://msdn.microsoft.com/en-gb/magazine/cc163643.aspx#S4

Denoting

Answer 2

In the most general case, when you use delegates, lambda expressions, etc., I have the intuition that this problem is equivalent to checking if a given program will finish (the Halting Problem), so I wouldn't hope to find a reasonable answer.

If your real goal is to be able to quickly test if your change broke something, I would recommend:

• refactoring your tests
• setting up a parallel build infrastructure with a cluster of build machines (easier than you might think with modern tools like TeamCity)

Kicksorter

The halting problem can be solve in a lot of useful realworld cases, it just needs 3 outputs, yes|no|maybe - the fact that we can prove there is not general solution, does not show that there is no solution for a lot of useful cases. – Egotist

And for cases in which the answer is "maybe", you can conservatively (depending on your needs) push it to "treat as if yes" or "treat as if no". For test coverage, if the outcome is "maybe this test checks this code", you can treat it like "yes, run the test again". – Fanatical

Answer 3

What you need is a relation between each test and the code it exercises.

This can be computed statically but it is hard and I don't know any tools that do it. Worse, if you had such a tool, the static analysis to decide what the test affected, might actually take longer than just running the test itself, so this doesn't look like an attractive direction.

However, this can be computed using a test coverage tool. For each individual test, run that test (we presume it passes) and collect the test coverage data. We now have a lot of pairs (t_i,c_i) for "test i has coverage c".

When the code base changes, the test data coverage sets can be consulted. A simple check: if for any (t_i,c_i), if c_i mentions file F, and F has changed, you need to run t_i again. Given test coverage data in almost any representation this is sort of easy to detect in the abstract. Given that most test coverage tools don't specifically tell you how they store the test coverage data, this is harder than it looks in practice.

Actually, what you want ideally is if c_i mentions any programmatic element of F, and that programmatic element has changed, you need to run t_i again.

Our SD Test Coverage tools provide this capability for Java and C#, at the level of methods. You need to set up some scripting to associate the actual test, however you have packaged it, with collected test coverage vectors. As a practical matter this tends to be pretty easy.

Fanatical

Thanks for the great answer; I was starting to think along these lines. Is this the same as what Visual Studio 2010 does with its Test Impact Analysis? Fortunately we don’t have much code that is attribute (and/or config file) driven so it is unlikely that a change to a method that is not executed by a test would affect the outcome of the test. – Egotist

@Ian Ringrose: AFAIK, VS 2010 basically uses instrumentation, as does our tool, to collect this coverage data. I don't know how they associate tests with test coverage data, nor do I understand the programmatic element level at which it gives answers; is it file? method? statement?. Our tools can do this at the method level because they actually parse the text and do a compare. – Fanatical

Answer 4

You can use VS2010's "View Call Hierarchy" or ReSharpers "ReSharper -> Inspect -> Incoming calls". You can even export the result from ReSharper.
If you want to use automation to achieve your goal, i.e. because this is not a one-time thing, but something you want to integrate into your build process, I suggest, you build your own solution. You could use Reflection or Mono.Cecil on the assemblies and walk the call hierarchy yourself. However, this can be quite time consuming, because you would need to create a complete call hierarchy tree over all your assemblies. Another possibility would be to create a Visual Studio or ReSharper plugin, so you can access the object model they create from your source code.
Basically, what I am saying is: I don't know of any currently existing method to achieve your goal via automation and writing your own solution will be fun but tricky and possibly time consuming, either when developing or running it.

Gemology

Does that cope with calls may be made ver interfaces and delegates etc? Also how do I get it to output in a format I can pass into a unit test runner? – Egotist

Hm, I am not sure it handles these cases well. I just tested it on a bigger project and it looks "strange". Anyway, you can't export that tree, so it might not be that helpful to you, if you want to use some automation. – Gemology