.NET 4.0 code contracts - How will they affect unit testing?

Question

For example this article introduces them.

What is the benefit?

Static analysis seems cool but at the same time it would prevent the ability to pass null as a parameter in unit test. (if you followed the example in the article that is)

While on the topic of unit testing - given how things are now surely there is no point for code contracts if you already practice automated testing?

Update

Having played with Code Contracts I'm a little disappointed. For example, based on the code in the accepted answer:

public double CalculateTotal(Order order)
{
    Contract.Requires(order != null);
    Contract.Ensures(Contract.Result<double>() >= 0);
    return 2.0;
}

For unit testing, you still have to write tests to ensure that null cannot be passed, and the result is greater than or equal to zero if the contracts are business logic. In other words, if I was to remove the first contract, no tests would break, unless I had specifically had a test for this feature. This is based on not using the static analysis built into the better (ultimate etc...) editions of Visual Studio however.

Essentially they all boil down to an alternate way of writing traditional if statements. My experience actually using TDD, with Code Contracts shows why, and how I went about it.

Psychognosis

Why do you still have write tests to ensure that null cannot be passed? Is it because you don't feel you can trust the static verifier? It's probably worth noting that order being null has no effect on the functionality of that method, so I'd personally treat the presence of that parameter as a bug. If you were actually using the order to perform some calculation, then I'd expect some other test to break with a NullReferenceException. – Eudo

@StephenJ.Anderson +1 for the nice code review comment on the order. – Guyette

Answer 1

I don't think unit testing and contracts interfere with each other that much, and if anything contracts should help unit testing since it removes the need to add tedious repetitive tests for invalid arguments. Contracts specify the minimum you can expect from the function, whereas unit tests attempt to validate the actual behaviour for a particular set of inputs. Consider this contrived example:

public class Order
{
    public IEnumerable Items { get; }
}

public class OrderCalculator
{
    public double CalculateTotal(Order order)
    {
        Contract.Requires(order != null);
        Contract.Ensures(Contract.Result<double>() >= 0);

        return 2.0;
    }
}

Clearly the code satisfies the contract, but you'd still need unit testing to validate it actually behaves as you'd expect.

Steeplebush

You really don't need to unit test what the contract specifies if you have static verification tools. Visual Studio 2010 Team edition will include those tools for .NET code contracts. If the static verifier doesn't emit any warnings or errors then the code satisfies the contract. – Fortyniner

Unit testing and contracts are not mutually exclusive. Contracts do things testing can't do, and testing is need to ensure the contracts are correct and calling code adheres to them. Just because you have a Requires statement does NOT mean you shouldn't have a test for that condition. – Shoat

I've come to the conclusion that writing tests to make sure your contracts are "correct" is not worthwhile. Just the fact that you're creating contracts means you're thinking whether or not null is a valid parameter. Testing the preconditions is taking focus away from what's really important; what the function actually does. I think the answer could be improved by adding an example of putting a contract on an interface. Would you really build a unit test for all implemenations to make sure null is rejected? So, +1 for this answer. – Wonderland

I think that code contracts presented in the example are ok, but they're not complete. As if you'd write unit test that tests for the same. Sum to be more than or equal to 0. This means that your contracts aren't complete. AFAIK contracts should be just as complete because they should provide all pre/post conditions under which the code execution will provide expected results. I think your code is not really a good example of contracts usage. I urge you to also check what NUnit theories do because they're essentially very similar to contracts, it's just that they're presented as tests. – Lovemaking

I completely agree with @wekempf. What if I fat-fingered a contract condition? Is the contract correct? No. Would I know it? Maybe. But maybe's not good enough. Nothing like a unit test will actually tell me if the code is working as I expect. Sure, I could mistype a unit test, too. But the odds are, that test will fail eventually because of it. But with a contract, my code may never fail and yet, the very conditions I meant to require could end up never being met due to a typo. – Generable

Saying what if I fat finger my code contract is a bad argument. You can fat finger unit tests and not get adequate code coverage with unit tests as well. Unit tests and still allow buggy code. The awesome thing about code contracts if they can be applied to interfaces and the static check will check all implementations without duplicating tests. – Pitchford

Answer 2

What is the benefit?

Let's say that you want to make sure that a method never returns null. Now with unit tests, you have to write a bunch of test cases where you call the method with varying inputs and verify that the output is not null. Trouble is, you can't test all possible inputs.

With code contracts, you just declare that the method never returns null. The static analyzer will then complain if it is not possible to prove that. If it doesn't complain, you know that your assertion is correct for all possible inputs.

Less work, perfect correctness guarantees. What's not to like?

Fortyniner

Code contracts are basically unit tests within code itself... The problem I'm having with them is how do you write contracts for an algorithm? Say you have a checksum validator method ValidateChecksum(inputWithChecksum). To provide complete contracts you'd have to recreate the algoritm just for the sake of assuring what code does. And if your algorithm has a bug, your contracts will likely have it as well. So what gives? – Lovemaking

@RobertKoritnik: Contracts don't have to specify the full behavior, it's already useful to just specify and prove certain properties of the code (like not returning null). To make sure that a calculation is giving exactly the expected outputs, unit tests are still the way to go. – Fortyniner

Answer 3

Contracts allow you say what the actual purpose of the code is, as opposed to letting whatever the code does with whatever random arguments are handed it standing as the definition from the point of view of the compiler, or the next reader of the code. This allows significantly better static analysis and code optimization.

For instance, if I declare an integer parameter (using the contract notation) to be in the range of 1 to 10, and I have a local array in my function declared the same size, that is indexed by the parameter, the compiler can tell that there is no possibility of subscript error, thus producing better code.

You can state that null is valid value in a contract.

The purpose of unit testing is to verify dynamically that the code achieves whatever stated purpose it has. Just because you've written a contract for a function, doesn't mean the code does that, or that static analysis can verify the code does that. Unit testing won't go away.

Fils

+1 You are right on. Contracts are helpful for other developers and static/runtime analysis, always presuming the contracts are correct--which you can't really know unless you've done proper unit testing. And beyond the contracts, unit testing verifies that even if contract conditions are met, the code still does what you intended it to do. – Generable

Answer 4

Well it will not interfere with unit-testing in general. But as I saw you mentioned something about TDD.

If I think about it from that perspective I guess it could/may change the procedure from the standard one

• create method (just signature)
• create Unit test -> implement the test
• run the test: let it fail
• implement the method, hack it to the end just to make it working
• run the test: see it pass
• refactor your (possibly messy) method body
• (re-run the test just to see you've not broken anything)

This would be the really hard-full-featured unit-testing procedure. In such a context I guess you could insert code contracts between the 1st and 2nd point like

• create method (just signature)
• insert code contracts for the methods input parameters
• create Unit test -> implement the test
• ...

The advantage I see at the moment is that you can write easier unit tests in the sense that you wouldn't have to check every possible path since some is already taken into account by your defined contracts. It just gives you additional checking, but it wouldn't replace unit testing since there will always be more logic within the code, more path that have to be tested with unit tests as usual.

Edit

Another possibility I didn't consider before would be to add the code contracts in the refactoring part. Basically as additional way of assuring things. But that would somehow be redundant and since people don't like to do redundant stuff...

Ottava

Would you still have tests that test what is being confirmed via the contracts? – Psychognosis

I guess I wouldn't test those things that have been assured by the contracts already. Would just be "duplicate" work somehow. Therefore I also placed them before actually writing the unit test in the TDD cycle. – Ottava

@Juri, what if you mis-typed a contract condition, e.g. Contract.Requires(x < 2)...oops, meant x < 1? Without a test, how would you know if your code is really doing what you think it's doing? I don't believe contracts are enough. Contracts are not a replacement for testing. With IDE support, they can let other users know what pre-/post-conditions can be expected (presuming those conditions are correct). And a run-time/static checker can perform analysis (again, presuming the contracts are correct). Beyond that, you still need tests. – Generable

If you were to write a bad contract what is going to stop you from writing a bad test? – Pitchford