How come you cannot catch Code Contract exceptions?

Question

System.Diagnostics.Contracts.ContractException is not accessible in my test project. Note this code is purely myself messing around with my shiney new copy of Visual Studio, but I'd like to know what I'm doing wrong.

I'm using the professional edition of VS, therefore I do not have static checking. In order to still use code contracts (which I like) I figured the only way my method can work is to catch the exception that is thrown at runtime, but I'm not finding this possible.

TestMethod

[TestMethod, ExpectedException(typeof(System.Diagnostics.Contracts.ContractException))]
public void returning_a_value_less_than_one_throws_exception()
{
    var person = new Person();
    person.Number();
}

Method

public int Number()
{
    Contract.Ensures(Contract.Result<int>() >= 0);
    return -1;
}

Error

Error 1 'System.Diagnostics.Contracts.ContractException' is inaccessible
due to its protection level.

Edit

After some more thought I've come to the conclusion discussed in the comments, as well as the following. Given a method, if this had a requirement which could be expressed in Code Contract form, I'd write tests as such.

[TestMethod]
[ExpectedException(typeof(ArgumentException))]
public void value_input_must_be_greater_than_zero()
{
    // Arrange
    var person = new Person();
    // Act
    person.Number(-1);
}

This would ensure the contract is part of the code, and will not be removed. This would require the Code Contract to actually throw the specified exception however. In some cases this would not be required however.

Concession

I should add this doesn't compile, naturally. – Concession

I would really like to see NUnit deprecate ExpectedExceptionAttribute. Exception Asserts give you far more control. – Mont

@Mont Your wish has finally come true with NUnit 3.0: github.com/nunit/nunit/wiki/Breaking-Changes#body – Daunt

Answer 1

This is deliberate - although a slight pain for testing.

The point is that in production code you should never want to catch a contract exception; it indicates a bug in your code, so you shouldn't be expecting that any more than arbitrary unexpected exceptions which you may want to catch right at the top of your call stack so you can move onto the next request. Basically you shouldn't view contract exceptions as ones which can be "handled" as such.

Now, for testing that's a pain... but do you really want to test your contracts anyway? Isn't that a bit like testing that the compiler stops you from passing in a string to a method which has an int parameter? You've declared the contract, it can be documented appropriately, and enforced appropriately (based on settings, anyway).

If you do want to test contract exceptions, you can either catch a bare Exception in the test and check its full name, or you can mess around with the Contract.ContractFailed event. I would expect unit testing frameworks to have built-in support for this over time - but it'll take a little while to get there. In the meantime you probably want to have a utility method to expect a contract violation. One possible implementation:

const string ContractExceptionName =
    "System.Diagnostics.Contracts.__ContractsRuntime.ContractException";

public static void ExpectContractFailure(Action action)
{
    try
    {
        action();
        Assert.Fail("Expected contract failure");
    }
    catch (Exception e)
    {
        if (e.GetType().FullName != ContractExceptionName)
        {
            throw;
        }
        // Correct exception was thrown. Fine.
    }
}

Blakeley

I see what you're getting at. And you make a good point about unit testing support in the future. In my example method, the usage of a code contract is pointless then? If this was a real method, normal code/test code would be more suitable. – Concession

@Finglas: I personally don't think contracts need testing - at least unless they're complex, and they usually shouldn't be. However, that's only a personal view, and you should absolutely expect to hear others. Think about what I've written, but please don't feel any obligation to agree :) Hopefully the code in the answer will help you if you feel you do want to test the contract. – Blakeley

@Jon - no, it's actually just clicked. See: #1384035 The answer and comments here make sense now I've actually played with CC. My tests would fail if a contract is not met, then my tests would pass if both my contract is satisfied and my test is satisfied. This is in contrast to testing the contract, which is what I was doing prior. – Concession

It's worth noting that Pex will actually add tests for contract failures. – Agraffe

@Jon Skeet: I'm not sure if I agree that contracts don't need testing. More specifically, it's not the contract that is being tested, but rather the behavior that the method will throw when a contract is violated. Let's say the specification for a method void M(string s) requires that M throw if s is null or white space. One way to implement this is to use Contract.Requires. Another way is the classic if-throw construct, or possibly use a built-in Guard.Against. The point is, a test shouldn't be testing the contract, but that M meets the specification. – Aparejo

@Jon Skeet: So, I don't think that the specific type of exception should be tested for (because that's an implementation detail), but rather just that an exception is thrown when the condition is violated (unless the specification is that a specific type of exception is thrown). – Aparejo

@Jason: But the contract is the specification. If it's expressed declaratively, in a way which can be documented automatically, viewed etc, what's the benefit of the test? I can understand if there's a complex contract which may need some examples to explain, but I can't see the benefit for a simple null argument. – Blakeley

@Jon Skeet: Disagree. The contract is an implementation of part of the specification. So if I'm to implement a method void M(string s) that throws if s is null or white space and otherwise prints s to the console, that is the specification. I could fail to fully implement the specification by neglecting to include a Contract.Requires or an if-throw construct. The point of the unit tests is to ensure that I fully implement the specification, as well as protect against future refactoring and maintenance. – Aparejo

@Jason: Do you also test that the method can't be called with the wrong argument types, too? I mean surely the method signature is part of the specification, too... so do you have a unit test which uses reflection to try to call a method taking a string, but using an int argument? If not, doesn't that mean you're not testing the specification fully? What's the difference between that sort of declarative aspect (the method signature) and the contract? We may have to agree to differ in the end, but it's an interesting discussion... – Blakeley

@Jason: i think i thought that Contracts spell out things that don't need to be tested. For example, Pex won't pursue code paths that are cut off by a contract. – Dapple

Could you not also do this logic in a derived class from ExpectedExceptionBaseAttribute, by overriding the Verify(Exception) method? – Anosmia

@Steve: Possibly - I don't know. However, I'm not keen on decorating tests with attributes for expected exceptions: I usually want to make sure that the exception is only thrown where I expect it. – Blakeley

@Jon Skeet: It's really easy to put the contract in, but not activate the compile-time rewrite, so the contract does nothing. Unit-testing that bad arguments are rejected helps avoid that (even if all you are testing is the code contract implementation), though I would try and avoid making the test 'know' that CodeContracts are the implementation. – Buran

@piers7: In that case, I'd probably do it for one method within the assembly - it's not like you're accidentally going to turn rewriting on for that one method but not others. Adding unit tests for every single contract feels like a waste of time to me. – Blakeley

@JonSkeet @Jason For me, the crucial difference is that the compiler will prevent me calling void M(string s) with an int parameter. The unit test should test code behaviour, so if I expect an exception when I call it with null, then I should be able to verify the expected behaviour with a unit test. – Dorran

@BernhardHofmann: But the static analysis tool will prevent you from calling the method with a null argument too... if you're using the static analysis tool, of course. – Blakeley

Jon - you are right, ExpectedException and other attributes are not the way to go. I coded an Assert extension Assert.Throws<T>() similar to xUnit and this is much cleaner and more precise. As regards testing contracts, of course you should do this. If someone removes or modifies the contract in such a way that the behaviour changes, you get a failing test indicating a breaking change. This is especially important for developing libraries. – Anosmia

The trick is realizing that the argument types and the contracts are the specification. Now the spec and code can not get out of sync. (However I would want to test that specification checking is turned on, if it can be turned off). – Biblioclast

I'm a bit late to the conversation here but I believe the key difference between a method signature and a contract is that Contracts don't break the build by default. It is not possible to compile your code where you call a method with an int when it expects a string. It is possible (via configuration) to call a method in a way that violates the contract. So I would say, if your build system requires the static analyzer goes green then drop the tests. If your build system doesn't depend on the static analyzer passing then add the tests instead. – Nonjuror

@Micah, you can easily pass the wrong type to a method and not have the compiler complain. Reflection, box/cast, and the dynamic keyword for starters. – Forget

For everyone that is worried about Contract being written wrong because you have the dev do the work where you have your architect/lead write the spec test. You can implement contracts on your interfaces. The awesome thing about that is you don't have to duplicate the contract for every implementation and the static checker will ensure implementations are correct. – Forget

@MicahZoltu I agree. Plus, a more complex contract can itself have a bug and if you don't cover it with tests, it will show up in production. – Acalia

Answer 2

EDIT: I have had a conversion and no longer use either ExpectedException or this attribute below, but rather have coded some extension methods:

AssertEx.Throws<T>(Action action);
AssertEx.ThrowsExact<T>(Action action);
AssertEx.ContractFailure(Action action);

These allow me to be more precise about where the exception is raised.

Example of ContractFailure method:

    [SuppressMessage("Microsoft.Design", "CA1031:DoNotCatchGeneralExceptionTypes", Justification = "Cannot catch ContractException")]
    public static void ContractFailure(Action operation)
    {
        try
        {
            operation();
        }
        catch (Exception ex)
        {
            if (ex.GetType().FullName == "System.Diagnostics.Contracts.__ContractsRuntime+ContractException")
                return;

            throw;
        }

        Assert.Fail("Operation did not result in a code contract failure");
    }

---

I created an attribute for MSTest, that behaves similarly to the ExpectedExceptionAttribute:

public sealed class ExpectContractFailureAttribute : ExpectedExceptionBaseAttribute
{
    const string ContractExceptionName = "System.Diagnostics.Contracts.__ContractsRuntime+ContractException";

    protected override void Verify(Exception exception)
    {
        if (exception.GetType().FullName != ContractExceptionName)
        {
            base.RethrowIfAssertException(exception);
            throw new Exception(
                string.Format(
                    CultureInfo.InvariantCulture,
                    "Test method {0}.{1} threw exception {2}, but contract exception was expected. Exception message: {3}",
                    base.TestContext.FullyQualifiedTestClassName,
                    base.TestContext.TestName,
                    exception.GetType().FullName,
                    exception.Message
                )
            );
        }
    }
}

And this can be used similarly:

    [TestMethod, ExpectContractFailure]
    public void Test_Constructor2_NullArg()
    {
        IEnumerable arg = null;

        MyClass mc = new MyClass(arg);
    }

Anosmia

Answer 3

in the vs2010 rtm, the full name has been changed to "System.Diagnostics.Contracts.__ContractsRuntime+ContractException". HTH

Piscary

Answer 4

Though this question is getting old, and an answer has already been supplied, I feel like I have a nice solution that keeps things simple and readable. In the end, it allows us to write tests on preconditions as simple as:

[Test]
public void Test()
{
    Assert.That(FailingPrecondition, Violates.Precondition);
}

public void FailingPrecondition() {
    Contracts.Require(false);
}

Okay, so the idea is to provide the Code Contracts rewriter with a custom contract runtime class. This can be setup in the assembly's properties under "Custom Rewriter Methods" (see the Code Contracts User Manual section 7.7):

Remember to also check Call-site Requires Checking!

The custom class looks something like this:

public static class TestFailureMethods
{
    public static void Requires(bool condition, string userMessage, string conditionText)
    {
        if (!condition)
        {
            throw new PreconditionException(userMessage, conditionText);
        }
    }

    public static void Requires<TException>(bool condition, string userMessage, string conditionText) where TException : Exception
    {
        if (!condition)
        {
            throw new PreconditionException(userMessage, conditionText, typeof(TException));
        }
    }
}

Using a custom PreconditionException class (it contains nothing fancy!). We furthermore add a small helper class:

public static class Violates
{
    public static ExactTypeConstraint Precondition => Throws.TypeOf<PreconditionException>();
}

This allows us to write simple, readable tests on precondition violations, as shown above.

Daunt