C# How to simplify unit testing string parameters using AutoFixture

Question

I'm trying to create a simple way to test string parameters in unit tests, for most string parameters, I want to check the behaviour when the parameter is Null, Empty or consists only of whitespaces.

In most of the cases, I'm checking the parameters with string.IsNullOrWhiteSpace(), and throw an exception if it has one of those three values.

Now for unit testing, it seems I have to write, for every string parameter, three unit tests. One for null values, one for empty values and one for Whitespaces.

Imagine a method with 3 or 4 string parameters, then I need to write 9 or 12 unit tests...

Can anyone come up with a simple way to test this? Maybe using AutoFixture?

Harmaning

Search for AutoFixture Idioms - it has good support for the specific guard clauses you describe in the OP – C

While that's certainly a possibility, I wouldn't recommend that approach for a few reasons. First, the OP is asking to verify more than just null guards – which is supported out-of-the-box by AutoFixture.Idioms – so that would require writing new behavior expectations. Granted, that's not too hard. Second, AFAIK AutoFixture.Idioms requires invoking methods by their name through Reflection. Third, I think that Idioms, while useful, tend to separate the assertion logic too far from the tests for the sake of DRY at the expense of readability. – Thurston

Nonetheless, it still is a viable solution. I've added it to my answer for the sake of completeness. Thank you @RubenBartelink for pointing it out. – Thurston

@EnricoCampidoglio Re #2, Idioms provides overloads at varying levels from Assembly to method name strings - while it's true that the lowest levels function at the MethodInfo level, the whole point is that it enables you to make broad assertions against various sets of items - you simply wouldnt do a GuardClauseAssertion style test for s single method. – C

Re #3, I'd counter that it's actually extremely valuable to be able to pull the noise of guard clause tests out of a suite of tests in order to let them focus on emphasising the intended behaviour of a sut over the busywork and boilerplate that leaning on primitives can necessitate in some languages/API styles. For example, one might define a single Test Method that checks the GuardClauseAssertion for all methods/properties of a type - and as you add/remove/refactor you don't need to adjust the busywork tests to match. – C

In some cases I've had success with implementing blanket guard clauses across all Types in a given Assembly (e.g. a Command Handlers assembly) with a single Test Method. Obviously, this sort of usage strays into the sweet spot of mechanisms such as Code Analysis. Also (especially when combinatorial explosion comes into play if you have lots of arguments and lots of Idioms-Assertions), the execution time cost may become too prohibitive for them to sit within your <10 seconds unit test suite -- in which case one simply migrates them to a test suite that's less in your face – C

@RubenBartelink Requiring that all methods in a type have null guards seems like too broad of an assumption, let alone in an entire assembly. What if that requirement doesn't apply to some methods? Also, do these types really need to be in a separate assembly or is it dictated by having the ability to write these tests? – Thurston

It seems these things are best tested on a case by case basis. As an aside, being too defensive with null-checking might be treating a symptom of a larger design problem. – Thurston

@EnricoCampidoglio Yes, accept all those points (and I should say I agree with 95% of everything else you've said too; I'm just trying to raise awareness of the library's facilities in this space). My main point is that you've decided you absolutely must have some code verifying some set of guards and that's starting to swamp the general tests that talk about he useful stuff the type accomplishes, there is a hammer/carpet available, with a giant listen to your tests proviso :) – C

Answer 1

To avoid duplicating the same test multiple times you can write a parameterized test.

If you're using xUnit, you would write a so called theory. A theory means you're proving a principle, that is that a certain function behaves as expected when given different samples of the same kind of input data. For example:

[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void Should_throw_argument_exception_when_input_string_is_invalid(string input)
{
    Assert.Throws<ArgumentException>(() => SystemUnderTest.SomeFunction(input));
}

The xUnit runner will run this test multiple times, each time assigning the input parameter to one of the values specified in the [InlineData] attribute.

If the function being tested has more than one parameter, you might not care what values are passed to the remaining ones, as long as at least one of them is a string that's either null, empty or contains only whitespace.

In this case, you can combine parameterized tests with AutoFixture. AutoFixture is designed to give you general-purpose test data that is good enough to use in most scenarios when you don't necessarily care about what the exact values are.

In order to use it with xUnit theories, you'll need to add the AutoFixture.Xunit NuGet package (or AutoFixture.Xunit2 depending on which version you're using) to your test project and use the InlineAutoData attribute:

[Theory]
[InlineAutoData(null)]
[InlineAutoData("")]
[InlineAutoData(" ")]
public void Should_throw_argument_exception_when_the_first_input_string_is_invalid(
    string input1,
    string input2,
    string input3)
{
    Assert.Throws<ArgumentException>(() =>
        SystemUnderTest.SomeFunction(input1, input2, input3));
}

Only the input1 parameter will have a specific value, while the rest will be assigned random strings by AutoFixture.

One thing to note here is that the values passed through the [InlineAutoData] attribute are assigned to the test parameters based on their position. Since we need to test the same behavior for all three parameters individually, we'll have to write three tests:

[Theory]
[InlineAutoData(null)]
[InlineAutoData("")]
[InlineAutoData(" ")]
public void Should_throw_argument_exception_when_the_second_input_string_is_invalid(
    string input2,
    string input1,
    string input3)
{
    Assert.Throws<ArgumentException>(() =>
        SystemUnderTest.SomeFunction(input1, input2, input3));
}

[Theory]
[InlineAutoData(null)]
[InlineAutoData("")]
[InlineAutoData(" ")]
public void Should_throw_argument_exception_when_the_third_input_string_is_invalid(
    string input3,
    string input1,
    string input2)
{
    Assert.Throws<ArgumentException>(() =>
        SystemUnderTest.SomeFunction(input1, input2, input3));
}

A word on property-based testing

I can't help but think that these kinds of test scenarios are a perfect match for property-based testing. Without going into too much detail, property-based testing is about proving a specific behavior (or "property") of a function by running it multiple times with generated input.

In other words:

Property-based tests make statements about the output of your code based on the input, and these statements are verified for many different possible inputs.

In your case, you could write a single test to verify that the function throws an ArgumentException whenever at least one of the arguments is either null, an empty string or a string that only contains whitespace.

In .NET, you can use a library called FsCheck to write and execute property-based tests. While the API is primarily designed to be used from F#, it can also be used from C#.

However, in this particular scenario, I think you're better off sticking with regular parameterized tests and AutoFixture to achieve the same goal. In fact, writing these tests with FsCheck and C# would end up being more verbose and wouldn't really buy you much in terms of robustness.

Update: AutoFixture.Idioms

As @RubenBartelink pointed out in the comments, there is another option. AutoFixture encapsulates common assertions in a small library called AutoFixture.Idioms. You can use it to centralize the expectations on how the string parameters are validated by a method and use them in your tests.

While I do have my reservations on this approach, I'll add it here as another possible solution for the sake of completeness:

[Theory, AutoData]
public void Should_throw_argument_exception_when_the_input_strings_are_invalid(
    ValidatesTheStringArguments assertion)
{
    var sut = typeof(SystemUnderTest).GetMethod("SomeMethod");

    assertion.Verify(sut);
}

public class ValidatesTheStringArguments : GuardClauseAssertion
{
    public ValidatesTheStringArguments(ISpecimenBuilder builder)
        : base(
              builder,
              new CompositeBehaviorExpectation(
                  new NullReferenceBehaviorExpectation(),
                  new EmptyStringBehaviorExpectation(),
                  new WhitespaceOnlyStringBehaviorExpectation()))
    {
    }
}

public class EmptyStringBehaviorExpectation : IBehaviorExpectation
{
    public void Verify(IGuardClauseCommand command)
    {
        if (!command.RequestedType.IsClass
            && !command.RequestedType.IsInterface)
        {
            return;
        }

        try
        {
            command.Execute(string.Empty);
        }
        catch (ArgumentException)
        {
            return;
        }
        catch (Exception e)
        {
            throw command.CreateException("empty", e);
        }

        throw command.CreateException("empty");
    }
}

public class WhitespaceOnlyStringBehaviorExpectation : IBehaviorExpectation
{
    public void Verify(IGuardClauseCommand command)
    {
        if (!command.RequestedType.IsClass
            && !command.RequestedType.IsInterface)
        {
            return;
        }

        try
        {
            command.Execute(" ");
        }
        catch (ArgumentException)
        {
            return;
        }
        catch (Exception e)
        {
            throw command.CreateException("whitespace", e);
        }

        throw command.CreateException("whitespace");
    }
}

Based on the expectations expressed in NullReferenceBehaviorExpectation, EmptyStringBehaviorExpectation, and WhitespaceOnlyStringBehaviorExpectation AutoFixture will automatically attempt to invoke the method named "SomeMethod" with null, empty strings and whitespace respectively.

If the method doesn't throw the correct exception – as specified in the catch block inside the expectation classes – then AutoFixture will itself throw an exception explaining what happend. Here's an example:

An attempt was made to assign the value whitespace to the parameter "p1" of the method "SomeMethod", and no Guard Clause prevented this. Are you missing a Guard Clause?

You can also use AutoFixture.Idioms without parameterized tests by simply instantiating the objects yourself:

[Fact]
public void Should_throw_argument_exception_when_the_input_strings_are_invalid()
{
    var assertion = new ValidatesTheStringArguments(new Fixture());
    var sut = typeof(SystemUnderTest).GetMethod("SomeMethod");

    assertion.Verify(sut);
}

Thurston

Thanks for the answer, the only 'problem' is that we use MSTest, and it looks like there is no support for parameterized testing. – Harmaning

In the AF.Idioms example, you [indirectly] use AutoData to generate suts. From memory, Idioms also provides a layer that does full end-to-end verification of an assertion (you can supply a Fixture/Specimen Builder or it will even build it) - addressing @Harmaning addressed (i.e. you shouldnt need to use 'AutoData). This makes sense as for guard clause tests, you shouldn't be leaning on any other conventions that would typically be embodied in an AutoData`[-derived Attribute] anyway. – C

@Harmaning As Ruben Bartelink correctly pointed out, you can also use AutoFixture.Idioms without parameterized tests. I updated my answer with an example. – Thurston

Answer 2

Based on the accepted answer, now those behavior expectations are already included in the package, so final code can be shorten like this with the same usage:

public class StringGuardClauseAssertion(ISpecimenBuilder builder) : GuardClauseAssertion(
    builder, 
    new CompositeBehaviorExpectation(
        new NullReferenceBehaviorExpectation(),
        new EmptyStringBehaviorExpectation(),
        new WhiteSpaceStringBehaviorExpectation()))
{
}

Gamo