xunit test Fact multiple times

Question

I have some methods that rely on some random calculations to make a suggestion and I need to run the Fact several times to make sure is ok.

I could include a for loop inside the fact i want to test but because there are several test where I want to do this I was lookup for a cleaner approach, something like this Repeat attribute in junit: http://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without-loops/

Can I easily implement something like this in xunit?

Dry

Answer 1

You'll have to create a new DataAttribute to tell xunit to run the same test multiple times.

Here's is a sample following the same idea of junit:

public class RepeatAttribute : DataAttribute
{
    private readonly int _count;

    public RepeatAttribute(int count)
    {
        if (count < 1)
        {
            throw new ArgumentOutOfRangeException(nameof(count), 
                  "Repeat count must be greater than 0.");
        }
        _count = count;
    }

    public override IEnumerable<object[]> GetData(MethodInfo testMethod)
    {
        return Enumerable.Repeat(new object[0], _count);
    }
}

With this code in place, you just need to change your Fact to Theory and use the Repeat like this:

[Theory]
[Repeat(10)]
public void MyTest()
{
    ...
}

Irresponsive

Works nicely. I did have to remove the Type[] parameterTypes parameter from GetData, though. Now if this would just work with other attributes like CombinatorialData. – Seymourseys

I updated the sample code to reflect the newer xunit APIs. Thanks! – Nudibranch

I have updated the code so that it works with the latest xunit. It will work so long as repetitionNumber is a parameter of the Theory test – Stepchild

Doesn't seem to work @user357783 can you give an example? – Timothytimour

@Timothytimour I guess my edits have been lost in peer review. I've just tried resubmitting them. In the meantime, see gistlyn.com/?gist=52c37e37b51a0ec92810477be34695ae – Stepchild

Is it possible to run those repetitions in parallel? – Beaubeauchamp

With the code above I get this error: Skipping test case with duplicate ID – Beaubeauchamp

With [Theory] attribute I got the following Error :: Theory methods should have parameters If I change Theory to [Fact] :: Fact methods should not have test data – Furfuran

Answer 2

Had the same requirement but the accepted answers code was not repeating the tests so I have adapted it to:

public sealed class RepeatAttribute : Xunit.Sdk.DataAttribute
{
    private readonly int count;

    public RepeatAttribute(int count)
    {
        if (count < 1)
        {
            throw new System.ArgumentOutOfRangeException(
                paramName: nameof(count),
                message: "Repeat count must be greater than 0."
                );
        }
        this.count = count;
    }

    public override System.Collections.Generic.IEnumerable<object[]> GetData(System.Reflection.MethodInfo testMethod)
    {
        foreach (var iterationNumber in Enumerable.Range(start: 1, count: this.count))
        {
            yield return new object[] { iterationNumber };
        }
    }
}

While on the previous example the Enumerable.Repeat was being used it would only run the test 1 time, somehow xUnit is not repeating the test. Probably something they have changed a while ago. By changing to a foreach loop we are able to repeat each test but we also provide the "iteration number". When using it on the test function you have to add a parameter to your test function and decorate it as a Theory as shown here:

[Theory(DisplayName = "It should work")]
[Repeat(10)]
public void It_should_work(int iterationNumber)
{
...
}

This works for xUnit 2.4.0.

I've created a NuGet package to use this in case anyone is interested.

Frown

Thanks for noticing that, @ShanteshwarInde – Frown

A feature suggestion I have a [Theory] testing a Cache-Aside pattern. The first run will insert data into a distributed cache, second pulls the previous data from cache however your nuget doesn't appear to handle the [InlineData] attribute correctly, i.e. [InlineData("myCacheKey")] (and also the UI in VS2k19 16.3 is a bit messed up)... :) – Formulate

Thanks for the feedback @Formulate but that seems like a feature request for xUnit... ;-) – Frown

It's shorter to have the GetData to just use: return Enumerable.Range(1, this.count).Select(n => new object[] { n }) – Angular

Answer 3

Simplest approach for small number of iterations: Make it a Theory instead of a Fact. Insert one line of [InlineData] for each iteration.

using Xunit;

namespace MyUnitTests
{
    public class Class1
    {

        [Theory]
        [InlineData]
        [InlineData]
        public void TestThis()
        {
            // test code here
        }
    }
}

Tested with XUnit 2.4.1

Gelasias

To ignore compiler errors add this to the method: [SuppressMessage("Usage", "xUnit1025:InlineData should be unique within the Theory it belongs to")] [SuppressMessage("Usage", "xUnit1006:Theory methods should have parameters")] – Schoenberg

Answer 4

I know this is an old thread, but here is a little trick you can apply if you need to repeat a test a small number of times:

First, create some dummy data:

public static IEnumerable<object[]> DummyTestData()
{
   yield return new object[] { 0 };
   yield return new object[] { 1 };
   yield return new object[] { 2 };
   yield return new object[] { 3 };
 }

Then use the dummy data in order to force your test to run for each vector. In that case, the same test will be called 4 times (but the dummy data is actually not being used):

private static int _counter = 0;

[Theory]
[MemberData(nameof(DummyTestData))]
public void SomeTest(int dummyParam)     // dummyParam is unused
{
    _counter+= 1;
    DoSomething();

    Assert.True(...);           
}    

I find this approach very useful and less cumbesome than creating a new attribute.

Of course, this is not a good solution if you need the number of repetitions to be parameterisable (although I am sure someone could suggest a way to make my solution parameterisable :-)).

Striate

Answer 5

Since MemberData also takes parameters for the member, you may simply pass the iteration number to create the "DummyTestData".

public static IEnumerable<object[]> MockParallel(int n)
{
    foreach (var iterationNumber in Enumerable.Range(start: 1, count: n))
    {
        yield return new object[] { iterationNumber };
    }
}

Now you can pass your repeat times as a parameter.

[Theory]
[MemberData(nameof(MockParallel), n)] // repeat n times
public void SomeTest(int repeat)

Mitzi

Answer 6

You can use ITestCaseOrderer to order tests. While ordering you can specify some test multiple times. E.g.

using System.Collections.Generic;
using System.Linq;
using Xunit.Abstractions;
using Xunit.Sdk;

namespace XUnit.Project.Orderers {
    public class RepeatOrderer : ITestCaseOrderer {
        public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase {
            List<TTestCase> result = new();
            testCases.ToList().ForEach(testCase => {
                var repeat = (ReflectionAttributeInfo)testCase.TestMethod.Method.GetCustomAttributes(typeof(RepeatAttribute)).FirstOrDefault();
                if (repeat != null && repeat.Attribute is RepeatAttribute) {
                    var attr = repeat.Attribute as RepeatAttribute;
                    Enumerable.Range(0, attr.Count).ToList().ForEach((_) => { result.Add(testCase); });
                }
                else {
                    result.Add(testCase);
                }
            });

            return result;
        }
    }
}

Attribute

public class RepeatAttribute : Attribute {
    public int Count { get; init; }
    public RepeatAttribute(int count) {
        this.Count = count;
    }
}

Instruct test project to use orderer. In my example TestProject1 is assembly where ordere is located.

[assembly : TestCaseOrderer("XUnit.Project.Orderers.RepeatOrderer", "TestProject1")]

And use Repeate attribute

[Fact]
[Repeat(6)]
public void Test1() {

...
}

Garrettgarrick