xUnit - Display test names for theory memberdata (TestCase)

Question

I've been using NUnit for testing and I'm really fond of test cases. In NUnit you can easily set each test name in the test case using the SetName function in a TestCaseData class.

Does xUnit have a similar function for this?

Currently I can only see one test in the test explorer even tho I have 6 tests in the test case.

xUnit test

public class LogHandler : TestBase
{
    private ILogger _logger;

    public LogHandler()
    {
        //Arrange
        LogAppSettings logAppSettings = GetAppSettings<LogAppSettings>("Log");

        IOptions<LogAppSettings> options = Options.Create(logAppSettings);

        LogService logService = new LogService(new Mock<IIdentityService>().Object, options);

        LogProvider logProvider = new LogProvider(logService);

        _logger = logProvider.CreateLogger(null);
    }

    public static IEnumerable<object[]> TestCases => new[]
    {
        new object[] { LogLevel.Critical,
            new EventId(),
            new Exception(),
            1 },
        new object[] { LogLevel.Error,
            new EventId(),
            new Exception(),
            1 },
        new object[] { LogLevel.Warning,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.Information,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.Debug,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.Trace,
            new EventId(),
            new Exception(),
            0 },
        new object[] { LogLevel.None,
            new EventId(),
            new Exception(),
            0 }
    };

    [Theory, MemberData(nameof(TestCases))]
    public void Test(LogLevel logLevel, EventId eventId, Exception exception, int count)
    {
        //Act
        _logger.Log<object>(logLevel, eventId, null, exception, null);

        //Assert
        int exceptionCount = Database.Exception.Count();

        Assert.Equal(exceptionCount, count);
    }
}

xUnit test window

Should be 6 tests here instead of one! (ignore GetOrganisationStatuses).

NUnit test case

public static IEnumerable TestDatabaseCases
{
    get
    {
        yield return new TestCaseData(LogLevel.Critical,
            new EventId(1),
            new Exception("Exception"),
            0,
            1).SetName("InsertException_Should_Insert_When_LogLevel_Critical");

        yield return new TestCaseData(LogLevel.Error,
            new EventId(1),
            new Exception("Exception"),
            0,
            1).SetName("InsertException_Should_Insert_When_LogLevel_Error");

        yield return new TestCaseData(LogLevel.Warning,
            new EventId(1),
            new Exception("Exception"),
            0,
            0).SetName("InsertException_Should_Not_Insert_When_LogLevel_Warning");

        yield return new TestCaseData(LogLevel.Information,
            new EventId(1),
            new Exception("Exception"),
            0,
            0).SetName("InsertException_Should_Not_Insert_When_LogLevel_Information");

        yield return new TestCaseData(LogLevel.Debug,
            new EventId(1),
            new Exception("Exception"),
            0,
            0).SetName("InsertException_Should_Not_Insert_When_LogLevel_Debug");
    }
}

NUnit test window

This is what I want in xUnit!

How do I in xUnit set a name for each test in the test case?

Ponceau

I had the same problem before. You class will need to inherit from IXunitSerializable. It's quite cumbersome to implement.. Not sure if there is a better method. – Contiguity

There is a ready to use implementation which bypasses a need to change code under test and test code. Simply include new custom xUnit TheoryAttribute, XunitDiscoverer class and two test classes - see second answer for details and MIT licensed working code. – Eudosia

Answer 1

Currently I can only see one test in the test explorer even tho I have 6 tests in the test case.

This is because your test data is not considered serializable by xUnit.net. See this issue: https://github.com/xunit/xunit/issues/1473

The salient details are:

The short answer

If some of your theory data can't be "serialized" by xUnit.net, then it cannot be encapsulated into the serialization of a test case which we're required to do for the Visual Studio test runner.

The long answer

In the Visual Studio test runner, test cases are discovered in one process, and executed in another. Therefore, test cases must be able to be turned into an unqualified string representation (aka, "serialized") in order to be run. We can also serialize at the test method level, because that just involves knowing the type and method name (both strings). When you start putting data into the mix, we need to ensure we know how to serialize that data; if we can't serialize all of the data for a theory, then we have to fall back to just a single method (which we know we can serialize).

The full answer contains more detail.

In NUnit you can easily set each test name in the test case using the SetName function in a TestCaseData class.

Does xUnit have a similar function for this?

Not at this time.

Cappello

That's a shame.. As it stands now I'll continue with NUnit. Hopefully they will fix this. – Ponceau

@Ponceau Common it's not a shame but small limitation which can be easily compensated for by small piece of code - see below for my answer. Anyway xUnit is a test framework of choice for large number of most important in DotNet world repos. – Eudosia

@JacekBlaszczynski Can you comment on what those reasons might be? – Trickery

Answer 2

This worked for me:

public class TestScenario
{
...
    public override string ToString()
    {
        return $"Role: {Role}...";
    }
}

    [Theory]
    [ClassData(typeof(MyProvider))]
    public void TestScenarios(TestScenario scenaro)
    {
        TestScenarioInternal(scenaro);

    }

Kigali

This was exactly what I was looking for, thanks! – Glasgo

What is a relation between MyProvider and TestScenario classes? – Commodus

@Dima, I believe MyProvider returns list of TestScenario instances. – Gerhardine

Answer 3

One readable and short way to do it would be:

public record AddNumbersTestCase(
    int Number1,
    int Number2,
    int ExpectedResult)
{
    private static readonly AddNumbersTestCase[] TestCases =
    {
        new(0, 0, 0),
        new(0, 1, 1),
        new(1, 0, 1),
        new(1, 1, 2),
        new(2, 3, 5),
    };

    public static IEnumerable<object[]> TestCasesData =>
        TestCases.Select(testCase => new object[] { testCase });

    public override string ToString() =>
        $"{Number1} + {Number2} should equal {ExpectedResult}";
}

[Theory]
[MemberData(nameof(AddNumbersTestCase.TestCasesData), MemberType = typeof(AddNumbersTestCase))]
public void FindDayOfWeek(AddNumbersTestCase testCase)
{
    int result = testCase.Number1 + testCase.Number2;

    Assert.Equal(testCase.ExpectedResult, result);
}

Lynn

This is a nice workaround without too much plumbing! – Bridgettebridgewater

This looks very nice, but it seems like xUnit doesn't know how to serialize the custom record type, and thus it displays only one test case, and outputs multiple results in this one test case. The output uses overridden ToString(), so it looks pretty there, but still doesn't resolve the initial question. Or am I missing something? – Presentable

probably related to issue stated by Dasha: github.com/xunit/xunit/issues/1473 – Accuse

Answer 4

Actually there is a working solution requiring some plumbing code which should work with unchanged tests. It requires implementation of custom TheoryAttribute, custom TheoryDiscoverer, and custom TestCase classes. Whole solution is available under MIT license in this repo DjvuNet/DjvuNet.Shared.Tests.

Required files with implementations are: DjvuTheoryAttribute, DjvuTheoryDiscoverer, DjvuNamedDataRowTestCase, DjvuDataRowTestCase

Usage is straightforward: compile above files either including them directly into test assembly or as a separate assembly and use them in code as follows:

    [DjvuTheory]
    [ClassData(typeof(DjvuJsonDataSource))]
    public void DirmChunk_Theory(DjvuJsonDocument doc, int index)
    {
        int pageCount = 0;
        using (DjvuDocument document = DjvuNet.Tests.Util.GetTestDocument(index, out pageCount))
        {
            DjvuNet.Tests.Util.VerifyDjvuDocument(pageCount, document);
            DjvuNet.Tests.Util.VerifyDjvuDocumentCtor(pageCount, document);

            // DirmChunk is present only in multi page documents
            // in which root form is of DjvmChunk type
            if (document.RootForm.ChunkType == ChunkType.Djvm)
            {
                DirmChunk dirm = ((DjvmChunk)document.RootForm).Dirm;

                Assert.NotNull(dirm);

                Assert.True(dirm.IsBundled ? doc.Data.Dirm.DocumentType == "bundled" : doc.Data.Dirm.DocumentType == "indirect");

                var components = dirm.Components;
                Assert.Equal<int>(components.Count, doc.Data.Dirm.FileCount);
            }
        }
    }

One of the theory function arguments is not serializable in xUnit but despite that theory tests will be displayed individually and numbered. If first argument to theory function is of the string type it will be used as a name of the test besides being an argument to function invocation.

Credit for the idea goes to other developer - I have to find a link to his code - but it was reimplemented from scratch for DjvuNet project.

Eudosia

Answer 5

A very silly solution that works nonetheless with very little extra code. Define only one test input - a string that you want to be displayed in test explorer. And create some way to fetch actual test data by that string - can be a dictionary, a method, whatever works best in your case.

For example:

private (string someTestData, int moreTestData, string expectedResult) GetTestSomethingTestCaseData(string testCase)
{
    return testCase switch
    {
        "when it's A" => ("A data", 10, "A data 10"),
        "when it's B" => ("B data", 20, "B data 20"),
        _ => throw new ArgumentException(nameof(testCase))
    };
}


[Theory(DisplayName = "Test something")]
[InlineData("when it's A")]
[InlineData("when it's B")]
public void TestSomething(string testCase)
{
    var (someTestData, moreTestData, expectedResult) = GetTestSomethingTestCaseData(testCase);

    var actualResult = $"{someTestData} {moreTestData}";

    Assert.Equal(expectedResult, actualResult);
}

Presentable

I don't think it is silly. I guess it can be summarized as this: if you have non-serializable test data, replace it with serializable key and inside the test replace back the key with the non-serializable data. Simple and effective! I'll post my solution below but the idea is very much the same. – Agler

Answer 6

Here is my solution similar in spirit to @Dasha's proposal:

    public static readonly TheoryData<string, object?, string> TestData = new ()
    {
        TestDataLine(), "no hocus-pocus", "no hocus-pocus expected result", }
        TestDataLine(), "ArraySegment<byte>", "ByteArraySegment result" },
    };

    static readonly Dictionary<string, object> _substitute = new()
    {
        ["ArraySegment<byte>"] = new ArraySegment<byte>([1, 2, 3, 1, 2, 3, 1, 2, 3, 10], 1, 8),
    };

    static object? Substitute(ref object? value)
        => value is string key && _dataReplacement.TryGetValue(key, out var v1)
            ? value = v1 : value;

    [Theory]
    [MemberData(nameof(TestData))]
    public async Task TestConstantAsync(string _, object? value, string expected)
    {
        Substitute(ref value);
        ... // here goes your test with non-serializable data

TestLine is another small function that I use. It returns a string with the file name and the line of the location where it is being called from. Something like:

    public static string TestLine(
        [CallerFilePath] string pathTestFile = "",
        [CallerLineNumber] int lineNumber = 0)
        => $"{pathTestFile} : {lineNumber}";

Agler