xUnit test pass but getting "Catastrophic failure: System.ArgumentException"

Question

When running test from command line (dotnet test) in a .net 5.0 project using xunit all tests seems to pass but the process crashes with the following message from the detailed verbosity of dotnet test:

Catastrophic failure: System.ArgumentException : There is at least one object in this array that cannot be serialized (Parameter 'array')
[xUnit.net 00:00:03.74]     [FATAL ERROR] System.ArgumentException
[xUnit.net 00:00:03.74]       System.ArgumentException : There is at least one object in this array that cannot be serialized (Parameter 'array')
[xUnit.net 00:00:03.74]       Stack Trace:
[xUnit.net 00:00:03.74]         C:\Dev\xunit\xunit\src\xunit.runner.utility\Extensions\MessageSinkMessageExtensions.cs(44,0): at MessageSinkMessageExtensions.Dispatch[TMessage](IMessageSinkMessage message, HashSet`1 messageTypes, MessageHandler`1 callback)

This just happened when running dotnet test from the command line, running the test from VisualStudio works.

I'm testing a dotnet 5 rest API using TestServer.

Any ideas what could be the cause?

Packages version used:

<PackageReference Include="Microsoft.AspNetCore.TestHost" Version="5.0.8" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
<PackageReference Include="xunit" Version="2.4.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">

-- UPDATE --

I just realize that the error described here is happening when using an xUnit Theory that take the following example class as argument:

    public record TruncatedString
    {
        public TruncatedString(string value)
        {
            Value = FormatTruncatedString(value);
        }

        protected string Value { get; }

        protected static string FormatTruncatedString(string value)
        {
            return value.Substring(0,4);
        }

        public static implicit operator string(TruncatedString truncated) => truncated.Value;
        public static implicit operator TruncatedString(string text) => new (text);

        public override string ToString() => Value;
    }

And is used in a xUnit Theory like this:

[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("ABC")]
public async Task ThestWithTheErrorMessage(TruncatedString value)
{
    // ...
    // THIS TEST PRODUCE THE SERIALIZATION ERROR
    // System.ArgumentException : There is at least one object in this array that cannot be serialized (Parameter 'array')
    //   Stack Trace:
    //     C:\Dev\xunit\xunit\src\xunit.runner.utility\Extensions\MessageSinkMessageExtensions.cs(39,0): at MessageSinkMessageExtensions.Dispatch[TMessage](IMessageSinkMessage message, HashSet`1 messageTypes, MessageHandler`1 callback)
}

Freewill

Answer 1

I've run into the same issue. It seems that xUnit will not run the theory test if there's an implicit conversion happening.

Rewrite your test to be like:

[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("ABC")]
public async Task Test(string input)
{
    TruncatedString value = input;
    // ...
}

This bug in xUnit is documented here: https://github.com/xunit/xunit/issues/1742

Marchak

Answer 2

As a workaround for unit tests. I have added a support of IXunitSerializable to help XUnit to build the correct test case data.

public sealed record CountryCode(string Value) : IXunitSerializable
{
    // It's required by IXunitSerializable
    public CountryCode(): this(string.Empty) { }

    public override string ToString()
    {
        return Value;
    }

    public static implicit operator CountryCode(string self)
    {
        return new CountryCode(self);
    }

    public static implicit operator string(CountryCode self)
    {
        return self.Value;
    }

    void IXunitSerializable.Serialize(IXunitSerializationInfo info)
    {
        info.AddValue(nameof(Value), Value, typeof(string));
    }

    void IXunitSerializable.Deserialize(IXunitSerializationInfo info)
    {
        throw new NotSupportedException("This should not be used.");
    }
}

Gaffe