AutoFixture with NUnit AutoMoq prevents tests from running

Question

UPDATE: AutoFixture team released a fix for this in version 3.51.

Simply extend the AutoDataAttribute doing so:

public class AutoDataFixedNameAttribute : AutoDataAttribute
{
    public AutoDataFixedNameAttribute()
    {
        this.TestMethodBuilder = new FixedNameTestMethodBuilder();
    }
}

Then use this new attribute instead of the built-in AutoData in your NUnit tests.

Starting from v4, this behavior is the default one.

---

Previous post

I'm trying to use AutoFixture with NUnit and Moq, using the following AutoMoqDataAttribute :

public class AutoMoqDataAttribute : AutoDataAttribute
{
    public AutoMoqDataAttribute()
        : base(new Fixture().Customize(new AutoMoqCustomization()))
    {
    }
}

But when I run this test :

[Test, AutoMoqData]
public void Test(Mock<IUser> user)
{
    // do stuff with user
}

The test never runs. AutomMoqData is hit correctly, but the code inside the test is never executed and everything ends without any warning with the following message :

Test adapter sent back a result for an unknown test case. Ignoring result for 'Test(Mock<Sandbox.IUser>)'

The test also doesn't appear in the test runner list.

But if I remove the parameter :

[Test, AutoMoqData]
public void Test()
{
    // do stuff without user
}

Everything runs fine, but this is less useful without the parameters passed :)

Am I missing something here ?

Here is the list of the Nuget packages versions :

<package id="AutoFixture" version="3.50.2" targetFramework="net452" />
<package id="AutoFixture.AutoMoq" version="3.50.2" targetFramework="net452" />
<package id="AutoFixture.NUnit3" version="3.50.2" targetFramework="net452" />
<package id="Moq" version="4.5.3" targetFramework="net452" />
<package id="NUnit" version="3.7.1" targetFramework="net452" />

EDIT: Following @MarkSeemann's advice, I filed an issue on Github.

Cilium

Have you read this blog.ploeh.dk/2010/10/08/AutoDataTheorieswithAutoFixture – Deflect

Answer 1

Visual Studio Test Runner repro

This looks like an issue with the NUnit Visual Studio test adapter. I can reproduce the issue when I also add the NUnit3TestAdapter package to my repro solution.

I'm also assuming that the test class has the [TestFixture] attribute, so that the entire repro class looks like this:

[TestFixture]
public class Tests
{
    [Test, AutoMoqData]
    public void Test(Mock<IUser> user)
    {
        Assert.NotNull(user);
    }
}

When I attempt to run all tests using Visual Studio 2015's test runner, the test never runs, and this is the output to test output window:

------ Run test started ------
NUnit Adapter 3.7.0.0: Test execution started
Running all tests in C:\Users\mark\Documents\Stack Overflow\44564377\44564377\bin\Debug\Ploeh.StackOverflow.Q44564377.dll
NUnit3TestExecutor converted 1 of 1 NUnit test cases
NUnit Adapter 3.7.0.0: Test execution complete
Test adapter sent back a result for an unknown test case. Ignoring result for 'Test(Mock<Ploeh.StackOverflow.Q44564377.IUser:8e33>)'.
========== Run test finished: 0 run (0:00:01,1763498) ==========

TestDriven.Net

If, on the other hand, I try to run it with TestDriven.Net, it runs just fine:

------ Test started: Assembly: Ploeh.StackOverflow.Q44564377.dll ------

1 passed, 0 failed, 0 skipped, took 0,79 seconds (NUnit 3.7.1).

TestDriven.Net is sometimes extraordinarily tolerant of small errors in the test code, so this may not be that telling in itself.

NUnit 3 console runner

Since TestDriven.Net may be too liberal in what it accepts, a better test would be to try with the official NUnit 3 console runner:

$ packages/NUnit.ConsoleRunner.3.6.1/tools/nunit3-console.exe 44564377/bin/Debug/Ploeh.StackOverflow.Q44564377.dll
NUnit Console Runner 3.6.1
Copyright (C) 2017 Charlie Poole

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.15063.0
  CLR Version: 4.0.30319.42000

Test Files
    44564377/bin/Debug/Ploeh.StackOverflow.Q44564377.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: C:\Users\mark\Documents\Stack Overflow\44564377
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 4

Test Run Summary
  Overall result: Passed
  Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2017-06-15 11:09:21Z
    End time: 2017-06-15 11:09:22Z
    Duration: 0.933 seconds

Results (nunit3) saved as TestResult.xml

This, too, successfully executes the test.

Interim conclusion

Since both the official console runner and TestDriven.Net successfully executes the test, I'd tentatively conclude that this looks like a defect in the NUnit3TestAdapter package. May I suggest filing an issue for it?

Eventuate

Thanks a lot for your answer, I'll test that right away, and file an issue. Just out of curiosity, is it possible to use AutoMoqDataAttributes of some sort with MSTests ? I wasn't able to find information on that. Thanks! – Cilium

@X.L.Ant AFAIK, MSTest has no extensibility points. Unless this has changed recently, then the answer to your question is: no. – Eventuate

I was suspecting smth like that. Thanks a lot. – Cilium

Is AutoMoq generating random values? That would explain why it doesn't work with the adapter. Unlike "normal" execution, running under the adapter requires two separate invocations of NUnit, once at discovery and once for execution. This annoys the devs of both xUnit and NUnit, but we "solved" the problem in different ways. NUnit's only works if the data extension uses NUnit's repeatable random number generation. – Shithead

@Shithead Yes, AutoFixture generates random values. – Eventuate

That's it then. If their attribute got those values from NUnit, then it would work. We have an issue on this now - hoping for someone from AutoMoq to comment on it. – Shithead

@Shithead I'm the original author of the AutoFixture and AutoMoq. This particular feature works with xUnit.net 2, even with the Visual Studio test runner. I'm no longer the maintainer of AutoFixture, so I can't say how the current maintainers will evaluate the situation, but given that xUnit.net demonstrates that it is possible to make this work with the Visual Studio test runner, I'd consider this a defect in the NUnit VS test adapter. – Eventuate

The problem is that xUnit and NUnit took two entirely different paths in dealing with the VS 2-pass issue. In retrospect, we see that their approach works best for AutoFixture, but of course neither of us were thinking about AutoFixture when we made our design decisions - we just wanted to be able to run our own tests under VS! The software that makes use of another piece of software usually has to adjust. That's what we had to do, even though both groups (n and xUnit) agree that the 2 pass thing is a defect in Visual Studio! – Shithead

BTW, I wrote the NUnit VS adapter but I'm not the maintainer. He may have a different view than I do. It sounds like a big redesign would be called for however. We should have our respective replacements talk together. ;-) – Shithead

@Shithead Indeed, that's the sort of thing that happens with software development - particularly with integration software. I didn't mean to imply that anyone had been unprofessional, or bad programmers, because of this problem. FWIW, however, this issue is hardly constrained to only AutoFixture. At least FsCheck and Hedgehog also generate random test values. There may be others... – Eventuate

@Mark Yes, I agree. It's quite likely that many of the developers of the software that uses NUnit don't use the VS adapter. Heck, I barely use it myself! And the "ease of fixing" rule has to consider that there is more than one product using NUnit this way. – Shithead