NUnit vs. MbUnit vs. MSTest vs. xUnit.net [closed]

Question

There are quite a lot of unittesting frameworks out there for .NET. I found this little feature comparison: http://xunit.github.io/docs/comparisons.html

Now I am to choose the best one for us. But how? Does it matter? Which one is most future proof and has a decent momentum behind it? Should I care about the features? While xUnit seems to be most modern and specifically designed for .NET, NUnit again seems to be the one that is widely accepted. MSTest again is already integrated into Visual Studio ...

Inverse

That comparison table is years out of date. For example, NUnit also has Assert.Throws etc, and everything in the Assertions table is the old API. The new Assert.That(..., Is....) fluent syntax is much nicer, and has been around for a good while now. – Swob

Do you know of any table that is more up to date? – Inverse

Late 2013, moved from xUnit.net => NUnit. Also note that xUnit.NET (the project) != xUnit (the category, of which NUnit is a member) – Rollins

@Sid why you moved from xUnity.net => NUnit? – Cerargyrite

Similar question asked in 2014 Visual Studio 2013 MSTest vs NUnit – Piet

Answer 1

I know this is an old thread, but I thought I'd post a vote for xUnit.NET. While most of the other testing frameworks mentioned are all pretty much the same, xUnit.NET has taken a pretty unique, modern, and flexible approach to unit testing. It changes terminology, so you no longer define TestFixtures and Tests...you specify Facts and Theories about your code, which integrates better with the concept of what a test is from a TDD/BDD perspective.

xUnit.NET is also EXTREMELY extensible. Its FactAttribute and TraitAttribute attribute classes are not sealed, and provide overridable base methods that give you a lot of control over how the methods those attributes decorate should be executed. While xUnit.NET in its default form allows you to write test classes that are similar to NUnit test fixtures with their test methods, you are not confined to this form of unit testing at all. You are free to extend the framework to support BDD-style Concern/Context/Observation specifications, as depicted here.

xUnit.NET also supports fit-style testing directly out of the box with its Theory attribute and corresponding data attributes. Fit input data may be loaded from excel, database, or even a custom data source such as a Word document (by extending the base data attribute.) This allows you to capitalize on a single testing platform for both unit tests and integration tests, which can be huge in reducing product dependencies and required training.

Other approaches to testing may also be implemented with xUnit.NET...the possibilities are pretty limitless. Combined with another very forward looking mocking framework, Moq, the two create a very flexible, extensible, and powerful platform for implementing automated testing.

Incontrovertible

While this was true a year+ ago, NUnit has since added most of the attributes in question. In NUnit you can write tests either way. – Adkisson

Its not so much about which attributes are available, but how they can be used. xUnit.NET was designed from the ground up to be a highly flexible and extensible framework that did not lock you into any particular method of testing, and does not require you to regularly update the core framework to get the latest capabilities. – Incontrovertible

1. Sightly different names on attributes won't make much of a point. 2. NUnit was extensible and it continues to be extensible :? 3. data row parameters for tests are supported in nunit. Time ago they were supported in an extension :) 4. Nunit combined with Moq creates the same thing. 5. For BDD I'd say specflow, that integrates easily with many unit testing frameworks. – Yoko

I like the sound of xUnit, however it has zilch documentation :( – Martelli

@ColonelPanic: xUnit is pretty self explanatory. There is a help file included with the download, and there are a bunch of various resources that explain how to use the extensibility model to do a whole variety of things. Two good documentation pages are How To and Comparisons to other frameworks. – Incontrovertible

xUnit has no documentation whatsoever! eg: Try finding what Trait really does or if you can group different tests within single parent test (eg. all tests within a testfixture). nUnit creates a great hierarchical view instead of xUnit's flat view of tests. Plus the nomenclature makes no sense - facts and theory? Be realistic! Those are better called tests and data. – Rollins

I guess we will agree to disagree here, @Sid. There are other ways to approach software testing...the "classic" nUnit approach is not the only valid approach, nor does it dictate the only valid terms with which we can describe a test, or for that matter describe our software. xUnit.NET also provides the flexibility to build your own test frameworks that exercise code in unique ways. As for reporting, I find xUnit.NET's boring "flat" report to be quite refreshing...I am not a huge fan of digging through trees to find information. – Incontrovertible

I honestly really dislike the xUnit terminology. Turned me off entirely - especially when, as far as I can tell, it adds nothing over more "traditional" names/unit-test. Syntax different - yes. Approach different - no. (MBUnit, which was not compared, has good support for "data driven" Tests.) – Tyr

As I've mentioned in the past, xUnit.NET is not so much about the terminology as about flexibility. Everything about the xUnit.NET framework is extensible. If you don't like [Fact] and [Trait], you are free to extend those attributes and rename them [Test] and [Property]. You are also free to create your own attributes that work in whatever way you want them to, supporting any variety of testing methodology you prefer. It is not about Facts and Traits...it is about flexibility, extensibility, and achieving your goals. – Incontrovertible

But as simple thing as Text Output (Console/Debug/Trace) or integration with logging library is almost impossible in xUnit (it swallows or output, and reported issues were closed as Design Decision). – Unbecoming

My integration with XUnit was painless, and in another project I am currently living a nightmare that NUnit is. – Fossa

Answer 2

NUnit is probably the most supported by the 3rd party tools. It's also been around longer than the other three.

I personally don't care much about unit test frameworks, mocking libraries are IMHO much more important (and lock you in much more). Just pick one and stick with it.

California

what is your top mock library pick? – Mcglone

I like Moq, RhinoMocks is also good. – California

It might also be worthwhile to check Pex and Moles, the moles part especially is useful for mocking. – Offcenter

MSPec with FakeItEasy... making test cases more readable – Tc

MSpec with NSubstitute and AutoFixture is my choice. – Hueyhuff

Agreed, but I would rule out MSTest immediately (at least in my experience). With its concept of test lists (those vsmdi files) and integrating too tightly with Visual Studio (I never got it to run tests outside of VS2010). – Almonte

MVCContrib TestHelper is very good for mocking – Chema

New to Moq style testing, but, FakeItEasy looks so cool. – Millenarian

Answer 3

I wouldn't go with MSTest. Although it's probably the most future proof of the frameworks with Microsoft behind it's not the most flexible solution. It won't run stand alone without some hacks. So running it on a build server other than TFS without installing Visual Studio is hard. The visual studio test-runner is actually slower than Testdriven.Net + any of the other frameworks. And because the releases of this framework are tied to releases of Visual Studio there are less updates and if you have to work with an older VS you're tied to an older MSTest.

I don't think it matters a lot which of the other frameworks you use. It's really easy to switch from one to another.

I personally use XUnit.Net or NUnit depending on the preference of my coworkers. NUnit is the most standard. XUnit.Net is the leanest framework.

Lilytrotter

I've been dragged kicking and screaming to this same conclusion. I really wanted to use MSTest because of it's integration with Visual Studio, but that's also its weakness. I need to run tests on a non-Microsoft build server and there aint no way I am installing Visual Studio on it just to get that. It's a shame that Microsoft produces great tools and then renders them almost unattainable. – Furbish

+1 for calling out the awfulness that is MSTest. At the end of the day, it doesn't matter which unit testing framework you use, just as long it as is not MSTest – Shoelace

This answer was undoubtedly true in 2010, but it needs revisit because current rewritten MSTest v2 addresses some of issues raised on this answer. (It's open-source, for one) – Help

You are talking about MSTest v1 (proprietary code). Microsoft has since then evolved it to MSTest v2 (open source), which runs with dotnet.exe test on many platforms. It covers most (if not all) main features provided by other test frameworks, and at least for me has a more intuitive interface. – Hardshell

Answer 4

Consider supplementing, not replacing, MSTest with another testing framework. You can keep Visual Studio MSTest integration while getting the benefits of a more full-featured testing framework.

For example, i use xUnit with MSTest. Add a reference to the xUnit.dll assembly, and just do something like this. Suprisingly, it just works!

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Assert = Xunit.Assert;  // <-- Aliasing the Xunit namespace is key

namespace TestSample
{
    [TestClass]
    public class XunitTestIntegrationSample
    {
        [TestMethod]
        public void TrueTest()
        {
            Assert.True(true);  // <-- this is the Xunit.Assert class
        }

        [TestMethod]
        public void FalseTest()
        {
            Assert.False(true);
        }
    }
}

Driscoll

This technique may also work for NUnit, MBUnit, or other testing frameworks mentioned in other answers, but i haven't tried them. – Driscoll

do you think I could get parameterized tests to work with MSTest with this approach, Matt? – Chema

@Chema No. In his example he just used a class from another assembly. If you want parameterized tests you would need another testing framework that is specifically built to extend MSTest. – Larvicide

Suprisingly, it just works! You just called a static function from another assembly. Why are you surprised that it works? Also if you just need assertions that why not use an assembly specifically made for that? – Larvicide

Answer 5

Nunit doesnt work well with mixed-mode projects in C++ so I had to drop it

Hargrove

I am not proud of this answer but I dropped unit testing for that project. I resorted to write a lot of validation procedures for runtime detection of errors – Hargrove

I'd hoped to use NUnit in mixed-mode but also found it inadequate, in the end I went for googletest which is an excellent C++ unit test framework and v simple to set up. – Ratha

Answer 6

It's not a big deal on a small/personal scale, but it can become a bigger deal quickly on a larger scale. My employer is a large Microsoft shop, but won't/can't buy into Team System/TFS for a number of reasons. We currently use Subversion + Orcas + MBUnit + TestDriven.NET and it works well, but getting TD.NET was a huge hassle. The version sensitivity of MBUnit + TestDriven.NET is also a big hassle, and having one additional commercial thing (TD.NET) for legal to review and procurement to handle and manage, isn't trivial. My company, like a lot of companies, are fat and happy with a MSDN Subscription model, and it's just not used to handling one off procurements for hundreds of developers. In other words, the fully integrated MS offer, while definitely not always best-of-bread, is a significant value-add in my opinion.

I think we'll stay with our current step because it works and we've already gotten over the hump organizationally, but I sure do wish MS had a compelling offering in this space so we could consolidate and simplify our dev stack a bit.

Charentemaritime

ReSharper does have a compelling offering in this space! – Byran

Your answer is curious, and seems self-contradictory. You say you're a big Microsoft-shop, but won't use TFS (which is the whole point- you won't get the benefit of the vertical integration without it) and use an MSDN subscription model, but are using a non-MS approach. I'm lost, to be honest. Well out-of-date answer, unfortunately. – Catacomb

Answer 7

It's not a big deal, it's pretty easy to switch between them. MSTest being integrated isn't a big deal either, just grab testdriven.net.

Like the previous person said pick a mocking framework, my favourite at the moment is Moq.

Greeley