NUnit, is it possible to continue executing test after Assert fails?

Question

In a test that contains some asserts, for example:

Assert.AreEqual(1,1);
Assert.AreEqual(2,1);
Assert.AreEqual(2,2);

is it possible to let the test keep running after it fails at some point? In the example, first condition is true, second fails and the test stops. I'd like to evaluate also the following condition.

Predecessor

There are plenty of valid instances where you may want multiple asserts. For example, let's say that you're creating a collection of x objects. It's arguable that you may want to test whether the collection is not null first, and then that it has x objects. If your test runs without the first check in place and the collection is not initialised, you'll get a null ref exception, which can be unhelpful. – Yurik

Answer 1

NUnit 3.6 adds Assert.Multiple method and MultipleAsserts attribute.

See https://docs.nunit.org/articles/nunit/writing-tests/assertions/multiple-asserts.html

Vasculum

From the last comment here (github.com/nunit/nunit/issues/391) it seems that this feature was postponed until version 3.2, which should be ready somewhere this month from the Github milestone, however, currently it's not available as of now. – Urita

It is available only in 3.6 – Nysa

Answer 2

I prefer to be practical and put several related assertions in one method.

I have a helper class which enables the following syntax (I use):

AssertAll.Succeed(
    () => Assert.AreEqual("bb", id.Context),
    () => Assert.AreEqual("abc", id.FullName),
    () => Assert.AreEqual("b", id.SessionID));

which gives me error messages like this:

Assert.AreEqual failed. Expected:<bb>. Actual:<b\c>. 
Assert.AreEqual failed. Expected:<abc>. Actual:<[b\c]a{103}>. 
at FXP_COM.Tests.EnumToStringConverterterTests.<>c__DisplayClass3.<ShouldConvert>b__0() in UnitTest1.cs: line 31
at FXP_COM.Tests.AssertAll.Succeed(Action[] assertions) in UnitTest1.cs: line 46 at FXP_COM.Tests.AssertAll.Succeed(Action[] assertions) in UnitTest1.cs: line 62
at FXP_COM.Tests.EnumToStringConverterterTests.ShouldConvert() in UnitTest1.cs: line 30

and the helper class is the following:

using System;
using NUnit.Framework;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;

public static class AssertAll
{
    public static void Succeed(params Action[] assertions)
    {
        var errors = new List<Exception>();

        foreach (var assertion in assertions)
            try
            {
                assertion();
            }
            catch (Exception ex)
            {
                errors.Add(ex);
            }

        if (errors.Any())
        {
            var ex = new AssertionException(
                string.Join(Environment.NewLine, errors.Select(e => e.Message)),
                errors.First());

            // Use stack trace from the first exception to ensure first
            // failed Assert is one click away
            ReplaceStackTrace(ex, errors.First().StackTrace);

            throw ex;
        }
    }

    static void ReplaceStackTrace(Exception exception, string stackTrace)
    {
        var remoteStackTraceString = typeof(Exception)
            .GetField("_remoteStackTraceString",
                BindingFlags.Instance | BindingFlags.NonPublic);

        remoteStackTraceString.SetValue(exception, stackTrace);
    }
}

Beaker

Answer 3

No. Typically in this situation you would put all the code above the asserts into a setup method, then write each assert into its own test case.

Intendment

@Anna [TestCase] is useful when you have a numeric procedure, but if your setup is something more complex it quickly becomes a bit of a spagetti situation – Paction

I agree with all of you with one assert per test; btw in this particular situation Anna's suggestion to use [TestCase] or catching AssertionException as suggested below do the trick. – Predecessor

Answer 4

No you can't do it with NUnit alone. You have to do something like @Konstantin Spirin said. I created a small extension that you can use; it's called NUnit-GroupAssert. It can be found here: https://github.com/slvnperron/NUnit-GroupAssert

How to use it:

[Test]
public void Verify_GroupsExceptions()
{
    var group = new AssertGroup();
    group.Add(() => Assert.AreEqual(10, 20));
    group.Add(() => Assert.AreEqual(1, 1));
    group.Add(() => Assert.AreEqual(3, 4));
    group.Add(() => Assert.IsTrue(1 > 3));
    group.Verify();
}

// OR

public void Verify_GroupsExceptions()
{
    // Verifies on disposal
    using (var group = new AssertGroup())
    {
        group.Add(() => Assert.AreEqual(10, 20));
        group.Add(() => Assert.AreEqual(1, 1));
        group.Add(() => Assert.AreEqual(3, 4));
        group.Add(() => Assert.IsTrue(1 > 3));
    }
}

it will output:

Test failed because one or more assertions failed:
1) Expected: 10
But was: 20
From Verify_GroupsExceptions at line 18

2) Expected: 3
But was: 4
From Verify_GroupsExceptions at line 20

3) Expected: True
But was: False
From Verify_GroupsExceptions at line 21

Wiatt

Answer 5

You could restructure your test to wrap the assertions in try/catch block and keep track of them for later validation. I don't recommend this, however. You really should be using separate tests for each condition if you want them to be tested independently.

  bool[] assertionSuccesses = new bool[] { false, false, false };

  try
  {
       Assert.AreEqual( 1, 1 );
       assertionSuccesses[0] = true;
  }
  catch (AssertionException) {}
  ...

  if (assertionSuccesses.Any( s => !s ))
  {
       Assert.Fail("one of the assertions failed");
  }

Occasional

Answer 6

You can cheat a little and not actually fail at a given point, but rather mark for failure, then fail at the very end, something like the following:

var sbError = new StringBuilder();
if (!SomeCondition()) {
  sbError.AppendLine("SomeCondition failed");
}
if (!SomeOtherCondition()) {
  sbError.AppendLine("SomeOtherCondition failed");
}
Assert.AreEqual(0, sbError.Length, sbError.ToString());

I wouldn't recommend this, but if you need to do it once or twice, it shouldn't be that bad.

Unilingual

Answer 7

Nope. You shouldn't really have more than one assert per test anyway, that reduces the seperation and makes it more difficult to find out which one failed.

If you have a lot of code that needs executing before the Assert, seperate it out into a [SetUp] function, or make it a seperate procedure.

Paction

one assertion per test is rather pedantic. I prefer to think of it as "assert one thing" per test. It may take multiple assertions to assert that "one thing", like does the thing I created have all the right properties. I'd prefer this over having one test per property because what you're really testing is "did the object get created properly?" – Occasional

@Occasional I'm just going with the general consensus here: it's easy enough to use [SetUp] to set up the relevant object, then do an Assert on it from multiple tests: it makes your tests easier to read, and if you use a CI thing (TeamCity, CruiseControl.Net, etc.) that handles unit tests for you it's much easier to see exactly what failed. – Paction

@Occasional - IMHO, it would be okay to not write separate test case for each assert. But after the complexity builds, the test case will become brittle. That's when you want to refactor your test case to individual test cases for each assert. – Myopic

No one ever mentions integration tests. In an integration test it is fine to do multiple asserts, and may even make more sense. – Kootenay

Answer 8

Asserts thrown an NUnit.Framework.AssertionException if they fail. You could catch that exception on the second assert, evaluate the third assert, then re-throw the exception.

Not something I'd recommend, though, for the reasons pointed-out by Ed Woodcock and Carl Manaster.

Homozygous

That's one of those things that's useful to know, but also very bad to know, because you might actually do it :) – Paction

@Ed I agree, but the OP might have a good reason for doing this – Homozygous

other than laziness I can't think of one, apart from testing state-dependant data, which shouldn't really be done like that anyway. – Paction

Answer 9

I rewrote Konstantin Spirin's code sample into VB.

Imports NUnit.Framework
Imports System.Reflection

Public Class Group_LIB_NUnit_Assert_Multiple

    Public Shared Sub Multiple(ParamArray Assertions As Action())
        Dim ExceptionObj As Exception
        Dim Exceptions As New List(Of Exception)
        Dim Message As String

        For Each Assertion In Assertions
            Try
                Assertion()
            Catch ex As Exception
                Exceptions.Add(ex)
            End Try
        Next

        If Exceptions.Count > 0 Then
            Message = String.Format("{0}{1} assertions failed: {2}{3}", Environment.NewLine, Exceptions.Count, Environment.NewLine, String.Join(Environment.NewLine, Exceptions.Select(Function(e) e.Message).ToList()))

            ExceptionObj = New AssertionException(Message)

            StackTraceReplace(ExceptionObj, Exceptions.First.StackTrace)

            Throw ExceptionObj
        End If
    End Sub

    Public Shared Sub StackTraceReplace(ExceptionObj As Exception, StackTrace As String)
        Dim RemoteStackTraceString As FieldInfo

        RemoteStackTraceString = GetType(Exception).GetField("_remoteStackTraceString", Reflection.BindingFlags.Instance Or Reflection.BindingFlags.NonPublic)

        RemoteStackTraceString.SetValue(ExceptionObj, StackTrace)

    End Sub
End Class

Imports Group.Library4
Imports NUnit.Framework

<TestFixture()> _
Public Class TEST_Group_LIB_NUnit_Assert_Multiple

    <Test()> _
    <TestCaseSource("Factory")> _
    Public Sub Multiple(TestObj As TEST_DATA_Group_LIB_NUnit_Assert_Multiple)

        Group_LIB_NUnit_Assert_Multiple.Multiple(Sub() Assert.That(TestObj.Gender, [Is].EqualTo("F"c), "Gender"), Sub() Assert.That(TestObj.Name, [Is].EqualTo("George Washington"), "Name"))
    End Sub

    Public Function Factory() As List(Of TEST_DATA_Group_LIB_NUnit_Assert_Multiple)
        Dim L As New List(Of TEST_DATA_Group_LIB_NUnit_Assert_Multiple)
        Dim TestObj As TEST_DATA_Group_LIB_NUnit_Assert_Multiple

        TestObj = New TEST_DATA_Group_LIB_NUnit_Assert_Multiple()
        TestObj.DOB = New DateTime(2015, 8, 12)
        TestObj.Gender = "M"c
        TestObj.Name = "Abraham Lincoln"
        L.Add(TestObj)

        TestObj = New TEST_DATA_Group_LIB_NUnit_Assert_Multiple()
        TestObj.DOB = New DateTime(2015, 8, 12)
        TestObj.Gender = "F"c
        TestObj.Name = "George Washington"
        L.Add(TestObj)

        TestObj = New TEST_DATA_Group_LIB_NUnit_Assert_Multiple()
        TestObj.DOB = New DateTime(2015, 8, 12)
        TestObj.Gender = "A"c
        TestObj.Name = "John Hancock"
        L.Add(TestObj)

        Return L
    End Function
End Class

Public Class TEST_DATA_Group_LIB_NUnit_Assert_Multiple

    Public Property DOB As Date

    Public Property Gender As Char

    Public Property Name As String

End Class

Photina

Answer 10

Try to use Assertions in stepDefinitions class then remaining test scenarios and features will get executed

Apian

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Pellagra