Execute test sequentially with Xunit in dotnet core

Question

I want to run my tests sequentially as they change the same database and may affect one another. I have tried many solutions on the internet but none of them works for me. These solutions are described in the link "Execute unit tests serially (rather than in parallel)". I am kind of stuck now. Could anyone have some idea on this issue?

Sacrosanct

Answer 1

From the docs:

By default, each test class is a unique test collection.
Tests within the same test class will not run in parallel against each other.

For tests from different classes

If we need to indicate that multiple test classes should not be run in parallel against one another, then we place them into the same test collection.
This is simply a matter of decorating each test class with an attribute that places them into the same uniquely named test collection

[Collection("Database tests")]
public class DeleteTests
{
    [Fact]
    public void RemovesItemFromDatabase()
    {
        // test
    }
}

[Collection("Database tests")]
public class InsertTests
{
    [Fact]
    public void InsertsItemToDatabase()
    {
        // test
    }
}

Decurved

I tried that approach. The decoration did not work for me. The tests inside the same class were still running in parallel. – Sacrosanct

@ThanhBinhNguyen, hmm, I doubt that this is a case. You probably need to share your code and explain how you are observing that tests in same class executed in parallel. – Decurved

Unfortunately, it is not easy to put the code here as my tests requires interacting with several docker programs. To observe the executions, I put debug messages before and after each test. What I saw was that sometimes a test has not finished before another test enters its execution. That how I observed the parallelism. Nevertheless, by the time I write to you this message, I have managed to force executing my tests in sequential manner (but not with the decoration described above). So far, I have not noticed any parallelism behaviors, so hopefully I did correctly. – Sacrosanct

What solution finally worked for you, Thanh Binh Nguyen ? – Lucubration

Indeed, despite this being also in the official documentation of xunit, it doesn't appear to work. – Michelemichelina

Answer 2

To order xUnit tests with custom attributes, you first need an attribute to rely on. Define a PriorityAttribute as follows:

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class PriorityAttribute : Attribute
{
    public PriorityAttribute(int priority)
    {
        Priority = priority;
    }

    public int Priority { get; private set; }
}

Next, consider the following PriorityOrderer implementation of he ITestCaseOrderer interface.

public class PriorityOrderer : ITestCaseOrderer
{
    public const string Name = "Namespace.PriorityOrderer";
    public const string Assembly = "Assembly name";

    private static string _priorityAttributeName = typeof(PriorityAttribute).AssemblyQualifiedName;
    private static string _priorityArgumentName = nameof(PriorityAttribute.Priority);

    private static ConcurrentDictionary<string, int> _defaultPriorities = new ConcurrentDictionary<string, int>();

    public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
    {
        var groupedTestCases = new Dictionary<int, List<ITestCase>>();
        var defaultPriorities = new Dictionary<Type, int>();

        foreach (var testCase in testCases)
        {
            var defaultPriority = DefaultPriorityForClass(testCase);
            var priority = PriorityForTest(testCase, defaultPriority);

            if (!groupedTestCases.ContainsKey(priority))
                groupedTestCases[priority] = new List<ITestCase>();

            groupedTestCases[priority].Add(testCase);
        }

        var orderedKeys = groupedTestCases.Keys.OrderBy(k => k);
        foreach (var list in orderedKeys.Select(priority => groupedTestCases[priority]))
        {
            list.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
            foreach (TTestCase testCase in list)
                yield return testCase;
        }
    }

    private int PriorityForTest(ITestCase testCase, int defaultPriority)
    {
        var priorityAttribute = testCase.TestMethod.Method.GetCustomAttributes(_priorityAttributeName).SingleOrDefault();
        return priorityAttribute?.GetNamedArgument<int>(_priorityArgumentName) ?? defaultPriority;
    }

    private int DefaultPriorityForClass(ITestCase testCase)
    {
        var testClass = testCase.TestMethod.TestClass.Class;
        if (!_defaultPriorities.TryGetValue(testClass.Name, out var result))
        {
            var defaultAttribute = testClass.GetCustomAttributes(_defaultPriorityAttributeName).SingleOrDefault();
            result = defaultAttribute?.GetNamedArgument<int>(_priorityArgumentName) ?? int.MaxValue;
            _defaultPriorities[testClass.Name] = result;
        }

        return result;
    }
}

Add the following attribute to classes for which you want tests run in order:

[TestCaseOrderer(PriorityOrderer.Name, PriorityOrderer.Assembly)]
public abstract class TestsBase
{
    private FieldInfo _counterField;

    public TestsBase()
    {
        _counterField = this.GetType().GetField("_counter", BindingFlags.Static | BindingFlags.NonPublic);

        if (GetCounter() == null)
        {
            var facts = this.GetType().GetMethods()
                .Where(m => m.GetCustomAttribute<FactAttribute>() != null);
            SetCounter(new bool[facts.Count()]);
        }
    }

    private bool[] GetCounter() => _counterField.GetValue(this) as bool[];

    private void SetCounter(bool[] value) => _counterField.SetValue(this, value);

    protected void VerifyAndFlip(int testNumber)
    {
        var counter = GetCounter();
        counter.Take(testNumber).Should().AllBeEquivalentTo(true);
        counter.Skip(testNumber).Should().AllBeEquivalentTo(false);

        counter[testNumber] = true;
        SetCounter(counter);
    }
}

Then decorate your test methods with the Priority attribute.

    [Fact, Priority(1)]
    public async Task Fact1_CorrectValue1_ReturnsTrue()
    {
       Assert.True(true);
    }

    [Fact, Priority(2)]
    public async Task Fact2_Value2_ReturnsTrue()
    {
        Assert.True(true);
    }

Swimming