How to run SpecFlow tests in parallel

Question

I try to run SpecFlow tests with NUnit test runner in parallel. I am using:

• C# / .Net Core 3.1
• NUnit as testrunner
• Specflow

I have added this line on top of a file (as describe here: https://specflow.org/documentation/parallel-execution/):

[assembly: Parallelizable(ParallelScope.All)]

Tests start to execute in parallel but immediately throw an error:

Message: 
    System.ArgumentException : An item with the same key has already been added. Key: NUnitTestProject1.SpecFlowFeature1Steps
Stack Trace: 
    Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
    Dictionary`2.Add(TKey key, TValue value)
    TypeRegistration.Resolve(ObjectContainer container, RegistrationKey keyToResolve, ResolutionList resolutionPath)
    ObjectContainer.ResolveObject(RegistrationKey keyToResolve, ResolutionList resolutionPath)
    ObjectContainer.Resolve(Type typeToResolve, ResolutionList resolutionPath, String name)
    ObjectContainer.Resolve(Type typeToResolve, String name)
    TestObjectResolver.ResolveBindingInstance(Type bindingType, IObjectContainer container)
    lambda_method(Closure , IContextManager , Int32 )
    BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration)
    TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments)
    TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance)
    TestExecutionEngine.OnAfterLastStep()
    TestRunner.CollectScenarioErrors()
    SpecFlowFeature1Feature.ScenarioCleanup()
    SpecFlowFeature1Feature.AddTwoNumbers() line 8

I tried without SpecFlow (just raw NUnit) and it worked. I also tried with MSTest test runner but got the same exception.

I have made a very tiny example project which you can clone here: https://github.com/davidguidali/specflowerror

Any help would be appreciated. :)

EDIT: Full code of steps

using NUnit.Framework;
using System;
using System.Threading;
using TechTalk.SpecFlow;

[assembly: Parallelizable(ParallelScope.All)]

namespace NUnitTestProject1
{
    [Binding]
    public class SpecFlowFeature1Steps
    {
        [Given(@"test(.*)")]
        public void GivenTest(int p0)
        {
            Thread.Sleep(5000);
            Assert.IsTrue(true);
        }
    }
}

Loggia

Please post the code in SpecFlowFeature1Steps – Penland

Maybe this answer (#46554672) can help you – Walkthrough

Answer 1

From the documentation, the SpecFlow+ Runner supports Scenario parallelism. The NUnit, MsTest, xUnit runners only support Feature parallelism.

I noticed you are using NUnit. It is also documented under NUnit configuration: "Note: SpecFlow does not support scenario level parallelization with NUnit (when scenarios from the same feature execute in parallel). If you configure a higher level NUnit parallelization than “Fixtures” your tests will fail with runtime errors."

Obadias

Answer 2

I'm using this:

[assembly: Parallelizable(ParallelScope.Fixtures)]

Warplane

Thats not what I want. I want to run all the tests in parallel. ParallelScope.Fixtures only runs tests as per feature (class) in parallel. I want to run them also as per Scenario (method). – Loggia

You can run SpecFlow tests only in this parallel mode. Scenarios in one Feature are executed one by one. As for non-specFlow tests from one class: yes, they can run in parallel. – Warplane

Is there any official source for this? I couldn't find anything. Why is this the case? Is it the same for MSTest(s)? – Loggia

Check here: specflow.org/documentation/parallel-execution – Warplane

its nowehere written that execution on method level does not work. – Loggia

See also github.com/SpecFlowOSS/SpecFlow/issues/1535 – Emmenagogue

Answer 3

You should create new class usings.cs with

global using NUnit.Framework;
[assembly: Parallelizable(ParallelScope.Fixtures)] // Set on parallel
[assembly: LevelOfParallelism(13)] // Count of threads

Hubbs

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Civilly