How can I run NUnit tests in parallel?

Question

I've got a large acceptance test (~10 seconds per test) test suite written using NUnit. I would like to make use of the fact that my machines are all multiple core boxes. Ideally, I'd be able to have one test running per core, independently of other tests.

There is PNUnit, but it's designed for testing for threading synchronization issues and things like that, and I didn't see an obvious way to accomplish this.

Is there a switch/tool/option I can use to run the tests in parallel?

Ashleyashli

Even I would like to know more about this. @Billy ONeal please post an answer if you find one. – Macula

You say ten seconds per test and ideally one test running per core. Are the tests that CPU intensive? Otherwise it should be possible to have many more running concurrently. – Stomatology

@Mattias: Yes, the tests are CPU intensive. – Ashleyashli

Answer 1

If you want to run NUnit tests in parallel, there are at least 2 options:

• NCrunch offers it out of the box (without changing anything, but is a commercial product)
• NUnit 3 offers a Parallelizable attribute, which can be used to denote which tests can be run in parallel

Hoot

This answer seems wrong given the question, more a good suggestion than true answer. To my mind NCrunch needs your upvotes and should be the top answer as it can can run (NUnit) tests concurrently within the same or multiple processes and across one or many machines and will do so from VS or from a build server. – Allegorize

@Allegorize NCrunch was released after this question was answered, and is a good choice (as this should just work out of the box, although it's overly pricey for some). Other options in the future may include NUnit 3, which may offer parallel running of tests (according to github.com/nunit/dev/wiki/Roadmap). – Hoot

Although this is the top answer, it's no longer as correct as it once was. Nunit 3.0 was released at the tail end of 2015 which now has a Parallelizable attribute. Ref: github.com/nunit/nunit/wiki/Parallelizable-Attribute – Table

Answer 2

NUnit version 3 will support running tests in parallel:

Adding the attribute to a class: [Parallelizable(ParallelScope.Self)] will run your tests in parallel.

• ParallelScope.None indicates that the test may not be run in parallel with other tests.

• ParallelScope.Self indicates that the test itself may be run in parallel with other tests.

• ParallelScope.Children indicates that the descendants of the test may be run in parallel with respect to one another.

• ParallelScope.Fixtures indicates that fixtures may be run in parallel with one another.

NUnit Framework-Parallel-Test-Execution

Darondarooge

Answer 3

If your project contains multiple test DLLs you can run them in parallel using this MSBuild script. Obviously you'll need to tweak the paths to suit your project layout.

To run with 8 cores run with: c:\proj> msbuild /m:8 RunTests.xml

RunTests.xml

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="RunTestsInParallel" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
  <PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
    <Nunit Condition=" '$(Nunit)' == '' ">$(MSBuildProjectDirectory)\..\tools\nunit-console-x86.exe</Nunit>
  </PropertyGroup>

  <!-- see http://mikefourie.wordpress.com/2010/12/04/running-targets-in-parallel-in-msbuild/ -->

  <Target Name="RunTestsInParallel">
    <ItemGroup> 
      <TestDlls Include="..\bin\Tests\$(Configuration)\*.Tests.dll" />
    </ItemGroup>

    <ItemGroup> 
      <TempProjects Include="$(MSBuildProjectFile)" > 
        <Properties>TestDllFile=%(TestDlls.FullPath)</Properties> 
      </TempProjects> 
    </ItemGroup> 

    <MSBuild Projects="@(TempProjects)" BuildInParallel="true" Targets="RunOneTestDll" /> 
  </Target>

  <Target Name="RunOneTestDll"> 
    <Message Text="$(TestDllFile)" />
    <Exec Command="$(Nunit) /exclude=Integration $(TestDllFile)  /labels /xml:$(TestDllFile).results.xml"
      WorkingDirectory="$(MSBuildProjectDirectory)\..\bin\Tests\$(Configuration)" /> 
  </Target>

</Project>

Update If I were answering this question now I would highly recommend NCrunch and its command line test running tool for maximum test run performance. There's nothing like it and it'll revolutionise your code-test-debug cycle at the same time.

Allegorize

This helped me to reduce unit tests run time from 3 min to 2 min. I've tested on a 2-core CPU. – Pent

Answer 4

As an alternative to adding the Parallelizable attribute to every test class:

Add this into the test project AssemblyInfo.cs class for nunit3 or greater:

// Make all tests in the test assembly run in parallel
[assembly: Parallelizable(ParallelScope.Fixtures)]

Destalinization

I had to use [assembly: Parallelizable(ParallelScope.All)] – Rutan

The correct answer is actually [assembly: Parallelizable(ParallelScope.ContextMask)] in AssemblyInfo.cs ContextMask is not documented, but if you open up the Enum, you will discover that ContextMask = (ParallelScope.Fixtures | ParallelScope.Children) Which means every single test that exists in this assembly will be considered parallelizable, unless you set a different value for a specific class, method, or test. – Graviton

Answer 5

In this article it is mentioned that in order to speed up tests the poster runs multiple instances of NUnit with command parameters specifying which tests each instance should run.

FTA:

I ran into an odd problem.

We use nunit-console to run test on our continuous integration server. Recently we were moving from Nunit 2.4.8 to 2.5.5 and from .Net 3.5 to 4.0. To speed up test execution we run multiple instances of Nunit in parallel with different command line arguments

• We have two copies of our test assemblies and the nunit binaries in folder A and B.
• In folder A we execute

nunit-console-x86.exe Model.dll Test.dll /exclude:MyCategory /xml=TestResults.xml /framework=net-4.0 /noshadow

• In folder B we execute

nunit-console-x86.exe Model.dll Test.dll /include:MyCategory /xml=TestResults.xml /framework=net-4.0 /noshadow

If we execute the commands in sequence both run successfully. But if we execute them in parallel only one succeeds. As far as I can tell it's the one that first loads the test fixtures. The other fails with the message "Unable to locate fixture".

Is this problem already known? I could not find anything related in the bug list on launchpad. BTW Our server runs Windows Server 2008 64-bit. I could also reproduce the problem on Windows 7 64-bit.

Assuming this bug is fixed or you are not running the newer version(s) of the software mentioned you should be able to replicate their technique.

Update

TeamCity looks like a tool you can use to automatically run NUnit tests. They have an NUnit launcher discussed here that could be used to launch multiple NUnit instances. Here is a blog post discussing the mergind of multiple NUnit XML results into a single result file.

So theoretically you could have TeamCity automatically launch multiple NUnit tests based on however you want to split up the workload and then merge the results into a single file for post test processing.

Is that automated enough for your needs?

Johannejohannes

This is the same idea already posted about categories... I don't want to have to maintain reasonably equal runtimes between the instances here. I'd rather write my own NUnit runner before I'd do this. – Ashleyashli

As far as I know NUnit does not support this without a workaround such as running multiple instances. If you want I guess you can make a tool that splits the tests into N sets and runs N instances of NUnit automatically where N is the number of processors/cores you have. That would be the only way to have some sort of automated parallel testing that I can think of with NUnit. – Johannejohannes

I added an update discussing the TeamCity continuous integration tool and included a few posts on how to use that tool to solve your automation needs. – Johannejohannes

Answer 6

Just because PNUnit can do synchronization inside test code doesn't mean that you actually have to use that aspect. As far as I can see there's nothing to prevent you from just spawning a set and ignoring the rest till you need it.

BTW I don't have the time to read all of their source but was curious to check out the Barrier class and it's a very simple lock counter. It just waits till N threads enter and then sends the pulse for all of them to continue running at the same time. That's all there is to it - if you don't touch it, it won't bite you.

Might be a bit counter intuitive for a normal threaded development (locks are normally used to serialize access - 1 by 1) but it is quite a spirited diversion :-)

Easting

Answer 7

You can now use NCrunch to parallelize your unit tests and you can even configure how many cores should be used by NCrunch and how many should be used by Visual Studio.

plus you get continuous testing as a bonus :)

Hemophilia

NCrunch is great when you're using it within Visual Studio, but it doesn't help when you're trying to parallelize your tests on your build server. – Rosendorosene

NCrunch now has a command line tool available that works really nicely on build servers. – Allegorize

Answer 8

It would be a bit of a hack, but you could split the unit tests into a number of categories. Then, start up a new instance of NUnit for each category.

Edit: It looks like they have added a /process option to the console app. The command-line help states this is the "Process model for tests: Single, Separate, Multiple". The test runner also appears to have this feature.

Edit 2: Unfortunately, although it does create separate processes for each assembly, the process isolation option (/process from the command line) runs the agents one at a time.

Tellford

Answer 9

Since the project hasn't been mentioned here, I would like to bring up NUnit.Multicore. I haven't tried the project myself, but it seems to have an interesting approach to the parallel test problem with NUnit.

Chirm

Answer 10

You can try my small tool TBox or console parallel Runner or even plugin to do distributed calulations, which also can run unit tests on the set of PCs SkyNet

TBox is created to simplify work with big solutions, which contains many projects. It supports many plugins and one of them provide ability to run NUnit tests in parallel. This plugin does not require any changes to your existing tests.

Also it support:

• Cloning of the folder with unit test (if your tests changes local data),

• Synchronizations of the tests (for example if your tests on testfixtureteardown kills all dev servers or chromerunner for qunit )

• x86 mode and Admin privileges to run tests

• Batch run - you can run tests for many assemblies in parallel

• Even for single thread run, works faster than standart nunit runner, if you have much small tests.

Also this tool supports command line tests runner (for parallel run) and you can use it with continuous integration.

Albi

If you are associated with this product, please disclose this. You've posted multiple things pointing to this. – Dog

Sure, TBox - is my own tool. I have written it in my free time alone. If it is bad practice here to say about free tools, I'll remove this answer, it's no problem :) – Albi

We just want people to be clear about the products they're involved with. If you fully disclose that it's yours, why it would be appropriate to solve the problem posed by the question, and don't promote it too aggressively, your answers should be acceptable here. – Dog

@brad It's open source on Codeplex, not sure why there is an issue here. – Mandie

@DanVallejo - Being an open source project is the reason why this wasn't deleted by the community upon posting. We give a little more leeway there, but we do still ask that you disclose your affiliation with the project so that people can understand the context behind your recommendation. Alex did so here, so his answer is perfectly fine as it stands now. – Dog

Answer 11

I have successfully used NUnit 3.0.0 beta-4 to run tests in parallel

• Runs on build server
• Runs Selenium tests
• Has Visual Studio support
• no Resharper support yet

Thanks for peers answer.

Gotchas:

• Parallelizable attribute is not inherited, so it has to be specified on the test class.

Markland

NUnit 3.0 is now out – Catboat

Answer 12

You can use following PowerShell command (for NUnit3, for NUnit2 change runner name):

PS> nunit3-console (ls -r *\bin\Debug\*.Tests.dll | % FullName | sort-object -Unique)

Presented command runs all test assemblies in single nunit instance, which allows to leverage engine built-in parallel test run.

Remarks

1. Remember to tweak directory search pattern. Given example runs only assemblies ending with .Tests.dll and inside \bin\Debug directories.

2. Be aware of Unique filtering - you may not want to have it.

Ionium

Answer 13

To achieve level of parallelism ensure to do these two:

1)Nunit Explorer - Settings - Run tests in parallel

2)LevelOfParallelism

This is an assembly-level attribute used to specify the level of parallelism, that is, the maximum number of worker threads executing tests in the assembly.

In Assemblyinfo.cs, set

[assembly:LevelOfParallelism(N)] => here N is number

Alfieri