How exactly do MbUnit's [Parallelizable] and DegreeOfParallelism work?

Question

I thought I understood how MbUnit's parallel test execution worked, but the behaviour I'm seeing differs sufficiently much from my expectation that I suspect I'm missing something!

I have a set of UI tests that I wish to run concurrently. All of the tests are in the same assembly, split across three different namespaces. All of the tests are completely independent of one another, so I'd like all of them to be eligible for parallel execution.

To that end, I put the following in the AssemblyInfo.cs:

[assembly: DegreeOfParallelism(8)]

[assembly: Parallelizable(TestScope.All)]

My understanding was that this combination of assembly attributes should cause all of the tests to be considered [Parallelizable], and that the test runner should use 8 threads during execution. My individual tests are marked with the [Test] attribute, and nothing else. None of them are data-driven.

However, what I actually see is at most 5-6 threads being used, meaning that my test runs are taking longer than they should be.

Am I missing something? Do I need to do anything else to ensure that all of my 8 threads are being used by the runner?

N.B. The behaviour is the same irrespective of which runner I use. The GUI, command line and TD.Net runners all behave the same as described above, again leading me to think I've missed something.

EDIT: As pointed out in the comments, I'm running v3.1 of MbUnit (update 2 build 397). The documentation suggests that the assembly level [parallelizable] attribute is available, but it does also seem to reference v3.2 of the framework despite that not yet being available.

EDIT 2: To further clarify, the structure of my assembly is as follows:

assembly
- namespace
   - fixture
      - tests (each carrying only the [Test] attribute)
   - fixture
      - tests (each carrying only the [Test] attribute)
- namespace
   - fixture
      - tests (each carrying only the [Test] attribute)
   - fixture
      - tests (each carrying only the [Test] attribute)
- namespace
   - fixture
      - tests (each carrying only the [Test] attribute)
   - fixture
      - tests (each carrying only the [Test] attribute)

EDIT 3: OK, I've now noticed that if I only ever run one fixture at a time, the maximum number of tests running concurrently is always 8. As soon as I select multiple fixtures, it drops to either 5 or 6. If I take the contents of two fixtures (currently they contain 12 tests each) and drop them into the same fixture (for a total of 24 tests in that one fixture) that fixture will also always run 8 tests concurrently.

This seems to show that it isn't an issue in the individual tests, but rather in how the assembly level attributes percolate down to the fixture, or how the test runner consumes those attributes.

Additionally, I also observed (when running two fixtures) that once one of the two fixtures had been executed in its entirety, the runner starts to execute more tests concurrently when its back down to running only one fixture. For me right now, the first fixture gets done executing when there are 7 tests left to run in the second fixture. As soon as that happens, the number of tests running concurrently jumps up from the previous 5 or 6 to the maximum available of 7.

Heliotrope

Answer 1

According to the release note of Gallio v3.0.6:

MbUnit helps you get the most out of your multi-core CPU. Mark any test [Parallelizable] and it will be permitted to run in parallel with other parallelizable tests in the same fixture.

Fixtures can also be marked parallelizable to enable them to be run in parallel with other parallelizable fixtures.

Please note that if you want all tests within a fixture to be considered parallelizable then you still need to add [Parallelizable] to each of them. (We might add a feature to set this at the fixture or assembly level later based on user feedback.)

Also note that just because a test or fixture is marked parallelizable does not mean it will run in parallel with other tests in particular. For the sake of efficiency, we limit the number of active tests threads based on the configured degree of parallelism. If you want a specific number of instances of a test to run in parallel with each other, consider using [ThreadedRepeat].

The degree of parallelism setting controls the maximum number of tests that MbUnit will attempt to run in parallel with one another. By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.

If you don't like the default then you can override the degree of parallelism at the assembly-level like this:

I don't know if it helps. Maybe Jeff could give more details as he had implemented that feature.

Glidden

I think the Parallelizable attribute at the assembly level is already implemented as of 3.1 – Mallorymallow

3.1 is the version I'm using. The documentation (gallio.org/api/index.aspx) suggests it can be done at the assembly level. The only thing I notice is that the documentation mentions version 3.2 not 3.1. But 3.2 isn't finished yet as far as I can see. – Heliotrope

As far as I can tell, you are using the Parallelizable API correctly. Keep in mind that DegreeOfParallelism only specifies the maximum number of threads to use. There are various reasons for which tests cannot be run in parallel such as test nesting structure, test ordering, and test dependencies. Some constructs like [Repeat] are not parallelized by [Parallelizable]. Likewise individual rows of data-driven tests are not parallelized with respect to each other. (That would be a cool feature to add!) If you set [DegreeOfParallelism(32)] do you see more concurrent tests running on average? – Mckenzie

I don't think I'm using any of the things you mention, but to try and help clarify the exact make up of my test assembly I've updated the original question with the hierarchy. I tried setting DegreeOfParallelism(32), and it did indeed provoke more tests to be run concurrently. My problem is that I'm using them to drive a selenium grid, and as soon as there are more threads than it has remote controls (8) the tests halt whilst the grid hub waits for an RC to become available. That's why I need to limit the DegreeOfParallelism to 8, as that's the number of RC's I have. – Heliotrope

I've just noticed that running a single fixture at a time always results in 8 tests running concurrently. I've updated the question with the further observations. – Heliotrope

Answer 2

Ran into same problem, my findings

• [assembly: Parallelizable(...)] at assembly level overrides fixture Parallelizable attributes and will result in fixture tests being run one at a time but at a fixture parallel level. Seems to have a max of between 5-6 fixtures in parallel.
• [Parallelizable(TestScope.Descendants)] at fixture level will result in fixtures being run one at a time but the tests being run in parallel. Seems to have no max on tests in parallel.

Ultimately due to the assembly level constraint on fixture parallel limits, the only way is to use fixture level attributes and have fixtures tests run in parallel.

I would suggest creating less fixture and more tests per fixture to get around this issue. You could always launch multiple runners per assembly fixture perhaps.

Shame this is the case.

Pulsatory

Answer 3

Override does not work for more than 5 test to run at a time. We have 25 systems on Sauce labs available to execute 25 scripts at a time, we over ride the DegreeOfParallelism to 20 only 5 executes at a time. [assembly: DegreeOfParallelism(20)] - Do not Work for Mbunit

Immediate

The OP disagrees with this along with most of the other answers – Tracee