ScalaTest: How to mix parallel and sequential tests

Question

Say I have 6 test suites: A B C D E F, and I want A B C to run sequentially and THEN run D E F in parrallel.
With an output like that:

A
B
C // always in that order
E
D
F // The order doesn't matter

The idea is to be able to test ABC in isolation of the rest of the tests.

What I have already tried

Create a super sequential test class like that and adding @DoNotDiscover on the sequential tests.

class MasterSuite extends Stepwise(
  Sequential(new A, new B, new C)
)

But, even if A B C are run sequentially, then are run in parrallel with the other tests.

I have also tried that

class MasterSuite extends Stepwise(
  Sequential(new A, new B, new C),
  Suites(new D, new E, new F)
)

But for me it run me all tests sequentially (maybe I have miss something in the build.sbt file).

Englishry

Answer 1

The documentation for Stepwise says the following:

The difference between Stepwise and Sequential is that although Stepwise executes its own nested suites sequentially, it passes whatever distributor was passed to it to those nested suites. Thus the nested suites could run their own nested suites and tests in parallel if that distributor is defined. By contrast, Sequential always passes None for the distributor to the nested suites, so any and every test and nested suite contained within the nested suites passed to the Sequential construtor will be executed sequentially.

So the obvious question is: what Distributor is passed to the runNestedSuites method of MasterSuite? Because that Distributor is what's ultimately going to be passed to the runNestedSuites method of the Suites object that contains D, E and F.

Through experimentation with a debugger, I found that the Distributor is normally None. But if you mix the ParallelTestExecution trait into your MasterSuite class, you will get a Some instead, and I've verified that in a debugger too.

class MasterSuite extends Stepwise(
  new A,
  new B,
  new C,
  new Suites(new D, new E, new F)) with ParallelTestExecution

Now, the MasterSuite will run A, B and C sequentially and then start running the other suites in parallel.

So, problem solved? Unfortunately no, because while it apparently started running D, E and F in parallel, it didn't wait for them to finish and just declared them all successful – even though I deliberately added a failing test in F to see if everything works correctly. So as far as I can see, this is how it's supposed to be done and it's just broken.

Which leads me to my personal conclusion after many years of experience with ScalaTest: it's a bug-ridden piece of garbage, and I would highly recommend staying away from it. I'm sorry I can't give a more optimistic answer than that.

Pankey

Thank you for your answer even if it's doesn't really fit my expectation – Englishry

Answer 2

Could be related to this issue, which I just reported.

Absorptivity

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Digitigrade