What is limiting the number of active workers in a Gradle build?

Question

I am trying to run tests in parallel on my laptop, which has 4 pyhsical and 8 logical CPUs.

➜  sysctl -n hw.ncpu
8
➜  sysctl -n hw.physicalcpu
4

I would like to run at most 4 tests in parallel.
This is the command that I am running:

./gradlew remoteChromeTest -Pparallel=4 --continue --max-workers=8

The parallel property goes into maxParallelForks of the Test task.

Still, Gradle only seems to occupy at most 4 workers concurrently (one of which seems to be reserved for the build itself), which coincides with the number of physical CPUs (not sure if that matters).

The output looks like this:

> Task :remoteChromeTest
Running tests in parallel using 4 processes.
<============-> 92% EXECUTING [12s]
> :remoteChromeTest > 0 tests completed
> :remoteChromeTest > Executing test spec.Spec1
> IDLE
> :remoteChromeTest > Executing test spec.Spec2
> :remoteChromeTest > Executing test spec.Spec3

So, only 3 tests are running in parallel.

What am I missing here? The documentation suggests this could be cranked up beyond the number of actual CPUs. The weird part is that one of the workers shows up as IDLE.

I am getting the same behaviour on another machine, which has only 2 physical cores, and there it is limited to two processes, i.e. no parallel execution at all.

Why is one worker IDLE?

Ce

Answer 1

--max-workers is not only for tests execution but also for parallel project execution. In the example, however, you are limiting tests execution parallelism to 4. Try setting maxParallelForks to more than 4 and you will see more parallelism in tests execution. But anyway it will be limited to the minimum between the number of tests and --max-workers.

You can also enable parallel project execution by using the --parallel option. And again, the build parallelism, including the test parallelism, will be limited to --max-workers:

subprojects {
    tasks.withType(Test) {
        maxParallelForks = 8
    }
}

gradle clean build --parallel --max-workers=6
...
<<===========--> 85% EXECUTING [15s]
>> :module-1:test > 4 tests completed
>> :module-2:test > 2 tests completed
>> :module-2:test > Executing test so.Module2Spec3
>> :module-2:test > Executing test so.Module2Spec1
>> :module-1:test > Executing test so.Module1Spec4
>> :module-1:test > Executing test so.Module1Spec2
>> :module-1:test > Executing test so.Module1Spec1
>> :module-1:test > Executing test so.Module1Spec3

gradle clean build --parallel --max-workers=12
...
<===========--> 85% EXECUTING [13s]
> :module-1:test > 0 tests completed
> :module-2:test > 0 tests completed
> :module-2:test > Executing test so.Module2Spec1
> :module-2:test > Executing test so.Module2Spec2
> :module-2:test > Executing test so.Module2Spec3
> :module-2:test > Executing test so.Module2Spec4
> :module-1:test > Executing test so.Module1Spec1
> :module-1:test > Executing test so.Module1Test2
> :module-1:test > Executing test so.Module1Spec4
> :module-1:test > Executing test so.Module1Test3
> :module-1:test > Executing test so.Module1Test1
> :module-1:test > Executing test so.Module1Spec3
> :module-1:test > Executing test so.Module1Spec2
> :module-1:test > Executing test so.Module1Test4

Miocene

This did not really answer my question. I know I am limiting to 4 tests, because that's what I want (the system under test cannot handle more at the same time). I wrote that also on my question. But I am wondering, why am I getting only three parallel tests, and one IDLE. My question is: what is causing the IDLE worker? – Ce

@ThomasHirsch have you found an answer for that? – Flounder

@DiegoMarinSantos No, and I do not have a way to check on this again right now. – Ce

Answer 2

The answer is there is no executable task left, the rest of tasks are depended on the current running ones, you will see that the status IDLE will change after some of in progress tasks are done.

Reprehend