parallelizing tasks in Luigi Orchestrator

Question

I defined three tasks T1, T2, and T3, and then a task T4 as follows:

class T4(luigi.Task)
    def requires(self):
        return [T1(), T2(), T3()]

Is there a natural way to tell Luigi that I want these tasks T1, T2, and T3 to be executed in parallel?

Skinhead

Answer 1

It depends on what dependencies T1, T2 and T3 have. If they haven't another task as a common dependency, you can just run your task specifying --workers=3 and Luigi will run each task in a separate worker.

Contessacontest

Can this be set through the luigi.cfg as well? – Understanding