How to make a Parameter available to all Luigi Tasks?

Question

In the Luigi docs, the use of a luigi.Config class is recommended for global configuration.

However, I am running into issues when using such a config class in order to pass a commandline argument to various Tasks in the pipeline.
Here's a lightweight example:

import datetime
import luigi


class HelloWorldTask(luigi.Task):

    def run(self):
        print("{task} says: Hello world on {date}!".format(task=self.__class__.__name__,
                                                           date=GlobalParams.date.strftime('%d-%b-%Y')))


class GlobalParams(luigi.Config):
    date = luigi.DateParameter(default=datetime.date.today())


if __name__ == '__main__':
    luigi.run(['HelloWorldTask', '--workers', '1', '--local-scheduler',
               '--GlobalParams-date', '2018-01-01'])

The class GlobalParams defines a DateParameter which I would like to later reference in the run() blocks of pipeline Tasks. However, this fails with the error, AttributeError: 'DateParameter' object has no attribute 'strftime'.

In the debugger, I can see that a DateParameter object is passed to the HelloWorldTask Task, but any attempts to extract the expected '2018-01-01' value passed at runtime fails.

Am I misunderstanding how to use these constructs? How should I be passing a single parameter to (possibly many) Tasks?

Incurious

just access the default value this way: GlobalParams.date._default.strftime('%d-%b-%Y') – Gluey

This only gives access to the default value as given in the parameter definition. I want to access the date passed to the parameter at runtime ('2018-01-01'). – Incurious

Answer 1

The issue in the example code is that the GlobalParams class is not being instantiated before its parameter is accessed.
GlobalParams.date.strftime('%d-%b-%Y') should read GlobalParams().date.strftime('%d-%b-%Y').

This is included, but easy to overlook, in the configuration docs linked in the question.

Incurious

This doesn't work if I try to run with more than 1 workers, any idea how to fix it? – Extract

Beware Windows people, it seems that Veltzer Doron recently opened a bug at Spotify, and the people there confirmed that Global parameters are not propagated to parallel workers on Windows (they are on Linux). See Global parameters when workers != 1 not being passed to workers · Issue #2693 · spotify/luigi. – Emia