I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging level at runtime. Our system uses Luigi to call spark from Jenkins. Thank you in advance.
How to log from Python Luigi
Asked Answered
Inside any of the Task class methods, you can do:
class Agg(luigi.Task):
_date = luigi.DateParameter()
def output(self):
return luigi.LocalTarget("file_%.txt" % self._date)
def run(self):
# Use the luigi-interface to log to console
logger = logging.getLogger('luigi-interface')
logger.info("Running --> Agg.Task")
Have you checked the logging_conf_file parameter of the configuration? you can set up there all your configuration regarding logging using Python's standard logging mechanism.
For some examples see:
It would be really great if you could provide solutions to answers rather than links to other issue pages. The other issues and their contents are just as unhelpful. –
Fragonard
© 2022 - 2024 — McMap. All rights reserved.