How to log from Python Luigi
Asked Answered
E

2

10

I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging level at runtime. Our system uses Luigi to call spark from Jenkins. Thank you in advance.

Entropy answered 16/6, 2016 at 18:48 Comment(0)
I
15

Inside any of the Task class methods, you can do:

class Agg(luigi.Task):
  _date = luigi.DateParameter()

  def output(self):
    return luigi.LocalTarget("file_%.txt" % self._date)

  def run(self):
    # Use the luigi-interface to log to console
    logger = logging.getLogger('luigi-interface')
    logger.info("Running --> Agg.Task")
Incise answered 31/3, 2019 at 17:24 Comment(0)
D
0

Have you checked the logging_conf_file parameter of the configuration? you can set up there all your configuration regarding logging using Python's standard logging mechanism.

For some examples see:

Drunkometer answered 20/6, 2016 at 10:24 Comment(1)
It would be really great if you could provide solutions to answers rather than links to other issue pages. The other issues and their contents are just as unhelpful.Fragonard

© 2022 - 2024 — McMap. All rights reserved.