How to handle output with Luigi

Question

I'm trying to grasp how luigi works, and I get the idea, but actual implementation is a bit harder ;) This is what i have:

class MyTask(luigi.Task):

    x = luigi.IntParameter()

    def requires(self):
        return OtherTask(self.x)

    def run(self):
        print(self.x)

class OtherTask(luigi.Task):

    x = luigi.IntParameter()

    def run(self):
        y = self.x + 1
        print(y)

And this fails with RuntimeError: Unfulfilled dependency at run time: OtherTask_3_5862334ee2. I've figured that I need to produce output using def output(self): to workaround this issue\feature. And I can't comprehend how do I produce reasonable output without writing to a file, say:

def output(self):
    return luigi.LocalTarget('words.txt')

def run(self):

    words = [
            'apple',
            'banana',
            'grapefruit'
            ]

    with self.output().open('w') as f:
        for word in words:
            f.write('{word}\n'.format(word=word))

I've tried reading the documentation, but I can't understand the concept behind output at all. What if I need to output to screen only. What if I need to output an object to another task? Thanks!

Arthur

You have a bunch of questions mixed in here, but only one with a question mark. – Listel

Answer 1

What if I need to output an object to another task?

Luigi tasks can run in different processes. Therefore you do usually have to write to disk, a database, pickle, or some external mechanism that allows data to be exchanged between the processes (and the existence of which can be verified) if you want to exchange an object that is the result of a task.

As opposed to writing the output() method, which requires a target, you can also override the complete() method where you can write any custom logic that allows the tasks to be considered complete.

Listel

Ok, that sounds fair, can you kindly point me to some examples? I started to suspect that I could override the complete method, but I wasn't able to find anything relevant on the topic. Thanks! – Arthur

yes, returning true when things are done is all you have to do to override complete(). the base behavior of complete() is to check for existence of the Target returned by output() – Listel