Luigi parameter default values and mocks

Question

I am trying to mock something that supplies a default value for a luigi parameter.

A dumb example showing what I'm trying to accomplish:

Task under test:

import luigi
from bar import Bar

bar = Bar()

class Baz(luigi.Task):

    qux = luigi.Parameter(default=bar.bar())

    def baz(self):
        return self.qux;

    def foo(self):
        return bar.bar()

Unit Test code:

import unittest
from mock import Mock, patch
from sut.baz import Baz

class TestMocking(unittest.TestCase):

    def test_baz_bar(self):
        self.assertEquals("bar", Baz().baz())

    @patch('sut.baz.bar')
    def test_patched_baz(self, mock_bar):
        mock_bar.bar = Mock(return_value="foo")
        self.assertEquals("foo", (Baz().baz()))

    @patch('sut.baz.bar')
    def test_patched_foo(self, mock_bar):
        mock_bar.bar = Mock(return_value="foo")
        self.assertEquals("foo", (Baz().foo()))

It appears that the luigi.Parameter logic happens earlier than the patch.

In this example, test_patched_foo passes and test_patched_baz fails. So the patch does happen, but happens after the call from the luigi.Parameter(default=bar.bar()) line.

Is it possible to mock and patch something called in this manner?

Latoria

In the actual code, I've gotten around this by putting in a lazy-load. Having trouble replicating the solution in my toy example, so I won't post it as a self-answer yet. But I also don't particularly like this workaround. – Latoria

Answer 1

Try moving the qux = luigi.Parameter(default=bar.bar()) line into the __init__ method for the Baz class. With it in outside the __init__, it is being set upon class definition, not instance creation, but putting it into the __init__ will delay its creation to the point where a Baz instance is created. Don't forget to call the __init__ on the super class:

class Baz(luigi.Task):

    def __init__(self, *args, **kwargs):
        super(Baz, self).__init__(*args, **kwargs)

        self.qux = luigi.Parameter(default=bar.bar())

    ...

Ravenravening

A promising thought, but not compatible with the way luigi parameters work. The line qux = luigi.Parameter(default=bar.bar()) creates a field that can later be referenced as self.qux but it's not a luigi.Parameter - it's the value passed or the result of the default spec, and in this case a string. It might be there's a way to bypass this and use a more standard kwargs.pop with a default, but I'm not sure luigi will like it. Might try it though. – Latoria