How to use Parameters in Python Luigi

Question

How do I pass in parameters to Luigi? if I have a python file called FileFinder.py with a class named getFIles:

class getFiles(luigi.Task):

and I want to pass in a directory to this class such as:

C://Documents//fileName

and then use this parameter in my run method

def run(self):

how do I run this in command line and add the parameter for use in my code? I am accustomed to running this file in command line like this:

python FileFinder.py getFiles --local-scheduler

What do I add to my code to use a parameter, and how do I add that parameter to the command line argument?

Also, as an extension of this question, how would I use multiple arguments? or arguments of different data types such as strings or lists?

Importunity

Answer 1

As you have already figured out, you can pass arguments to luigi via

--param-name param-value

in the command line. Inside your code, you have to declare these variables by instantiating the Parameter class or one of it's subclasses. The subclasses are used to tell luigi if the variable has a data-type that is not string. Here is an example which uses two command line arguments, one Int and one List:

import luigi

class testClass(luigi.Task):
  int_var = luigi.IntParameter()
  list_var = luigi.ListParameter()

  def run(self):
      print('Integer Param + 1 = %i' % (self.int_var + 1))

      list_var = list(self.list_var)
      list_var.append('new_elem')
      print('List Param with added element: ' + str(list_var))

Note that ListParams actually get converted to tuples by luigi, so if you want to do list operations on them, you have to convert them back first (This is a known issue, but doesn't look like it will be fixed soon).

You can invoke the above module from the command line like this (i have saved the code as a file called "testmodule.py" and made the call from inside the same directory):

luigi --module testmodule testClass --int-var 3 --list-var '[1,2,3]'  --local-scheduler

Note here that for variables containing a _, this has to be replaced by -. The call yields (along with many status messages):

Integer Param + 1 = 4
List Param with added element: [1, 2, 3, 'new_elem']

Subotica

Can you describe what making a module out of the code means? I don't understand what my_module is or how to run the code that way along with other non-list parameters, because I would like to be able to use both single string parameters and list parameters – Importunity

The module stuff was not nessessary to answer your question, so i removed it from my answer. The updated answer should cover the things you need. – Subotica

Is there are way to run this without the command line args? I'm calling luigi from inside python code and have a list object I would like to pass in. Seems I can do something like testClass(1,[1,2]).run() but not sure if that's best. – Beatrizbeattie

Nvm answered my own question. In case someone comes across it's luigi.build([MyTask([1,2])], local_scheduler=True) – Beatrizbeattie

Answer 2

So I think this works, in the code I added:

fileName = luigi.Parameter()

if i run this in the command line:

python FileFinder.py getFiles --local-scheduler --getFiles-fileName C://Documents//fileName

but if anyone has any advice on parameters of different types and how to use them, especially numbers and lists, please let me know.

Importunity

Answer 3

Adding to Toterich's answer.
While passing a list of string arguments as a ListParameter():

python file_name.py --local-scheduler TaskName --arg '["a","b"]'  

The string arguments must be enclosed in double-quotes and not single quotes otherwise it'll give a JSONParsing error.

Gattis