Logging nested functions using joblib Parallel and delayed calls

Question

In one of my scripts I have something like:

import logging
from joblib import Parallel, delayed

def f_A(x):
    logging.info("f_A "+str(x))

def f_B():
    logging.info("f_B")
    res = Parallel(n_jobs=2, prefer="processes")(delayed(f_A)(x) for x in range(10))


if __name__ == "__main__":
   logging.basicConfig(level=logging.INFO)
   f_B()

I would expect that when I run python script.py something like:

INFO:root:f_B
INFO:root:f_A

to be shown in the console, instead I see:

INFO:root:f_B

but no information from f_A is shown.

How can I get f_A --and eventually functions called from there-- to show in the logs?

I think the issue is due to default logging level that is DEBUG and the main process doesn't share propagate the level to the children. If you modify slightly the script to:

import logging
from joblib import Parallel, delayed

def f_A(x):
    logging.basicConfig(level=logging.INFO)
    logging.info("f_A "+str(x))

def f_B():
    logging.info("f_B")
    res = Parallel(n_jobs=2, prefer="processes")(delayed(f_A)(x) for x in range(10))


if __name__ == "__main__":
   logging.basicConfig(level=logging.INFO)
   f_B()

then everything works as intended.

Namtar

Very relevant question. It is indeed telling that joblib only uses print in their examples and not logging. Their issue tracker has this issue, which contains two work-arounds: github.com/joblib/joblib/issues/1017 . – Resourceful

Answer 1

You are correct the issue is due to the logging level not being propagated to the child processes. In your original script, the basicConfig is only called in the main process. When using the joblib.Parallel with the prefer="processes" option, the child processes do not inherit the logging configuration of the main process.

Floridafloridia