subprocess child traceback

Question

I want to access the traceback of a python programm running in a subprocess.

The documentation says:

Exceptions raised in the child process, before the new program has started to execute, will be re-raised in the parent. Additionally, the exception object will have one extra attribute called child_traceback, which is a string containing traceback information from the child’s point of view.

Contents of my_sub_program.py:

raise Exception("I am raised!")

Contents of my_main_program.py:

import sys
import subprocess
try:
    subprocess.check_output([sys.executable, "my_sub_program.py"])
except Exception as e:
    print e.child_traceback

If I run my_main_program.py, I get the following error:

Traceback (most recent call last):
  File "my_main_program.py", line 6, in <module>
    print e.child_traceback
AttributeError: 'CalledProcessError' object has no attribute 'child_traceback'

How can I access the traceback of the subprocess without modifying the subprocess program code? This means, I want to avoid adding a large try/except clause around my whole sub-program code, but rather handle error logging from my main program.

Edit: sys.executable should be replaceable with an interpreter differing from the one running the main program.

Persephone

Doc says "before the new program has started to execute", in your case the exception was raised while the new program was executed, hence no child_traceback. Once the new program is running you need to catch the CalledProcessError exception and do as the following: #24850498 using CalledProcessError.output – Salsify

In my example, CalledProcessError.output did only capture the standard output, but not the traceback of the Exception. – Persephone

This is probably because output is sent on stderr. Look at answers from the question I sent you the link above for more details – Salsify

Answer 1

As you're starting another Python process, you can also try to use the multiprocessing Python module ; by sub-classing the Process class it is quite easy to get exceptions from the target function:

from multiprocessing import Process, Pipe
import traceback
import functools

class MyProcess(Process):
    def __init__(self, *args, **kwargs):
        Process.__init__(self, *args, **kwargs)
        self._pconn, self._cconn = Pipe()
        self._exception = None

    def run(self):
        try:
            Process.run(self)
            self._cconn.send(None)
        except Exception as e:
            tb = traceback.format_exc()
            self._cconn.send((e, tb))
            # raise e  # You can still rise this exception if you need to

    @property
    def exception(self):
        if self._pconn.poll():
            self._exception = self._pconn.recv()
        return self._exception


p = MyProcess(target=functools.partial(execfile, "my_sub_program.py"))
p.start()
p.join() #wait for sub-process to end

if p.exception:
    error, traceback = p.exception
    print 'you got', traceback

The trick is to have the target function executing the Python sub-program, this is done by using functools.partial.

Salsify

Does this solution allow the subprocess to run in a different interpreter than the main program? As I see your suggestion I notice that it was misleading to use sys.executable in my example. However, in the context of my question, I need to start scripts in another virtual environment than the main process. (the main process will be some kind of scheduler) – Persephone

Actually multiprocessing does os.fork on POSIX platforms, so it's not a new virtual environment ; however you can limit the amount of shared data between processes by starting the Process objects early, this is maybe fair enough for your needs – Salsify

If you're doing a kind of scheduler why not using something like Celery – Salsify