Reraise exception from subprocess

Question

I have code executing in a subprocess that is supposed to raise an exception.

I would like to raise the same exception in the main process when the exception is returned from the subprocess (preferably while preserving the stack trace) but I am not sure how to do this.

I capture the stderr from the subprocess just fine but I can't find how to parse it so I get the type of exception. How would I accomplish this?

I am using python 2.7

main method

import subprocess

example=subprocess.Popen(["python","example.py"],
                                    stdout = subprocess.PIPE,
                                    stderr = subprocess.PIPE)
return_obj, return_error = example.communicate()

if return_error:
# replace with error from subprocess
  print "Should raise ",NameError('HiThere')
  raise TypeError('Wrong type')

subprocess

raise NameError('HiThere')

Lowborn

I don't think information is available to you, without reparsing the exception and throwing a new one – Ricercar

You should consider importing instead of running another interpreter. Note you can still run any functions inside example.py in a separate process if you do so. – Breast

if return_error: raise Exception, return_error – Pruter

you probably want execfile() to run the example.py script in the same process in order to get the exception as an object easily. – Gilliam

Answer 1

If all you want is your python code running in a separate process, you should probably not use subprocess. As Serge Ballesta said, subprocess' purpose is to run a different program at the OS level, without particularly caring for what it is - there's nothing to help you properly handle the intricacies of a python interpreter process.

For this kind of purpose, it's probably best to simply import your code and use multiprocessing, which exposes a high-level interface to help you run python code in multiple processes.

Assuming you have a well defined main function on example.py:

from examply import main as example_main
import multiprocessing
pool= multiprocessing.Pool(1)
pool.apply( example_main )

Using this code, both exceptions and return values will be transparently given to your main process.

You can also use Pool.apply_async, if you don't want to block waiting for the result.

Breast

The situation I have is that my program accepts input on standard in and on a certain input it should crash. I am using subprocess to test this behaviour. Is it possible to use multiprocessing for this? – Lowborn

@Lowborn it's doable, but quirky - see the programming guidelines section on replacing stdin. It's probably much easier to simply rewrite example.py to accept a optional argument with a file-file object, and pass it from your tester program. – Breast

@Lowborn Or use multiprocessing's built-in communication mechanisms, such as a pipe – Breast

Answer 2

The subprocess executes a different native application. It could be java, C++, lisp or even Fortran or Cobol. So you have only 2 ways to get the exception from a Python subprocess :

• forget the subprocess at all, and directly call python code in the same python program, at simple try except will do the work
• define in interface contract between the main and subprocess, for example requiring that the error output must be empty if no fatal exeption occurs and else contains a pickle of the exception and the stacktrace .

Justajustemilieu

Answer 3

If both processes are python processes, and have to use pipes, exception object can be serialized through the pipe.

Caller:

parent_conn.send(event)
result = parent_conn.recv()
if isinstance(result, Exception):
    raise result
else:
    return result

Receiver:

def subprocess_loop(parent_conn, child_conn):
    
    parent_conn.close()

    while True:
        event = child_conn.recv()
        try:
            result = event_handler(event)
        except Exception as exc:
            child_conn.send(exc)
            continue
        child_conn.send(result)

Receiver process creation:

parent_conn, child_conn = Pipe()
process = Process(target=subprocess_loop, args=(parent_conn, child_conn))
process.start()
child_conn.close()

Tavia