Processing messages from a child process thorough stderr and stdout with Python

Question

My python code spawns the child process, and it prints out messages both stdout and stderr. I need to print them differently.

I have the following code to spawn child process and get the stdout result from it.

cmd = ["vsmake.exe", "-f"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
    print line,
    sys.stdout.flush()
    pass
p.wait()

How can I modify the code to check if the child process prints out message through stderr also?

ADDED

I need to print out the stderr and stdout as soon as the child process prints out something. And it is cross platform implementation, so it should run on Mac/Linux/PC.

Applause

Would you like to print both the messages on stdout and stderr in realtime? If yes, you will probably need threads on Windows (because there is neither select() nor non-blocking I/O). – Abiotic

I edited my answer in order print both stderr and stdout as soon a line is printed in any of those two pipes. – Their

Answer 1

p = Popen(cmd, bufsize=1024,
stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
p.stdin.close()
print p.stdout.read() #This will print the standard output from the spawned process
print p.stderr.read() #This is what you need, error output <-----

So basically the error output gets redirected to the stderr Pipe.

If you need something more in real in time. I mean lines printed as soon as the spawned process prints something to stdout orstderr` then you can do something like:

def print_pipe(type_pipe,pipe):
    for line in iter(pipe.readline, ''):
         print "[%s] %s"%(type_pipe,line),

p = Popen(cmd, bufsize=1024,
stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)

t1 = Thread(target=print_pipe, args=("stdout",p.stdout,))
t1.start()
t2 = Thread(target=print_pipe, args=("stderr",p.stderr,))
t2.start()

#optionally you can join the threads to wait till p is done. This is avoidable but it 
# really depends on the application.
t1.join()
t2.join()

In this case two threads will print every time that a line is written either to stdout or stderr. The parameter type_pipe just makes the distinction when the lines are printed to know if they are coming from stderr or stdout.

Their

My favorite (for small outputs) is: stdin,stdout = p.communicate(). Gets it all done in one shot! Then do with stdin,stdout what you will. – Determinative

nice one !! (+1) I did not know that one. – Their

Given the example code in the question, I suspect the OP wants to print the output in real time. But we won't know this for sure until (s)he tells us. – Abiotic

@Sven : I added more to my question. – Applause

You should fix your print statement -- the % operator will fail the way it is now. Also remember that line ends with "\n", and you don't want to print this twice. – Abiotic

Cheers @Sven, fixed ... errors of fast typing ;) +1 for the suggestions. – Their

1. The first example can block if the output is sufficiently large. #164042 2. Don't use shell=True unless you must. The semantic with a list cmd is different. 3. You could just add , at the end of print statement instead of line.replace(). 3. there should be p.wait() and t1.join(), t2.join() somewhere. – Thermionic

@J.F Sebastian the first example is just to introduce that stderr exists. About the other suggestions ... thanks for the , thing in the print I did not know I could do that. Very nice. About p.wait() or the joins ... it is discussable if you really need them. It depends on the application. You might not want to wait till p is finished. Anyway I have added a comment on it in the code. Thanks a lot for your suggestions. Cheers. – Their

Answer 2

The easiest way to do this platform-independently is using threads (unfortunately). Here is some example code:

def redirect_to_stdout(stream):
    for line in stream:
        sys.stdout.write(line)
        sys.stdout.flush()

cmd = ["vsmake.exe", "-f"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stderr_thread = threading.Thread(target=redirect_to_stdout, args=(p.stderr,))
stderr_thread.start()
redirect_to_stdout(p.stdout)
p.wait()
stderr_thread.join()

Abiotic

-1: s/run/start/. Use stderr=subprocess.STDOUT to redirect stdout. – Thermionic

@J.F. Sebastian: Thanks for pointing out the start()/run() confusion, corrected now. With stderr=subprocess.STDOUT it is impossible to meet the OP's requirement to "print [stdout and stderr] differently". If you could go that way, you would not need threads at all. – Abiotic

Exactly, your solution does not print stdout, stderr differently; it doesn't need threads in this case. – Thermionic