How do I get 'real-time' information back from a subprocess.Popen in python (2.5)

Question

I'd like to use the subprocess module in the following way:

1. create a new process that potentially takes a long time to execute.
2. capture stdout (or stderr, or potentially both, either together or separately)
3. Process data from the subprocess as it comes in, perhaps firing events on every line received (in wxPython say) or simply printing them out for now.

I've created processes with Popen, but if I use communicate() the data comes at me all at once, once the process has terminated.

If I create a separate thread that does a blocking readline() of myprocess.stdout (using stdout = subprocess.PIPE) I don't get any lines with this method either, until the process terminates. (no matter what I set as bufsize)

Is there a way to deal with this that isn't horrendous, and works well on multiple platforms?

Savdeep

myprocess.stdout.readline() should work. Can you show us your code? – Leticia

Unbuffered reads from popen_obj.stdout() should indeed work -- but if you don't mind being limited to platforms with PTY support, your application might be suitable for the Pexpect library. – Yasukoyataghan

This is a great question and it still seems to be unanswered, at least for the "works well on multiple platforms" requirement. – Saundrasaunter

related: Getting realtime output using subprocess – Galvani

here's how to capture and display in "real-time" time both stdout and stderr separately from a child process line by line in a portable manner in a single thread in Python 3 (asyncio) and here's multithreaded solution (teed_call()). – Galvani

here's how to read subprocess' output in a GUI (threads/no threads solutions) – Galvani

Answer 1

Update with code that appears not to work (on windows anyway)

class ThreadWorker(threading.Thread):
    def __init__(self, callable, *args, **kwargs):
        super(ThreadWorker, self).__init__()
        self.callable = callable
        self.args = args
        self.kwargs = kwargs
        self.setDaemon(True)

    def run(self):
        try:
            self.callable(*self.args, **self.kwargs)
        except wx.PyDeadObjectError:
            pass
        except Exception, e:
            print e



if __name__ == "__main__":
    import os
    from subprocess import Popen, PIPE

    def worker(pipe):
        while True:
            line = pipe.readline()
            if line == '': break
            else: print line

    proc = Popen("python subprocess_test.py", shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)

    stdout_worker = ThreadWorker(worker, proc.stdout)
    stderr_worker = ThreadWorker(worker, proc.stderr)
    stdout_worker.start()
    stderr_worker.start()
    while True: pass

Savdeep

This is clearly the best answer. Thanks for showing me this 'pipe' type in Python! – Bregma

This is a great answer, and works for me. The key is that the reads can block without any problem due to the threads. – Adolfoadolph

while True: time.sleep(1) is better than a busy wait loop consuming all your CPU. – Leavy

How does it fail on Windows? – Galvani

call proc.stdin.close() if you use stdin=PIPE. Use for t in [stdout_worker, stderr_workerr]: t.join() instead of while True:pass. You could use iter(pipe.readline, b'') instead of the while loop in worker(). – Galvani

Answer 2

stdout will be buffered - so you won't get anything till that buffer is filled, or the subprocess exits.

You can try flushing stdout from the sub-process, or using stderr, or changing stdout on non-buffered mode.

Analogical

Shouldn't it be unbuffered by default? At least with bufsize=0 ? – Drooff

@Albert: the buffer is inside subprocess e.g., stdio buffer. Nothing outside the child process sees that data until it flushes its stdout buffer. Here're some workarounds for the buffering issue – Galvani

Answer 3

It sounds like the issue might be the use of buffered output by the subprocess - if a relatively small amount of output is created, it could be buffered until the subprocess exits. Some background can be found here:

Itself

Answer 4

Here's what worked for me:

cmd = ["./tester_script.bash"]
p = subprocess.Popen( cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
while p.poll() is None:
    out = p.stdout.readline()
    do_something_with( out, err )

In your case you could try to pass a reference to the sub-process to your Worker Thread, and do the polling inside the thread. I don't know how it will behave when two threads poll (and interact with) the same subprocess, but it may work.

Also note thate the while p.poll() is None: is intended as is. Do not replace it with while not p.poll() as in python 0 (the returncode for successful termination) is also considered False.

Gahan

I'm not sure if there's something I'm missing here but it seems like your code is actually blocking: that "while" loop means that whatever the output of that function should be, it won't be returned until p.poll() is None. – Mirk

Yes. This is blocking. I guess that back when I answered this, I did not notice the threading part of the question. Even so, my answer here is not quite "up-to-date". The main problem is that output is buffered (as mentioned elsewhere), and I don't mention this here. I will leave it here for posterity, but other submitted answers are better. – Gahan

Answer 5

I've been running into this problem as well. The problem occurs because you are trying to read stderr as well. If there are no errors, then trying to read from stderr would block.

On Windows, there is no easy way to poll() file descriptors (only Winsock sockets).

So a solution is not to try and read from stderr.

Cordwood

Answer 6

Using pexpect [http://www.noah.org/wiki/Pexpect] with non-blocking readlines will resolve this problem. It stems from the fact that pipes are buffered, and so your app's output is getting buffered by the pipe, therefore you can't get to that output until the buffer fills or the process dies.

Beyer

Answer 7

This seems to be a well-known Python limitation, see PEP 3145 and maybe others.

Atonement

Answer 8

Read one character at a time: http://blog.thelinuxkid.com/2013/06/get-python-subprocess-output-without.html

import contextlib
import subprocess

# Unix, Windows and old Macintosh end-of-line
newlines = ['\n', '\r\n', '\r']
def unbuffered(proc, stream='stdout'):
    stream = getattr(proc, stream)
    with contextlib.closing(stream):
        while True:
            out = []
            last = stream.read(1)
            # Don't loop forever
            if last == '' and proc.poll() is not None:
                break
            while last not in newlines:
                # Don't loop forever
                if last == '' and proc.poll() is not None:
                    break
                out.append(last)
                last = stream.read(1)
            out = ''.join(out)
            yield out

def example():
    cmd = ['ls', '-l', '/']
    proc = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        # Make all end-of-lines '\n'
        universal_newlines=True,
    )
    for line in unbuffered(proc):
        print line

example()

Plastered

it seems to be a duplicate answer. See my previous comment – Galvani

Answer 9

Using subprocess.Popen, I can run the .exe of one of my C# projects and redirect the output to my Python file. I am able now to print() all the information being output to the C# console (using Console.WriteLine()) to the Python console.

Python code:

from subprocess import Popen, PIPE, STDOUT

p = Popen('ConsoleDataImporter.exe', stdout = PIPE, stderr = STDOUT, shell = True)

while True:
    line = p.stdout.readline()
    print(line)
    if not line:
        break

This gets the console output of my .NET project line by line as it is created and breaks out of the enclosing while loop upon the project's termination. I'd imagine this would work for two python files as well.

Butyrin

Answer 10

I've used the pexpect module for this, it seems to work ok. http://sourceforge.net/projects/pexpect/

Proton