Unbuffered read from process using subprocess in Python

Question

I am trying to read from a process that produces long and time-consuming output. However, I want to catch it's output as and when it is produced. But using something like the following seems to be buffering the command's output, so I end up getting the output lines all at once:

p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, bufsize=0)
    for line in p.stdout:
        print line

I am trying this on MacOS 10.5

Arsis

Duplicate: #875315, #527697 – Otranto

Answer 1

The file iterator is doing some internal buffering on its own. Try this:

line = p.stdout.readline()
while line:
    print line
    line = p.stdout.readline()

You also need to make sure the process you are running is actually flushing its output buffers frequently.

Stellite

You also need to make sure the process you are running is actually flushing its output buffers frequently. I debugged more than half day to find out that stdout has quite a big buffer when directed into a pipe. So I had to add some flushes into the program's code to make the output actually appear. – Thrombokinase

Answer 2

Usually, every program will do more buffering on its input and/or output channels than you appear to desire... unless it's fooled into believing said channel's actually a terminal!

For that "fooling in a good cause" purpose, use pexpect -- it works just fine on a Mac (life is harder on Windows, though there are solutions that might help even there - fortunately we don't need to dwell on those as you use a Mac instead).

Superimpose

Answer 3

This was actually a bug that's fixed in Python 2.6: http://bugs.python.org/issue3907

Berceuse

On python 2.7 for line in p.stdout still delays its output while for line in iter(p.stdout.readline, b'') works as expected (it produces lines as soon as they are available). – Enclasp

Sorry, 2.x won't use the new io library by default, as the last comment on the bug report says; You'd have to do something like for line in io.open(p.stdin.fileno()). – Berceuse

@Ari: Should that be for line in io.open(p.stdout.fileno())? – Domenech