Output of subprocess both to PIPE and directly to stdout

Question

I found a number of questions which looks like mine, but which did not produce a solution I can use (closest is: subprocess output to stdout and to PIPE)

The problem: I want to start a process using subprocess which takes a long time. After running the command I need to parse the stdout-output and the stderr-output.

Currently I do it as follows:

p = subprocess.Popen( command_list, stdout=subprocess.PIPE, 
    stderr=subprocess.PIPE )
out, error_msg = p.communicate()
print out + "\n\n" + error_msg

#next comes code in which I check out and error_msg

But the drawback of this method is that the user does not see the output of the process while it is running. Only at the end the output is printed.

Is there a way that the output is printed while the command is running (as if I gave the command without stdout/stderr=subprocess.PIPE) and still have the output via p.communicate in the end?

Note: I'm currently developing on python 2.5 (old software release which uses this python version).

Supposing

related: Subprocess.Popen: cloning stdout and stderr both to terminal and variables – Handcar

related: Displaying subprocess output to stdout and redirecting it – Handcar

related: Python subprocess get children's output to file and terminal? – Handcar

Answer 1

This snippet has helped me once in a similar situation:

process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
    print line,
    sys.stdout.flush() # please see comments regarding the necessity of this line 
process.wait()
errcode = process.returncode

Pouter

The flush() may be useful if you really require instantaneous output, but it's not strictly necessary if all you care about is to see some progress from a process which generates enough output for buffering to not matter a lot. If it's a very slow writer, of course, it will take a long time before the buffer fills up; and then maybe you do want to flush() often. – Cryptanalysis

@tripleee: I find that sys.stdout.flush() is unnecessary here if we assume that python process and the child process have the same buffering policy (likely if they both stdio-based) i.e., if python script is run in the terminal (no redirection) then the stdout is already line-buffered -- no sys.stdout.flush() necessary. If the output is redirected to a file (despite what OP asks) then flush() unnecessarily slows down the printing (block-buffering that is typically used when the output is file is more efficient, the flushing after each line makes it slower). – Handcar

I would call flush() explicitly iff --line-buffering option is given (like grep). – Handcar

Note: stderr=STDOUT merges stdout/stderr here. OP might want to capture them separately. – Handcar

@J.F.Sebastian and tripleee thanks for the extra info I've edited the answer to draw attention to those comments – Pouter

Kudos. That almost solves my problem. As J.F. Sebastian is already stating I need to capture stdout and stderr seperately (and print them at the same time). But this helps – Supposing

@Nemelis: unfortunately, capturing stdout/stderr separately complicates the code a lot: you need threads or select() or other asynchronous I/O tools. See the questions that I've linked above. – Handcar

@J.F.Sebastian: Thanx. Will check them out. Already found out that capturing them separately was not easy. – Supposing

Answer 2

Here's a python function that watches the command stdout + stderr, and returns the output if finished successfully, else raises an exception:

import subprocess
def execute_command(cmd):
    """
    Execute and watch shell command, and return its output
    """
    output = ''
    with subprocess.Popen(
            cmd,
            shell=True,
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            universal_newlines=True,
            encoding='utf-8',
            errors='replace'
        ) as p:
            while True:
                line = p.stdout.readline()
                if line != '':
                    output += (line + "\n")
                    print(line)
                elif p.poll() != None:
                    break
            sys.stdout.flush()
            if (p.returncode == 0):
                return output
            else:
                raise Exception(cmd, p.returncode, output)

Atman