Python subprocess get children's output to file and terminal?

Question

I'm running a script that executes a number of executables by using

subprocess.call(cmdArgs,stdout=outf, stderr=errf)

when outf/errf is either None or a file descriptor (different files for stdout/stderr).

Is there any way I can execute each exe so that the stdout and stderr will be written to the files and terminal together?

Wilcher

asyncio version – Galiot

Possible duplicate of Python Popen: Write to stdout AND log file simultaneously – Indies

Answer 1

The call() function is just Popen(*args, **kwargs).wait(). You could call Popen directly and use stdout=PIPE argument to read from p.stdout:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE
from threading import Thread


def tee(infile, *files):
    """Print `infile` to `files` in a separate thread."""

    def fanout(infile, *files):
        with infile:
            for line in iter(infile.readline, b""):
                for f in files:
                    f.write(line)

    t = Thread(target=fanout, args=(infile,) + files)
    t.daemon = True
    t.start()
    return t


def teed_call(cmd_args, **kwargs):
    stdout, stderr = [kwargs.pop(s, None) for s in ["stdout", "stderr"]]
    p = Popen(
        cmd_args,
        stdout=PIPE if stdout is not None else None,
        stderr=PIPE if stderr is not None else None,
        **kwargs
    )
    threads = []
    if stdout is not None:
        threads.append(
            tee(p.stdout, stdout, getattr(sys.stdout, "buffer", sys.stdout))
        )
    if stderr is not None:
        threads.append(
            tee(p.stderr, stderr, getattr(sys.stderr, "buffer", sys.stderr))
        )
    for t in threads:
        t.join()  # wait for IO completion
    return p.wait()


outf, errf = open("out.txt", "wb"), open("err.txt", "wb")
assert not teed_call(["cat", __file__], stdout=None, stderr=errf)
assert not teed_call(["echo", "abc"], stdout=outf, stderr=errf, bufsize=0)
assert teed_call(["gcc", "a b"], close_fds=True, stdout=outf, stderr=errf)

Galiot

thanks for the quick response, but it doesn't work. the external process only sees OS-level file handles (the number you get from the fileno() method on your file objects). see bytes.com/topic/python/answers/541085-extend-file-type – Wilcher

thanks, what would you do if instead of subprocess.Call I'de like to run multiple execs using subprocess.Popen (and not Call), where each exec writes to a different file and to the terminal – Wilcher

@user515766: the solution is the same: set stdout, stderr to PIPE and call tee() when you'd like to write to more than one place. – Galiot

somebody has deleted comments that demonstrate that the first comment ("doesn't work") is wrong. It confusessubprocess.call and the function call (different) that is called teed_call now to avoid the ambiguity. – Galiot

@Peilonrayz: I've made the code in the answer to be Python 2/3 compatible. (it was pure Python 2 solution in 2011) – Galiot

I had to add this to get the example to work: if isinstance(f, io.TextIOBase): f.write(str(line)) else: f.write(line) – Neologize

@channon: str is likely a wrong thing to do. Notice, the code in the answer open files in binary mode. If you want support text files, specify an appropriate for your case character encoding such as utf-8, cp437, etc (or pass text=True if you want the subprocess module to chose it for you)$ Also, sys.stdout.buffer instead of sys.stdout should be used in binary mode on Python 3 (perhaps, this part causes the error) – Galiot

Yes you are correct, the better way is to use thesys.stdout.buffer. Now the test cases provided pass as expected. – Neologize

Answer 2

You could use something like this: https://github.com/waszil/subpiper

In your callbacks you can do whatever you like, log, write to file, print, etc. It also supports non-blocking mode.

from subpiper import subpiper

def my_stdout_callback(line: str):
    print(f'STDOUT: {line}')

def my_stderr_callback(line: str):
    print(f'STDERR: {line}')

my_additional_path_list = [r'c:\important_location']

retcode = subpiper(cmd='echo magic',
                   stdout_callback=my_stdout_callback,
                   stderr_callback=my_stderr_callback,
                   add_path_list=my_additional_path_list)

Beading

I tried this with cmd='python3 run2.py' but get an error. Am I missing something how to send run2.py as a parameter: FileNotFoundError: [Errno 2] No such file or directory: 'python3 run2.py': 'python3 run2.py' – Piranha