Wrap subprocess' stdout/stderr

Question

I'd like to both capture and display the output of a process that I invoke through Python's subprocess.

I thought I could just pass my file-like object as named parameter stdout and stderr

I can see that it accesses the filenoattribute - so it is doing something with the object. However, the write() method is never invoked. Is my approach completely off or am I just missing something?

class Process(object):
    class StreamWrapper(object):
        def __init__(self, stream):
            self._stream = stream
            self._buffer = []
        def _print(self, msg):
            print repr(self), msg
        def __getattr__(self, name):
            if not name in ['fileno']:
                self._print("# Redirecting: %s" % name)
            return getattr(self._stream, name)
        def write(self, data):
            print "###########"
            self._buffer.append(data)
            self._stream.write(data)
            self._stream.flush()
        def getBuffer(self):
            return self._buffer[:]
    def __init__(self, *args, **kwargs):
        print ">> Running `%s`" % " ".join(args[0])
        self._stdout = self.StreamWrapper(sys.stdout)
        self._stderr = self.StreamWrapper(sys.stderr)
        kwargs.setdefault('stdout', self._stdout)
        kwargs.setdefault('stderr', self._stderr)
        self._process = subprocess.Popen(*args, **kwargs)
        self._process.communicate()

Update:

Something I'd like to work as well, is the ANSI control characters to move the cursor and override previously output stuff. I don't know whether that is the correct term, but here's an example of what I meant: I'm trying to automate some GIT stuff and there they have the progress that updates itself without writing to a new line each time.

Update 2

It is important to me, that the output of the subprocess is displayed immediately. I've tried using subprocess.PIPE to capture the output, and display it manually, but I was only able to get it to display the output, once the process had completed. However, I'd like to see the output in real-time.

Charette

I'd prefer it to be a cross-platform compatible solution. – Charette

related: Python subprocess get children's output to file and terminal? – Bik

related: Subprocess.Popen: cloning stdout and stderr both to terminal and variables – Bik

Here is a very nice explanation on how to do it The ever useful and neat subprocess module – Holbrooke

Answer 1

Stdin, stdout and stderr of a process need to be real file descriptors. (That is actually not a restriction imposed by Python, but rather how pipes work on the OS level.) So you will need a different solution.

If you want to track both stdout an stderr in real time, you will need asynchronous I/O or threads.

• Asynchronous I/O: With the standard synchronous (=blocking) I/O, a read to one of the streams could block, disallowing access to the other one in real time. If you are on Unix, you can use non-blocking I/O as described in this answer. However, on Windows you will be out of luck with this approach. More on asynchronous I/O in Python and some alternatives are shown in this video.

• Threads: Another common way to deal with this problem is to create one thread for each file descriptor you want to read from in real time. The threads only handle the file descriptor they are assinged to, so blocking I/O won't harm.

Gond

Thanks a lot for the link, I will look into this. Maybe this question now will be clarified once I watch the video, but asking doesn't hurt I guess: (not sure whether I understood you correctly yet) It doesn't matter if it blocks for a short amount of time. To return to the GIT example, it could take minutes for a clone, it would be enough to update the information every few seconds, it doesn't have to be in "literal real time" - not sure whether that makes a difference for this concept :) – Charette

@phant0m: The problem with blocking I/O is that you don't know how long the calls will block. If you call subproc.stderr.readline(), it will block until the process writes a line to its stderr. If it never does, you will never get the chance to capture the process' stdout until the process finishes. If you only want to capture the stdout, that's fine. As soon as there are two file descriptors to read from, blocking I/O won't help you any more. – Gond

Ah i see. So I could also just readline() from the subprocess and have any stderr output cause an exception? Then I could read all that's been printed to stderr afterwards right? – Charette

@phant0m: I don't understand your last question. If you put, say, stdout into non-blocking mode as described in the linked answer, and then call subproc.stdout.readline(), there are two cases: 1. A line of data is available: It will be read from the process and returned. 2. No full line is available: Since readline() is not allowed to block any more, it will throw an IOError, which you can catch. – Gond

Answer 2

Have a look here.

p = subprocess.Popen(cmd,
                 shell=True,
                 bufsize=64,
                 stdin=subprocess.PIPE,
                 stderr=subprocess.PIPE,
                 stdout=subprocess.PIPE)

Quindecennial

I would like to display the output in real time, when I used PIPE in my tests, I was unable to do that. I could only output everything at the end of the process. – Charette

Answer 3

A file-like is not close enough. It must be an actual file with an actual file descriptor. Use subprocess's support for pipes and read from them as appropriate.

Least

The thing is, I don't know how to do both at the same time. – Charette

Answer 4

Here is a function from Chromium CI product, Catapult (BSD-3-Clause license).

It's non-blocking, and drains the output. That is, handles the problems that are present in most answers on SO.

I would also look into os.set_blocking() instead of fcntl here.

os.set_blocking() is introduced in Python 3.5, supports Windows pipes since Python 3.12. So since 3.12 it would make the code portable to Windows too.

def _IterProcessStdoutFcntl(process,
                            iter_timeout=None,
                            timeout=None,
                            buffer_size=4096,
                            poll_interval=1):
  """An fcntl-based implementation of _IterProcessStdout."""
  # pylint: disable=too-many-nested-blocks
  import fcntl
  try:
    # Enable non-blocking reads from the child's stdout.
    child_fd = process.stdout.fileno()
    fl = fcntl.fcntl(child_fd, fcntl.F_GETFL)
    fcntl.fcntl(child_fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

    end_time = (time.time() + timeout) if timeout else None
    iter_end_time = (time.time() + iter_timeout) if iter_timeout else None

    while True:
      if end_time and time.time() > end_time:
        raise TimeoutError()
      if iter_end_time and time.time() > iter_end_time:
        yield None
        iter_end_time = time.time() + iter_timeout

      if iter_end_time:
        iter_aware_poll_interval = min(poll_interval,
                                       max(0, iter_end_time - time.time()))
      else:
        iter_aware_poll_interval = poll_interval

      read_fds, _, _ = select.select([child_fd], [], [],
                                     iter_aware_poll_interval)
      if child_fd in read_fds:
        data = _read_and_decode(child_fd, buffer_size)
        if not data:
          break
        yield data

      if process.poll() is not None:
        # If process is closed, keep checking for output data (because of timing
        # issues).
        while True:
          read_fds, _, _ = select.select([child_fd], [], [],
                                         iter_aware_poll_interval)
          if child_fd in read_fds:
            data = _read_and_decode(child_fd, buffer_size)
            if data:
              yield data
              continue
          break
        break
  finally:
    try:
      if process.returncode is None:
        # Make sure the process doesn't stick around if we fail with an
        # exception.
        process.kill()
    except OSError:
      pass
    process.wait()

Coercion

Answer 5

Is there a reason you have a class inside a class? And stdout and stderr can take any file like line for example try. So simply passing an open file type or stringIO should be enough to modify the stream

import sys
sys.stdout = open('test.txt','w')
print "Testing!"
sys.stdout.write('\nhehehe')
sys.stdout = sys.__stdout__
sys.exit(0)

Nonagon

The class is inside it, because I thought it would be nicer to have it in the outer class' namespace - to make clear what it's use is for. Anyway, I'm not trying to output to a file. I'd like to print to stdout and capture the data for further processing. – Charette