Last unbuffered line can't be read

Question

I'm trying to read the last line from a command like 'apt-get download firefox'. Normally the output will be like

Get:1 http://archive.ubuntu.com/ubuntu/ utopic/main firefox amd64 32.0+build1-0ubuntu2 [34.9 MB]
2% [1 firefox 646 kB/34.9 MB 2%]

with the last line continuously updating (it is not writing a newline until it reaches 100%). My goal is now to read the progress in realtime. Here is my current example code:

#!/usr/bin/python3 -u
# coding=utf-8

import subprocess, sys

pipe = subprocess.Popen(['apt-get', 'download', 'firefox'], 0, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
while True:
    content = pipe.stdout.read(1).decode()
    if content == '':
        break
    sys.stdout.write(content)
    sys.stdout.flush()
pipe.wait()

I have disabled the output buffering for the subprocess call and also for binary output for the Python process (with the -u argument). But I'm only getting the first line but not the progress of the second line. Does somebody know how I can achieve this?

Sabadell

you need either threads or async.io to read both streams concurrently . See Displaying subprocess output to stdout and redirecting it. – Blither

Do you need to capture Get:1 http://.. line (stdout?) or is it enough if you get only the last (progress) line (stderr?)? – Blither

Things to check: 1. does apt-get writes only to stdout/stderr or can it write to the terminal directly? 2. Does it use the block-buffering if the stdout is a pipe? 3. Does it show the progress line if stderr is redirected to a pipe? – Blither

Is it acceptable to merge stdout/stderr in you case (or just ignore the stderr)? – Blither

The output comes normally from stdout. Catching stderr is just for future development and removing it doesn't change the result. – Sabadell

Answer 1

If stdout of apt-get is redirected to a pipe e.g.,

$ apt-get download firefox | cat

then it doesn't report progress (the last line e.g., 2% [1 firefox 646 kB/34.9 MB 2%] will not be in the output). stdout=subprocess.PIPE naturally creates a pipe; therefore apt-get doesn't print the download progress in your case.

If you want both to capture apt-get output and to see it on the screen in real-time with the last line (progress report) present then you could use pexpect module to trick the child process into thinking that it runs in a terminal:

import sys
import pexpect # $ pip install pexpect

output, exitstatus = pexpect.runu('apt-get download firefox',
                                  logfile=sys.stdout,
                                  withexitstatus=1)

You could do the same using only stdlib pty module:

#!/usr/bin/env python3
import os
import pty

output = []
def read(fd):
    data = os.read(fd, 1024)
    output.append(data)
    return data
status = pty.spawn(['apt-get', 'download', 'firefox'], read)

@eryksun on Python Issue tracker suggested apt-get's --quiet option:

#!/usr/bin/env python3
import shlex
from io import TextIOWrapper
from subprocess import Popen, PIPE

output = []
with Popen(shlex.split("apt-get --quiet=0 download firefox"),
           stdout=PIPE, bufsize=1) as p:
    # recognize '\r' as newline but don't convert it to '\n'
    for line in TextIOWrapper(p.stdout, newline=''):
        print(line, end='', flush=True) # print to terminal
        output.append(line) # save for later
print(p.returncode)

Blither