'yes' reporting error with subprocess communicate()

Question

I am using the following function to run a command in Python:

def run_proc(cmd):
    child = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = child.communicate()
    returncode = child.returncode
    return stdout, stderr, returncode

It has always been working fine, however now I'm trying to use the yes program to pipe output to stdin. The command I'm trying to run is the following:

yes '' | apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade

but I believe it could be substituted with a general example, like:

yes | head -3 | cat

My problem is that if I try to run any command which has yes | in it, the above subprocess.Popen will contain the error messages:

yes: standard output: Broken pipe
yes: write error

For me it seems that the piping still works, as can be seen from yes | head -3 | cat's answer: y y y.

I have the following questions:

1. Is the yes piping still functional, even though yes reports error?
2. How can I fix it?

Colchester

here's some info about yes, SIGPIPE, and EPIPE – Justinajustine

Answer 1

The issue is that subprocess module before Python 3.2+ doesn't restore SIGPIPE signal handler to default action. That is why you get EPIPE write error instead.

In Python 3.2+

>>> from subprocess import check_output
>>> check_output("yes | head -3", shell=True)
b'y\ny\ny\n'

yes is killed by SIGPIPE when head exits.

In Python 2:

>>> from subprocess import check_output
>>> check_output("yes | head -3", shell=True)
yes: standard output: Broken pipe
yes: write error
'y\ny\ny\n'

yes got EPIPE write error. It is safe to ignore the error. It communicates the same information as SIGPIPE.

To workaround the problem, you could emulate restore_signals in Python 2 using preexec_fn parameter :

>>> from subprocess import check_output
>>> import signal
>>> def restore_signals(): # from http://hg.python.org/cpython/rev/768722b2ae0a/
...     signals = ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ')
...     for sig in signals:
...         if hasattr(signal, sig):
...            signal.signal(getattr(signal, sig), signal.SIG_DFL)
... 
>>> check_output("yes | head -3", shell=True, preexec_fn=restore_signals)
'y\ny\ny\n'

Justinajustine

Thanks for this, I appreciate the fact that this doesn't require changing your code if you are porting over from shell scripts. – Colchester

Answer 2

the other question answers the why ... I'll try and give you a work around

could you not do something like

proc = subprocess.Popen(cmd, shell=True, 
                             stdout=subprocess.PIPE, 
                             stderr=subprocess.PIPE,
                             stdin = subprocess.PIPE)

for i in range(10):  #send 10 y's
    time.sleep(1) # 1 second apart
    proc.stdin.write("y") #you may also need to send a newline ...

print proc.communicate()

see below (I didnt bother with the delay since head isnt really doing much)

>>> import subprocess
>>> proc = subprocess.Popen("head -3",
...                          shell = True,
...                          stdout = subprocess.PIPE,
...                          stderr=subprocess.PIPE,
...                          stdin=subprocess.PIPE)
>>> for i in range(10):
...    proc.stdin.write("y\n")
...
>>> proc.communicate()
('y\ny\ny\n', '')

Inessential

This is cool, but for commands more complicated then 'yes' it might require rewriting all the functionality in Python, which might or might not be the preferred solution. – Colchester

Answer 3

Saying:

yes | head -3

causes head to send a SIGPIPE to yes once it's done reading 3 lines of input, i.e. it'd send a signal to terminate yes.

$ yes | head -3
y
y
y
$ echo "${PIPESTATUS[@]}"
141 0

The solution would be to avoid a SIGPIPE!

Sagamore

It is the opposite: the issues is that SIGPIPE signal is trapped in Python 2 – Justinajustine