IOError: [Errno 32] Broken pipe when piping: `prog.py | othercmd`
Asked Answered
E

10

120

I have a very simple Python 3 script:

f1 = open('a.txt', 'r')
print(f1.readlines())
f2 = open('b.txt', 'r')
print(f2.readlines())
f3 = open('c.txt', 'r')
print(f3.readlines())
f4 = open('d.txt', 'r')
print(f4.readlines())
f1.close()
f2.close()
f3.close()
f4.close()

But it always says:

IOError: [Errno 32] Broken pipe

I saw on the internet all the complicated ways to fix this, but I copied this code directly, so I think that there is something wrong with the code and not Python's SIGPIPE.

I am redirecting the output, so if the above script was named "open.py", then my command to run would be:

open.py | othercommand
Exploitation answered 8/1, 2013 at 3:18 Comment(5)
@squiguy line 2: print(f1.readlines())Vitality
You've got two IO operations occurring on line 2: a read from a.txt and a write to stdout. Perhaps try splitting those onto separate lines so you can see which operation triggers the exception. If stdout is a pipe and the read end has been closed, then that could account for the EPIPE error.Paynter
I can reproduce this error on output (given the right conditions), so I suspect the print call is the culprit. @JOHANNES_NYÅTT, can you clarify how you're launching your Python script? Are you redirecting standard output somewhere?Drive
This is a possible duplicate of the following question: #11423725Zacharia
related: #108683Kikuyu
M
51

I haven't reproduced the issue, but perhaps this method would solve it: (writing line by line to stdout rather than using print)

import sys
with open('a.txt', 'r') as f1:
    for line in f1:
        sys.stdout.write(line)

You could catch the broken pipe? This writes the file to stdout line by line until the pipe is closed.

import sys, errno
try:
    with open('a.txt', 'r') as f1:
        for line in f1:
            sys.stdout.write(line)
except IOError as e:
    if e.errno == errno.EPIPE:
        # Handle error

You also need to make sure that othercommand is reading from the pipe before it gets too big - https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer

Mainland answered 8/1, 2013 at 4:33 Comment(5)
While this is good programming practice, I don't think it has anything to do with the broken pipe error the questioner is getting (which probably has to do with the print call, not with reading the files).Drive
@Drive I added a few questions and alternative methods and was hoping for some feedback from the author. If the problem is sending a large quantity of data from an open file directly to the print statement then perhaps one of the alternatives above would fix it.Mainland
(The simplest solutions are often the best - unless there's a particular reason to load a whole file then print it, do it a different way)Mainland
Awesome work in troubleshooting this! While I could take this answer for granted, I could appreciate this only after seeing how the other answers (and my own approach) paled in comparison to your answer.Reply
You might want also a newline feed: sys.stdout.write(line + '\n')Starstarboard
M
128

The problem is due to SIGPIPE handling. You can solve this problem using the following code:

from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL) 

Update: As pointed out in the comments, python docs already have a good answer.

See here for background on this solution. Better answer here.

Mcguire answered 31/5, 2013 at 20:5 Comment(10)
This is very dangerous, as I just discovered, because if you ever get a SIGPIPE on a socket (httplib or whatever), your program will just exit without warning or error.Tegument
@DavidBennett, I am sure its application dependent and for your purposes the accepted answer is the right one. There is a much thorough Q&A here for people to go through and make an informed decision. IMO, for command line tools, its probably best to ignore the pipe signal in most cases.Mcguire
Perfect! I've made this the default for all python programs by adding these lines to /usr/lib/python2.7/site-packages/sitecustomize.py (or $PYTHONUSERBASE) BTW @DavidBennett I thinkhttplib should not rely on the default but install it's own signal handler.Tahiti
Any way to do this only temporarily?Matlick
@NateGlenn You can save the existing handler and restore it later.Mcguire
Could someone answer me why people are considering a blogspot article as a better source of truth than official documentation (hint: open the link to see how to fix the broken pipe error properly)? :)Carrack
@YuriiRabeshko Even better: #34718708Mcguire
@YuriiRabeshko I tried the official documentation solution to get pygmentize not to throw an error when you exit less before the end of the file, using lesspipe.sh/code2color, and it didn't work. Looking at pygments-doc.readthedocs.io/en/latest/_modules/pygments/…, I can see why: it catches all exceptions and prints an error there, so pygmentize itself cannot do anything. The "blog" solution works just fine.Sporangium
@YuriiRabeshko because the official documentation is bad. An answer below demonstrates how complex the actual solution is.Monolatry
Hi. Please update your answer to reflect that the docs you link contradict it. quote: Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.Counterspy
G
125

To bring information from the many helpful answers together, with some additional information:

  • Standard Unix signal SIGPIPE is sent to a process writing to a pipe when there's no process reading from the pipe (anymore).

    • This is not necessarily an error condition; some Unix utilities such as head by design stop reading prematurely from a pipe, once they've received enough data.
    • Therefore, an easy way to provoke this error is to pipe to head[1]; e.g.:
      • python -c 'for x in range(10000): print(x)' | head -n 1
  • By default - i.e., if the writing process does not explicitly trap SIGPIPE - the writing process is simply terminated, and its exit code is set to 141, which is calculated as 128 (to signal termination by signal in general) + 13 (SIGPIPE's specific signal number).

  • However, by design Python itself traps SIGPIPE and translates it into a Python BrokenPipeError (Python 3) / IOError (Python 2) instance with errno value errno.EPIPE.

    • Note: If you use a Unix emulation environment on Windows, the error may surface differently - see this answer.
  • If a Python script does not catch the exception, Python outputs error message BrokenPipeError: [Errno 32] Broken pipe (Python 3, possibly twice, with Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> sandwiched in between) / IOError: [Errno 32] Broken pipe (Python 2) and terminates the script with exit code 1[2] - this is the symptom Johannes (the OP) saw.

Windows considerations (SIGPIPE is a Unix-only signal)

  • If your script needs to run directly on Windows too, you may have to conditionally bypass code that references SIGPIPE, as shown in this answer.

  • If your script runs in a Unix subsystem on Windows, the SIGPIPE signal may surface differently than on Unix - see this answer.


There are two ways to solve this problem:

Generally, it is not advisable to silence this exception, as it may signal a severe error condition, depending on your script's purpose, such as the receiving end of a network socket unexpectedly closing.

  • However, if your script is a command-line utility, where quiet termination may not only be acceptable but preferred so as to play nicely with the standard head utility, for instance, you can abort quietly as follows, using signal.signal() to install the platform's default signal handler (which behaves as described above), as also shown in akhan's answer (works in both Python 3 and 2):
# ONLY SUITABLE FOR COMMAND-LINE UTILITIES

# Install the default signal handler.
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE, SIG_DFL)

# Start printing many lines.
# If this gets interrupted with SIGPIPE, 
# the script aborts quietly, and the process exit code is set to
# 141 (128 + SIGPIPE)
for x in range(10000): print(x)
  • Otherwise, if you want to handle the SIGPIPE-triggered exception yourself (works in both Python 3 and 2, adapted from the docs):
import sys, os, errno

try:

  # Start printing many lines.
  for x in range(10000): print(x)

  # IMPORTANT: Flush stdout here, to ensure that the 
  # SIGPIPE-triggered exception can be caught.
  sys.stdout.flush()

except IOError as e: 
  # Note: Python 3 has the more specific BrokenPipeError,
  #       but this way the code works in Python 2 too.
  if e.errno != errno.EPIPE: raise e # Unrelated error, re-throw.

  # Python flushes standard streams on exit; redirect remaining output
  # to devnull to avoid another BrokenPipeError at shutdown
  devnull = os.open(os.devnull, os.O_WRONLY)
  os.dup2(devnull, sys.stdout.fileno())

  # ... perform other handling.
  # Note: You can't write to stdout here.
  #       (print() and sys.stdout.write won't work)
  #       However, sys.stderr.write() can be used.
  sys.stderr.write("SIGPIPE received, terminating.\n")

  # Finally, exit with an exit code of choice.
  sys.exit(141)

[1] Note that in bash you will by default only see head's exit code - which is 0 - reflected in $? afterwards. Use echo ${PIPESTATUS[0]} to see Python's exit code.

[2] Curiously, on macOS 10.15.7 (Catalina), with Python 3.9.2 (but not 2.x), I see exit code 120, but the docs say 1, and that's what I also see on Linux.

Gauzy answered 7/5, 2015 at 3:50 Comment(0)
M
51

I haven't reproduced the issue, but perhaps this method would solve it: (writing line by line to stdout rather than using print)

import sys
with open('a.txt', 'r') as f1:
    for line in f1:
        sys.stdout.write(line)

You could catch the broken pipe? This writes the file to stdout line by line until the pipe is closed.

import sys, errno
try:
    with open('a.txt', 'r') as f1:
        for line in f1:
            sys.stdout.write(line)
except IOError as e:
    if e.errno == errno.EPIPE:
        # Handle error

You also need to make sure that othercommand is reading from the pipe before it gets too big - https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer

Mainland answered 8/1, 2013 at 4:33 Comment(5)
While this is good programming practice, I don't think it has anything to do with the broken pipe error the questioner is getting (which probably has to do with the print call, not with reading the files).Drive
@Drive I added a few questions and alternative methods and was hoping for some feedback from the author. If the problem is sending a large quantity of data from an open file directly to the print statement then perhaps one of the alternatives above would fix it.Mainland
(The simplest solutions are often the best - unless there's a particular reason to load a whole file then print it, do it a different way)Mainland
Awesome work in troubleshooting this! While I could take this answer for granted, I could appreciate this only after seeing how the other answers (and my own approach) paled in comparison to your answer.Reply
You might want also a newline feed: sys.stdout.write(line + '\n')Starstarboard
D
31

A "Broken Pipe" error occurs when you try to write to a pipe that has been closed on the other end. Since the code you've shown doesn't involve any pipes directly, I suspect you're doing something outside of Python to redirect the standard output of the Python interpreter to somewhere else. This could happen if you're running a script like this:

python foo.py | someothercommand

The issue you have is that someothercommand is exiting without reading everything available on its standard input. This causes your write (via print) to fail at some point.

I was able to reproduce the error with the following command on a Linux system:

python -c 'for i in range(1000): print i' | less

If I close the less pager without scrolling through all of its input (1000 lines), Python exits with the same IOError you have reported.

Drive answered 8/1, 2013 at 11:10 Comment(9)
Yes, this is true, but how do I fix it?Vitality
@JOHANNES_NYÅTT: It's probably not something that you can fix from within your Python program. Instead, you need to find out why the other program is exiting and fix that (either by changing its code, if you can, or by calling it differently). We might be able to help, but you'll need to tell us what othercommand is, and what you expect it to do with the input you're sending it.Drive
please let me know how to fix it.Vitality
For some reason when the file is a few kb it works, but it gives this error when the file is above 1 mbVitality
@JOHANNES_NYÅTT: It may work for small files because of the buffering provided by pipes on most Unix-like systems. If you can write the whole contents of the files into the buffer, it doesn't raise an error if the other program never reads that data. However, if the write blocks (because the buffer is full), then it will fail when the other program quits. To say again: What is the other command? We can't help you any more with only the Python code (since it's not the part that's doing the wrong thing).Drive
@Exploitation That's because your pipe is probably only around 64kB - if you try to write too much to it and other process isn't reading it then it'll throw an error. I've shown how to catch the error in my last example. Is there anything else you would like?Mainland
I got this problem when piping to head... exception after ten lines of output. Quite logical, but still unexpected :)Gluey
"That's because your pipe is probably only around 64kB - if you try to write too much to it and other process isn't reading it then it'll throw an error." Ref that comment I don't think it's true - the next write will simply block until the reader empties out [some of] the buffer by reading it.Zoubek
@Blckknght: Good info in general, but re "fix that: and "the part that's doing the wrong thing": a SIGPIPE signal does not necessarily indicate an error condition; some Unix utilities, notably head, by design, during normal operation close the pipe early, once they've read as much data as they needed.Gauzy
B
22

I feel obliged to point out that the method using

signal(SIGPIPE, SIG_DFL) 

is indeed dangerous (as already suggested by David Bennet in the comments) and in my case led to platform-dependent funny business when combined with multiprocessing.Manager (because the standard library relies on BrokenPipeError being raised in several places). To make a long and painful story short, this is how I fixed it:

First, you need to catch the IOError (Python 2) or BrokenPipeError (Python 3). Depending on your program you can try to exit early at that point or just ignore the exception:

from errno import EPIPE

try:
    broken_pipe_exception = BrokenPipeError
except NameError:  # Python 2
    broken_pipe_exception = IOError

try:
    YOUR CODE GOES HERE
except broken_pipe_exception as exc:
    if broken_pipe_exception == IOError:
        if exc.errno != EPIPE:
            raise

However, this isn't enough. Python 3 may still print a message like this:

Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

Unfortunately getting rid of that message is not straightforward, but I finally found http://bugs.python.org/issue11380 where Robert Collins suggests this workaround that I turned into a decorator you can wrap your main function with (yes, that's some crazy indentation):

from functools import wraps
from sys import exit, stderr, stdout
from traceback import print_exc


def suppress_broken_pipe_msg(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        try:
            return f(*args, **kwargs)
        except SystemExit:
            raise
        except:
            print_exc()
            exit(1)
        finally:
            try:
                stdout.flush()
            finally:
                try:
                    stdout.close()
                finally:
                    try:
                        stderr.flush()
                    finally:
                        stderr.close()
    return wrapper


@suppress_broken_pipe_msg
def main():
    YOUR CODE GOES HERE
Baroscope answered 3/3, 2016 at 0:59 Comment(4)
This didn't seem to fix it for me.Devora
It worked for me after I added except BrokenPipeError: pass in the supress_broken_pipe_msg functionJeu
Yeah, this requires BOTH the suppress_broken_pipe_msg bit to silence the Exception ignored in: <_io.TextIOWrapper message and the first except broken_pipe_exception code block to actually handle the broken pipe exception in a Python 2/3 compatible way.Monolatry
And even this doesn't work on Windows, which raises an OSError instead of BrokenPipeError.Monolatry
L
4

I know this is not the "proper" way to do it, but if you are simply interested in getting rid of the error message, you could try this workaround:

python your_python_code.py 2> /dev/null | other_command
Lease answered 6/12, 2018 at 20:52 Comment(2)
python your_python_code.py | tee /dev/null | other_command — also works. But I cannot understand why it works with stderr.Lymn
Would that send the whole output to /dev/null?Lease
D
3

The top answer (if e.errno == errno.EPIPE:) here didn't really work for me. I got:

AttributeError: 'BrokenPipeError' object has no attribute 'EPIPE'

However, this ought to work if all you care about is ignoring broken pipes on specific writes. I think it's safer than trapping SIGPIPE:

try:
    # writing, flushing, whatever goes here
except BrokenPipeError:
    exit( 0 )

You obviously have to make a decision as to whether your code is really, truly done if you hit the broken pipe, but for most purposes I think that's usually going to be true. (Don't forget to close file handles, etc.)

Dog answered 8/4, 2020 at 14:54 Comment(0)
A
2

Depending on the exact cause of the issue, it might help to set an environment variable PYTHONUNBUFFERED=1, which forces the stdout and stderr streams to be unbuffered. See: https://docs.python.org/3/using/cmdline.html#cmdoption-u

So, your command

open.py | othercommand

becomes:

PYTHONUNBUFFERED=1 open.py | othercommand

Example:

$ python3 -m http.server | tee -a access.log
^CException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

$ PYTHONUNBUFFERED=1 python3 -m http.server | tee -a access.log
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
^C
$ 
Annmarieannnora answered 8/7, 2021 at 1:16 Comment(0)
R
1

This can also occur if the read end of the output from your script dies prematurely

ie open.py | otherCommand

if otherCommand exits and open.py tries to write to stdout

I had a bad gawk script that did this lovely to me.

Radborne answered 17/10, 2013 at 18:23 Comment(1)
It's not about the process reading from the pipe dying, necessarily: some Unix utilities, notably head, by design, during normal operation close the pipe early, once they've read as much data as they needed. Most CLIs simply defer to the system for its default behavior: quietly terminating the reading process and reporting exit code 141 (which, in a shell, isn't readily apparent, because a pipeline's last command determines the overall exit code). Python's default behavior, unfortunately, is to die noisily.Gauzy
Q
-3

Closes should be done in reverse order of the opens.

Quadrature answered 28/1, 2015 at 9:56 Comment(1)
While that is good practice in general, not doing is not a problem in itself and doesn't explain the OP's symptoms.Gauzy

© 2022 - 2024 — McMap. All rights reserved.