Why does my Python3 script balk at piping its output to head or tail (sys module)?

Question

I have a Python3 script that writes its output to stdout, but it complains when I pipe that output into head or tail. Note in the sample output below that it sort of works, in that head is returning the first two lines of output as requested.

> ./script.py '../Testdata/*indels.ss' -m 5 | head -2                                                                                              ~/Databases/Avian_genomes/Sandbox/combined
xread
2999 50
Traceback (most recent call last):
  File "./new.py", line 194, in <module>
    sys.stdout.write(lineout)
IOError: [Errno 32] Broken pipe
Exception IOError: IOError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

In contrast, the script has no problem with its output being piped to awk as below.

> ./script.py '../Testdata/*indels.ss' -m 5 | awk 'NR < 3 {print $0}'                                                                              ~/Databases/Avian_genomes/Sandbox/combined
xread
2999 50

Let me know if you need some code from the script beyond what's contained in the error message. I'm not sure what would be relevant.

Ridley

Is there a Unicode character in your third line perhaps? – Cauldron

See also: stackoverflow.com/questions/15793886 – Dariusdarjeeling

Actually IOError: [Errno 32] Broken pipe when piping: prog.py | othercmd seems to be the most active version of this question. – Retorsion

Answer 1

./script.py '../Testdata/*indels.ss' -m 5 | awk 'NR >= 3 {exit} 1'

would show the same behavior as head -2.

You can turn set the SIGPIPE handler to one which quietly kills your program instead:

import signal
signal.signal(signal.SIGPIPE, signal.SIG_DFL)

Trichinosis

but note, this kills your program. If your program is reporting status on something it's doing, adding this code means your program will now silently fail to complete its job when run through head. – Armenian

Answer 2

I'll cite from here:

If a sequence of commands appears in a pipeline, and one of the

reading commands finishes before the writer has finished, the

writer receives a SIGPIPE signal.

That's what head does. Your script hasn't finished writing, but head is already done, so stdout is closed, hence the exception.

Rrhoea

I thought it was probably something like that. Thanks for the explanation and the reference. – Ridley