bash error in sort "sort: write failed: standard output: Broken pipe" [duplicate]

Question

When I run this script I recieve an error message with: "sort: write failed: standard output: Broken pipe"

If someone can help me it would be awesome, I am going crazy with this error

the input file is a list of files that all contain DNA sequences in a FASTA format, so each file has several sequences (each sequence in a single line) with the format: in $1 (Identifier) in $2,3,4,5,6,7&8 (more values) in $9 (the DNA sequence)

Then I want select each of this sequences by number of sequences ($common_hits) in each file (this number is not a fix value but i set 6 for the example) -All the files with less than 6 sequences must be removed -Files with 6 sequences are ok -The files with more than 6 sequences have to be reduced to 6 sequences (these sequences are selected by the higher values of field $5)

the output files must have all 6 sequences and the sequence (field $9) has to be in the line after the identifier

I am not removing the originals files with more than 6 sequences for now, because I want to be sure it works

par_list=`ls -1 *BR`

common_hits="6"

for i in ${par_list}

do

   if [ "`cat ${i} | wc -l`" -lt "${common_hits}" ]
   then
      rm -f ${i}
   elif [ "`cat ${i} | wc -l`" -gt "${common_hits}" ]
   then
      cat ${i} | sort -nr -k 5 | head -n ${common_hits} | \
      awk '{print $1"    " $2"    " $3"    " $4"    " $5"    " $6"    " $7"           "$8 ; print $9}' > ${i}.ph 
   fi

done 

Benefactress

Welcome to stack overflow site, please wrap up your code(s) into code tags as per forum rules and explain the issues more clearly so that we could try to help you, thank you. – Simons

I could see we could make lot of changes into your script, could you please post 2 things, 1st what is your requirement(with sample Input_file and expected output_file), 2nd in case you want the same script only, use set -x in the starting of your script and show us the output, though you have to still let us know about your requirement(what is Input_file and about output_file in code tags) please. – Simons

thank you for the advise, I hope it is easier to understand now – Benefactress

Thank you for adding information, could you please also add sample input and expected sample output too in code tags. – Simons

what do you mean by code tags?, a file of the input and a file of the output? – Benefactress

Every one of those cat program calls is unnecessary, wc and sort both take an input filename as their rightmost parameter. – Quintero

No point to head | awk either -- awk can do the work of picking out which lines to read itself. `awk '{ print ... } NR > 5 { exit }' – Willson

As another aside -- see Why you shouldn't parse the output from ls. The loop is better written (prone to fewer unexpected misbehaviors) as for i in *BR. – Willson

Thanks for the advises, I will try to improve my code – Benefactress

Answer 1

sort | head always reports an error, if head exits (or otherwise closes its stdin) before sort has written all its output (as will be the case, if the stream written by sort is much longer than that consumed by head). This is by-design: If sort can't write all its output, it's expected to fail; if it ignored such failures, it would also ignore cases where it couldn't write its output for other reasons (disk full, broken network connection, etc.

There's nothing unusual or undesirable about this. If you don't care about the error, ignore it, and check the number of lines of output from the pipeline to determine whether you had an error condition instead.

Willson

Thank you so much, so even if there is an error with sort it shouldnt affect to my output? – Benefactress

Right -- the "error" is just that sort isn't able to write more than $common_hits lines. – Willson

In other words, "Broken pipe" errors are often harmless and can be ignored or, if you're lucky, even suppressed. – Aphasic

BTW, consider running your code through shellcheck.net and fixing what it finds; you've got a bunch of quoting bugs. They may not be causing immediate effects now, but that doesn't mean they won't ever. – Willson

but then sort is not sorting checking all the sequences? I nedd to sort the sequences comparing all the sequences – Benefactress

In order to get that common_hits lines of output, it absolutely does need to read the complete input stream. What part of my answer or comment do you read as claiming it doesn't do so? – Willson

Pedantically, this error is caused by write syscall raising SIGPIPE. It is possible for (last) write to return successfully, yet for head never read it and not observe this error. – Execrative

@MaximEgorushkin, as someone known to get a bit pedantic myself, I'm happy to rephrase to be more precise. Better? – Willson

@CharlesDuffy Definitely better. – Execrative

You were right! The output files are correct, I panicked when I saw the error. Thanks to everyone – Benefactress

Hello again, I am running the script normally but sometimes the .error file has also this message: "sort: fflush failed: standard output: Broken pipe sort: write error". Is this normal too? it does not affect the output? – Benefactress

Yes, it's normal. No, it doesn't change the output -- just means that the failure happened while flushing buffers rather than doing a write into a buffer (and letting the C library decide when to flush) -- but it's the exact same error with the exact same effects, just happening with different timing. – Willson

Thank you so much! – Benefactress