Is sed blocking?

Question

I had the impression sed wasn't blocking, because when I do say:

iostat | sed

sed processes the data as it arrives, but when I do

iostat | sed | netcat

Then sed blocks netcat.

Am I right?

Setback

Answer 1

sed will work in buffered mode when it doesn't print to a terminal. This means that it will try to fill its internal buffer before doing any processing and output by default.

This is done to increase throughput, because normally in a pipe you don't care about the timing, but want as much data processed in a given time as possible.

Passing -u to sed will tell it to work unbuffered, therefore working the same way it works when output goes to a terminal.

Adit

sed: illegal option -- u – Moy

@CommaToast: then either use GNU sed or find out if your flavour of sed has an equivalent option (and post it here as an answer, when you do). – Adit

On my MacOSX comp, -u is missing, but -l works for line-buffering, which is preferred. – Landes

This doesn't work for me. I installed GNU sed and used -u but the results were the same: buffered. – Antigua

Answer 2

In addition to what @saua says, sed is at least line oriented, that it, it read a line, then operates on it so it will always be buffering at least one line. In addition, sed can work in multiline mode. If you are using a multiline pattern, then sed can't output it's current buffer until it knows that the pattern either doesn't apply or the pattern has been processed.

Tomy

Answer 3

I don't know if I understand the question right, but in your example, it should be like this:

• sed waits for iostat to send data
• iostat will wait for sed to process the data if sed is slow
• netcat waits for sed to send data
• sed will wait for netcat again, if it is slow

Other than that, sed shouldn't need to read all its input to produce output.

Do you observe any delays that cannot be explained by this and some small buffering?

Adder

Answer 4

stdbuf can assist with changing the buffering behavior of the standard io streams. You can try this to change the buffering behavior:

... | stdbuf -oL -eL sed -e ... | ...

Osculum