How to make output (stdout, stderr) of any shell command unbuffered?

Question

Is there a way to run shell commands without output buffering?

For example, hexdump file | ./my_script will only pass input from hexdump to my_script in buffered chunks, not line by line.

What is a general solution to make the output for any command unbuffered?

Bachman

more upvoted question unix.stackexchange.com/questions/25372 – Weathercock

Perhaps see also #7162321 – Sannyasi

Answer 1

AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:

• http://www.pixelbeat.org/programming/stdio_buffering/

The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.

Mayle

Thanks. I had never heard about pseudo terminals before. – Bachman

Answer 2

Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:

stdbuf -i0 -o0 -e0 command

Singleaction

This worked better than unbuffer for me. stdbuf passed any signals (SIGUSR2 in my case) I sent to it to the command (which is what I wanted to happen), while unbuffer didn't seem to want to. – Aeronaut

stdbuf uses one the aforementioned LD_PRELOAD tricks to do this and hence does not work with statically linked or setuid executables. See this question for a discussion: #13644524 – Mayle

I found this very useful, but limited, as it does not run scripts implicitly. However, stdbuf -o0 bash runs a complete session with all my scripts and aliases available, and with no output buffering on any of the commands, which is exactly what I wanted. Buffering was of course restored on exit from this bash instance. I'm using Ubuntu 16.04. – Aldis

I liked this so much, I put this first in my .bash_profile. if [[ "$0" == "-bash" ]]; then exec /usr/bin/stdbuf -i0 -oL -eL /bin/bash -l; fi – Disfranchise

Worked for me in an exec pattern for sending output to a logfile, with all output line buffered rather than buffering fully disabled: exec > >(stdbuf -i0 -oL -eL awk '{print strftime("%Y-%m-%d %H:%M:%S"), $0 }' | stdbuf -i0 -oL -eL tee "$LOGFILE") 2>&1 – Faison

@NordicMainframe How can I verify stdbuf whether for a specific command or not,say adb – Nostology

literally every, but openwrt. I bet other "embedded" busybox/toybox linuxes would also lack it. – Roumell

Answer 3

The command unbuffer from the expect package disables the output buffering:
Ubuntu Manpage: unbuffer - unbuffer output

Example usage:

unbuffer hexdump file | ./my_script

Foolproof

unbuffer seems to merge stdout and stderr of the command though... (!) – Ascham

@Ascham yes it merges stdout and stderr, that's by design: unbuffer uses expect, and expect is there to emulate the human in human-program interaction, and humans don't get to see whether an output is stdout or stderr. – Dianadiandra

Answer 4

You could also use the script command to make the output of hexdump line-buffered (hexdump will be run in a pseudo terminal which tricks hexdump into thinking its writing its stdout to a terminal, and not to a pipe).

# cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
stty -echo -onlcr
script -q /dev/null hexdump file | ./my_script         # FreeBSD, Mac OS X
script -q -c "hexdump file" /dev/null | ./my_script    # Linux
stty echo onlcr

Swacked

I used the -f parameter to flush, not sure if necessary but it worked. – Risner

In my case, stdbuf failed, but script works like a charm. The program buffers the output when it is being piped to tee, but not when running in terminal. So, script works! Thanks! – Cullie

Very clever solution, and it works!! Thank you. :smile: – Square

Only one problem with that: per script's man page as referenced by this, "Certain interactive commands, such as vi(1), create garbage in the typescript file. Script works best with commands that do not manipulate the screen, the results are meant to emulate a hardcopy terminal." – Perfumer

Answer 5

One should use grep or egrep "--line-buffered" options to solve this. no other tools needed.

Mcfarlane