Piping both stdout and stderr in bash?

Question

It seems that newer versions of bash have the &> operator, which (if I understand correctly), redirects both stdout and stderr to a file (&>> appends to the file instead, as Adrian clarified).

What's the simplest way to achieve the same thing, but instead piping to another command?

For example, in this line:

cmd-doesnt-respect-difference-between-stdout-and-stderr | grep -i SomeError

I'd like the grep to match on content both in stdout and stderr (effectively, have them combined into one stream).

Note: this question is asking about piping, not redirecting - so it is not a duplicate of the question it's currently marked as a duplicate of.

Jews

See the second answer (https://mcmap.net/q/12148/-redirect-stderr-and-stdout-in-bash-duplicate) on the linked question for the correct way to pipe both stdout and stderr. No need for another question. – Goahead

@triplee Not an exact duplicate, is it? Pipe vs. redirect to file? – Echinoid

@BenjaminW There is at least one answer there which solves both scenarios, though it's not the accepted answer. This is a fairly common question so we could probably find a better duplicate, or ask a moderator to merge these - or even, in the worst case, craft an entirely new canonical for this topic. If you find a better dupe, by all means propose it. Thanks in advance. – Flowerer

@Flowerer Solves, yes, but none of the answers use the |& shortcut, which I think is by far the most convenient solution to "redirect both stdout and stderr to a pipe". – Echinoid

This is not a duplicate of the linked question, and it wasn't clear that Marko's answer did what I wanted. Also, it doesn't mention |&. Voting to reopen. – Aguste

Answer 1

(Note that &>>file appends to a file while &> would redirect and overwrite a previously existing file.)

To combine stdout and stderr you would redirect the latter to the former using 1>&2. This redirects stdout (file descriptor 1) to stderr (file descriptor 2), e.g.:

$ { echo "stdout"; echo "stderr" 1>&2; } | grep -v std
stderr
$

stdout goes to stdout, stderr goes to stderr. grep only sees stdout, hence stderr prints to the terminal.

On the other hand:

$ { echo "stdout"; echo "stderr" 1>&2; } 2>&1 | grep -v std
$

After writing to both stdout and stderr, 2>&1 redirects stderr back to stdout and grep sees both strings on stdin, thus filters out both.

You can read more about redirection here.

Regarding your example (POSIX):

cmd-doesnt-respect-difference-between-stdout-and-stderr 2>&1 | grep -i SomeError

or, using >=bash-4:

cmd-doesnt-respect-difference-between-stdout-and-stderr |& grep -i SomeError

Meemeece

Thanks for the clarification on &>>. I've corrected my question. – Jews

I added your example to my answer, just in case it was not obvious based on my given examples. As a side-note, you can also use the bash-specific |& instead of 2>&1 |. – Overboard

Side note about the shortcut |& proposed by @AdrianFrühwirth for future readers: this feature is only supported with bash version 4+. If you're using 3 or below, you have to stick with 2>&1 |. – Ofris

Bash redirection is very well explained here. @AdrianFrühwirth has done a good job, the link pasted goes even further. Sometimes, I wish the official Bash documentation was that good. – Accredit

@AdrianFrühwirth FYI: your first example references 2>&1 in the description but actually uses 1&2 in the code. – Tray

Good answer @AdrianFrühwirth, thanks! I wonder if you can comment on my new question #73294878 which is related. Using your suggestions here I still can't find a solution. – Telmatelo

Answer 2

Bash has a shorthand for 2>&1 |, namely |&, which pipes both stdout and stderr (see the manual):

cmd-doesnt-respect-difference-between-stdout-and-stderr |& grep -i SomeError

This was introduced in Bash 4.0, see the release notes.

Echinoid

Thanks for adding this for completeness. I'm going to keep the other answer correct as many folks are still using bash pre-4.0. But this is useful. – Jews

Most notably perhaps, the Bash that ships on macOS is too old to support this. – Arnaldo

@Arnaldo but the zsh isn't – Earley

Since ksh uses |& for coproc, this seems like a bad choice for an unnecessary shorthand. I hate to see lines with a stack of dups and redirections as much as the next guy, but there's something to be said for being explicit....and I apologize that this comment doesn't add much. I just wanted to express distaste for the shorthand without downvoting an actually helpful answer, because it's good that people see this. I didn't know this, so thanks for making me aware. – Aerophagia

@PaulHodges I agree that it's not portable - I mainly like using it for interactive Bash sessions to avoid typing too much. – Echinoid

Just a small note. That does not work for input from the keyboard, since that doesn't go out of either stdout nor stderr (apparently). – Clava

@ScottFranco Do you have an example? Output has to go somewhere, but it might be buffered or similar. – Echinoid

Sure: samiam@samiam-h-pc-2:~/projects/pascal/pascal-p6/pascaline_tests$ p6 debug_test1 debug_test --debug --debugsrc &> junk hi there q The "hi there" and "q" were typed by me. Linux normally buffers the command line input (you can override this), and that does not go to stdout nor stderr. Apologies for messy example, it is the same with any program that takes input. – Clava

Answer 3

If you want to pipe STDOUT and STDERR to different subshells instead of both to the same one (as |& / 2>&1 does), I use this:

producer \
  >  >( stdout_pipeline... ) \
  2> >( stderr_pipeline... ) \
  ...

While this uses the redirection operator, it uses pipes under the hood to connect the producer to the subshells.

You can process multiple output streams (2 is not the limit) this way, and also pipe (potentially multiple) input streams from subshells in a similar way with the penguin operator ([n]< <(...)).

Simply