Pipe only STDERR through a filter

Question

Is there any way, in bash, to pipe STDERR through a filter before unifying it with STDOUT? That is, I want

STDOUT ────────────────┐
                       ├─────> terminal/file/whatever
STDERR ── [ filter ] ──┘

rather than

STDOUT ────┐
           ├────[ filter ]───> terminal/file/whatever
STDERR ────┘

Gumbotil

See also How to pipe stderr, and not stdout?. – Drus

This deserves an upvote simply because it includes a beautiful ASCII-art diagram. – Wilton

Answer 1

Here's an example, modeled after how to swap file descriptors in bash . The output of a.out is the following, without the 'STDXXX: ' prefix.

STDERR: stderr output
STDOUT: more regular

./a.out 3>&1 1>&2 2>&3 3>&- | sed 's/e/E/g'
more regular
stdErr output

Quoting from the above link:

1. First save stdout as &3 (&1 is duped into 3)
2. Next send stdout to stderr (&2 is duped into 1)
3. Send stderr to &3 (stdout) (&3 is duped into 2)
4. close &3 (&- is duped into 3)

Calvities

It doesn't work for me. Finally I make it work with 3>&2 2>&1 1>&3-. – Nicky

Caution: this assumes FD 3 is not in use, and doesn't undo the swapping of file descriptors 1 and 2, so you can't go on to pipe this to yet another command. See this answer for further detail and work-around. For a much cleaner syntax for {ba,z}sh, see this answer. – Huskamp

Answer 2

TL;DR:

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

This will work in both bash and zsh. Bash is pretty much ubiquitous these days, however, if you really do need a (really gnarly) solution for POSIX sh, then see here.

---

Explanation

By far, the easiest way to do this is to redirect STDERR via process substitution:

Process substitution allows a process’s input or output to be referred to using a filename. It takes the form of

>(list)

The process list is run asynchronously, and its input or output appears as a filename.

So what you get with process substituion is a filename.

Just like you could do:

$ cmd 2> filename

you can do

$ cmd 2> >(filter >&2)

The >&2 redirect's filter's STDOUT back to the original STDERR.

Huskamp

There is a caveat that goes along this answer: bash does not wait for the substituted process to complete, while the FD juggling to swap 1 and 2 does. This may be important. I got burned by this doing exec 2> >(while .. read .. echo) in a script running as a systemd service. journald captures fd2 of the service, and infers log level from an '<N>' prefix: prepend <2> to output lines, and you get ERROR level assigned to journal records. But sometimes systemd was swifter to kill the whole process group upon exit of the main one before it could log its last, the most important message! – Unimproved

Nice simple solution. Caveat: be careful using this as part of a cron job; process substitution is not available in some shells used by cron. – Hellas

I've been using unix/linux command line for many decades and did not know this one. Thanks. Here's a example of ignoring stdout and filtering stderr cat file 1>/dev/null 2> >(grep -v <search-term> >&2) – Acclamation

Answer 3

TL;DR: (bash and zsh)

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

---

Many answers on the StackExchange network have the form:

cat /non-existant 3>&1 1>&2 2>&3 3>&- | sed 's/e/E/g'

This has a built-in assumption: that file descriptor 3 isn't being used for something else.

Instead, use a named file descriptor, and {ba,z}sh will allocate the next available file descriptor >= 10:

cat /non-existant {tmp}>&1 1>&2 2>&$tmp {tmp}>&- | sed 's/e/E/g'

Note that named file descriptors aren't supported by POSIX sh.

The other issue with the above is that the command cannot be piped to further commands without again swapping STDOUT and STDERR back to their original values.

To allow onward piping in POSIX sh, (and still assuming FD 3 is not it use) it gets complicated:

(cmd 2>&1 >&3 3>&- | stderr-filter >&2 3>&-) 3>&1

So, Given the assumption and gnarly syntax of this, you're likely to be better off using the simpler bash/zsh syntax shown in the TL;DR above, and explained here.

---

practical demonstration, grepping only stderr:

$ ls -l . noexistABC noexistXYZ
ls: cannot access 'noexistABC': No such file or directory
ls: cannot access 'noexistXYZ': No such file or directory
.:
total 4
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 bar.txt
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 foo.txt
drwxrwxr-x 2 frank frank 4096 Aug 19 12:26 someFolder


$ ( ls -l . noexistABC noexistXYZ 2>&1 >&3 3>&- | grep ABC >&2 3>&-) 3>&1
.:
ls: cannot access 'noexistABC': No such file or directory
total 4
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 bar.txt
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 foo.txt
drwxrwxr-x 2 frank frank 4096 Aug 19 12:26 someFolder

Huskamp

Why post the same answer twice? – Dongola

Answer 4

A naive use of process substitution seems to allow filtering of stderr separately from stdout:

:; ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 )
out
e:err

Note that stderr comes out on stderr and stdout on stdout, which we can see by wrapping the whole thing in another subshell and redirecting to files o and e

( ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 ) ) 1>o 2>e

Voter

why the :; at the beginning? I tried the magic line without, and doesn't seem to make a difference. – Joeljoela

Some people use $ as a command prompt. Then they often write shell examples like: $ cat /var/log/syslog | fgrep .... However, this line is not copy-pastable because of the $. :; looks like a prompt but is basically a shell no-op; so you can select and paste the whole line safely. – Voter

Ok, but you could just omit the :; and the line would be also copy-pastable :) To me :; doesn't look like a prompt, it was not making the example clearer (separating commands from output) but confusing. Though I understand your point of view and let's not discuss syntax/conventions. – Joeljoela

Answer 5

I find the use of bash process substitution easier to remember and use as it reflects the original intention almost verbatim. For example:

$ cat ./p
echo stdout
echo stderr >&2
$ ./p 2> >(sed -e 's/s/S/') | sed 's/t/T/'
sTdout
STderr

uses the first sed command as a filter on stderr only and the second sed command to modify the joined output.

Note that the white space after 2> is mandatory for the command to be parsed correctly.

Lunarian

Answer 6

The last part of this page of the Advanced Bash Scripting Guide is "redirecting only stderr to a pipe".

# Redirecting only stderr to a pipe.

exec 3>&1                              # Save current "value" of stdout.
ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
#              ^^^^   ^^^^
exec 3>&-                              # Now close it for the remainder of the script.

# Thanks, S.C.

This may be what you want. If not, some other part of the ABSG should be able to help you, it is excellent.

Fabled

We are somewhat hesitant to recommend the ABSG as a reference, as it mixes documentation, prescription, and opinion without clearly marking the differences. Some sections also have dubious content, although the one you link to seems fine. – Dormouse

Answer 7

Take a look at named pipes:

$ mkfifo err
$ cmd1 2>err |cat - err |cmd2

Phenolic

won't cat - err break the interspersing of stdout and stderr? – Gumbotil

@Martin - it depends. If cmd1, cat, or cmd2 buffers output, then you could see output out of sequence. – Allopatric

This won't run. The shell can't finish opening err for writing until cat opens it for reading, but cat won't open it until it finishes reading stdin, which it can't until cmd1 dies, but cmd1 can't start until it finishes opening err. This is a deadlock. fifo(7) documents this: "Normally, opening the FIFO blocks until the other end is opened also." – Hector