tee stdout and stderr to separate files while retaining them on their respective streams

Question

I'm trying to write a script the essentially acts as a passthru log of all the output created by a (non-interactive) command, without affecting the output of the command to other processes. That is to say, stdout and stderr should appear the same as if they had not run through my command.

To do this, I'm trying to redirect stdout and stderr separately to two different tees, each for a different file, and then recombine them so that they still appear on stdout and stderr, respectively. I have seen a lot of other questions about teeing and redirecting and have tried some of the answers gleaned from those, but none of them seem to work combining both splitting the stream to separate tees and then recombining them correctly.

My attempts are successfully splitting the output into the right files, but the streams are not correctly retained for actual stdout/stderr output. I see this in a more complicated setting, so I created simplified commands where I echoed data to stdout or stderr as my "command" as shown below.

Here are a couple of things that I have tried:

{ command | tee ~/tee.txt; } 2>&1 | { tee ~/tee2.txt 1>&2; }

Running my simple test I see:

$ { { { echo "test" 1>&2; } | tee ~/tee.txt; } 2>&1 | { tee ~/tee2.txt 1>&2; } } > /dev/null
test
$ { { { echo "test" 1>&2; } | tee ~/tee.txt; } 2>&1 | { tee ~/tee2.txt 1>&2; } } 2> /dev/null
$

Ok, this is as I expect. I am echoing to stderr, so I expect to see nothing when I redirect the final stderr to /dev/null and my original echo when I only redirect stdout.

$ { { { echo "test";  } | tee ~/tee.txt; } 2>&1 | { tee ~/tee2.txt 1>&2; } } > /dev/null
test
$ { { { echo "test";  } | tee ~/tee.txt; } 2>&1 | { tee ~/tee2.txt 1>&2; } } 2> /dev/null
$

This is backwards! My command sends only data to stdout, so I would expect to see nothing when I redirect the final stdout to null. But the reverse is true.

Here is the second command I tried, it is a bit more complicated:

{ command 2>&3 | tee ~/tee.txt; } 3>&1 1>&2 | { tee /home/michael/tee2.txt 1>&2; }

Unfortunately, I see the same identical behavior as before.

I can't really see what I am doing wrong, but it appears that stdout is getting clobbered somehow. In the case of the first command, I suspect that this is because I am combining stdout and stderr (2>&1) before I pipe it to the second tee, but if this were the case I would expect to see both stdout and stderr in the tee2.txt file, which I don't - I only see stderr! In the case of the second command, my impression from reading the answer I adapted for this command is that descriptors are getting swapped around so as to avoid this problem, but obviously something is still going wrong.

Edit: I had another thought, that maybe the second command is failing because I am redirecting 1>&2 and that is killing stdout from the first tee. So I tried to redirecting it with 1>&4 and then redirecting that back to stdout at the end:

{ command 2>&3 | tee ~/tee.txt; } 3>&1 1>&4 | { tee /home/michael/tee2.txt 1>&2 4>&1; }

But now I get:

-bash: 4: Bad file descriptor

I also tried redirecting descriptor 2 back to descriptor 1 in the final tee:

{ command 2>&3 | tee ~/tee.txt; } 3>&1 1>&2 | { tee /home/michael/tee2.txt 1>&2 2>&1; }

and:

{ command 2>&3 | tee ~/tee.txt; } 3>&1 1>&2 | { tee /home/michael/tee2.txt 1>&2; } 2>&1

Lambert

Answer 1

A process-substitution-based solution is simple, although not as simple as you might think. My first attempt seemed like it should work

{ echo stdout; echo stderr >&2; } > >( tee ~/stdout.txt ) \
                                 2> >( tee ~/stderr.txt )

However, it doesn't quite work as intended in bash because the second tee inherits its standard output from the original command (and hence it goes to the first tee) rather than from the calling shell. It's not clear if this should be considered a bug in bash.

It can be fixed by separating the output redirections into two separate commands:

{ { echo stdout; echo stderr >&2; } > >(tee stdout.txt ); } \
                                   2> >(tee stderr.txt )

Update: the second tee should actually be tee stderr.txt >&2 so that what was read from standard error is printed back onto standard error.

Now, the redirection of standard error occurs in a command which does not have its standard output redirected, so it works in the intended fashion. The outer compound command has its standard error redirected to the outer tee, with its standard output left on the terminal. The inner compound command inherits its standard error from the outer (and so it also goes to the outer tee, while its standard output is redirected to the inner tee.

Theologue

This doesn't seem to be keeping the original stderr and stdout separate; it seems to combine them. For instance, if I wrap the entire statement in { ... } > /dev/null I see no output, but if I wrap it in { ... } 2> /dev/null I see both "stdout" and "stderr". – Lambert

changing the second tee to (tee stderr.txt 1&2) appears to have the desired effect. – Lambert

Assuming you mean 2> >(tee stderr 1>&2), yes, that was an omission on my part to restore the output of that tee to standard error. – Theologue

Doesn't the simpler version also work with this correction? I think it might be better to update the answer in the code blocks and explain in the comments, rather than leave the incorrect version most prominent. – Trinitroglycerin

@SamBrightman You're right, the first answer also works after adding the correction. One other way to make the first answer working is to invert the order of the redirections, as bash seems to evaluate the right one first. – Goldstone

Strange behavior for bash with version 5. tees in subprocesses hang for this examples. stdout by some reason is printed after $ (place in shell where user input is expected). stderr is printed where expected. – Cotton