Capture stdout to a variable but still display it in the console

Question

I have a bash script which calls several long-running processes. I want to capture the output of those calls into variables for processing reasons. However, because these are long running processes, I would like the output of the rsync calls to be displayed in the console in real-time and not after the fact.

To this end, I have found a way of doing it but it relies on outputting the text to /dev/stderr. I feel that outputting to /dev/stderr is not a good way of doing things.

VAR1=$(for i in {1..5}; do sleep 1; echo $i; done | tee /dev/stderr)

VAR2=$(rsync -r -t --out-format='%n%L' --delete -s /path/source1/ /path/target1 | tee /dev/stderr)

VAR3=$(rsync -r -t --out-format='%n%L' --delete -s /path/source2/ /path/target2 | tee /dev/stderr)

In the example above, I am calling rsync a few times and I want to see the file names as they are processed, but in the end I still want the output in a variable because I will be parsing it later.

Is there a 'cleaner' way of accomplishing this?

If it makes a difference, I am using Ubuntu 12.04, bash 4.2.24.

Versify

Answer 1

Duplicate &1 in your shell (in my example to 5) and use &5 in the subshell (so that you will write to stdout (&1) of the parent shell):

exec 5>&1
FF=$(echo aaa|tee >(cat - >&5))
echo $FF

This will print "aaa" two times, once because of the echo in the subshell, and the second time it prints the value of the variable.

In your code:

exec 5>&1
VAR1=$(for i in {1..5}; do sleep 1; echo $i; done | tee >(cat - >&5))
# use the value of VAR1

Kelci

This worked, thanks! So in this case, is the '5' must be a file descriptor, I will read up and learn about these. – Versify

Shouldn't the descriptor be closed in the parent shell? – Volin

@akhan: I assume that's exec 5>&- then? – Directions

How about FF=$(echo aaa | tee /dev/tty)? Source – Arius

I guess on my machine, 5 refers to /dev/fd/63? tee: /dev/fd/63: No such file or directory – Sour

This does not preserve output colors. – Wernsman

@user1011471, when you write tee >(inner_cmd...), the shell opens a pipe to the inner command, such that the pipe's write side is also inherited by the outer command (tee), typically under the descriptor 63. The outer command has no idea the shell did anything funny, it gets a file name argument /dev/fd/63. en.wikipedia.org/wiki/Process_substitution Don't know why you got this error, but it's unrelated to inner command writing to descriptor 5. – Veda

@Op De Cirkel Your version outputs 2 aaa to both stdout and when redirected to another file. However, if the FF=$(echo aaa|tee >(cat - >&5)) becomes FF=$(echo aaa|tee /dev/fd/5), this version outputs 2 aaa to stdout but 1 aaa when redirected to another file... Could you help to suggest how to understand this behavior ? Many thanks ! – Jam

Answer 2

Op De Cirkel's answer has the right idea. It can be simplified even more (avoiding use of cat):

exec 5>&1
FF=$(echo aaa|tee /dev/fd/5)
echo $FF

Footbridge

Wouldn't /dev/fd/5 be OS specific? – Volin

I had been wondering if you could just use tee /dev/fd/1, but that doesn't work because the output still gets captured by $(). So in case anyone else is wondering the same thing, it is necessary to use an extra file descriptor (like 5). – Annabelleannabergite

We could go simplifying even further and making this a oneliner without the exec: { FF=$(echo aaa|tee /dev/fd/5); } 5>&1 The braces allow for the redirection to happen before the subshell command is run, while $FF still remains in the scope of the current shell (that wouldn't work with normal brackets ( ). This way there's even no need to close FD 5 afterwards, which is a overlooked hygienic habit. – Privily

@Volin No it wouldn't, bash is said to be emulating this path should it not exist in the OS by itself. – Privily

If this is run using sudo -u <other non-root user> <script> then Op De Cirkel's answer works but this answer does not. Writing to /dev/fd/5 is equivalent to writing directly to the terminal. /dev/fd/5 is a symlink to the /dev/pts/ file for the terminal, which is owned by the user that originally logged in and is not writable by the sudo'd user. However, cat - >&5 writes to a file descriptor that is opened by bash within the process (which is not the same as writing to /dev/fd/5). This file descriptor forwards the write through each parent process, avoiding any permissions issues. – Disaccord

Why does @Annabelleannabergite solution not work? Specifically, why can't you just do tee /dev/fd/1? I don't understand if you're going to tee /dev/fd/5 where 5 points 1, how is that different than just tee /dev/fd/1? – Blare

@Russell Davis Op De Cirkel'version outputs 2 aaa to both stdout and when redirected to another file. However, if the FF=$(echo aaa|tee >(cat - >&5)) becomes FF=$(echo aaa|tee /dev/fd/5), this version outputs 2 aaa to stdout but 1 aaa when redirected to another file... Could you help to suggest how to understand this behavior ? Many thanks ! – Jam

Answer 3

Here's an example capturing stderr, stdout, and the command's exit code. This is building on the answer by Russell Davis.

exec 5>&1
FF=$(ls /taco/ 2>&1 |tee /dev/fd/5; exit ${PIPESTATUS[0]})
exit_code=$?
echo "$FF"
echo "Exit Code: $exit_code"

If the folder /taco/ exists, this will capture its contents. If the folder doesn't exist, it will capture an error message and the exit code will be 2.

If you omit 2>&1then only stdout will be captured.

Passed

"capturing both stderr and the command's exit code" -- it's capturing stderr, stdout, and the command's exit code. – Gritty

Good clarification @Gritty -- I edited my answer. – Passed

Answer 4

If by "the console" you mean your current TTY, try

variable=$(command with options | tee /dev/tty)

This is slightly dubious practice because people who try to use this sometimes are surprised when the output lands somewhere unexpected when they don't have a TTY (cron jobs etc).

Discontinuity

It does not work in Docker containers. tee: /dev/tty: No such device or address – Kappenne

I don't think that's generally true. It works for me here in a Debian Docker container just fine. – Discontinuity

You can use tty=$(readlink /proc/$$/fd/2) in docker and then NEW=$(echo "blah" | tee "$tty") – Mebane

Answer 5

You can use more than three file descriptors. Try here:

http://tldp.org/LDP/abs/html/io-redirection.html

"Each open file gets assigned a file descriptor. [2] The file descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively. For opening additional files, there remain descriptors 3 to 9. It is sometimes useful to assign one of these additional file descriptors to stdin, stdout, or stderr as a temporary duplicate link."

The point is whether it's worth to make script more complicated just to achieve this result. Actually it's not really wrong, the way you do it.

Pacifism