bash tee remove color

Question

I'm currently using the following to capture everything that goes to the terminal and throw it into a log file

exec 4<&1 5<&2 1>&2>&>(tee -a $LOG_FILE)

however, I don't want color escape codes/clutter going into the log file. so i have something like this that sorta works

exec 4<&1 5<&2 1>&2>&>(
    while read -u 0; do
        #to terminal
        echo "$REPLY"
        #to log file (color removed)
        echo "$REPLY" | sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' >> $LOG_FILE
    done
    unset REPLY #tidy
)

except read waits for carriage return which isn't ideal for some portions of the script (e.g. echo -n "..." or printf without \n).

---

Follow-up to Jonathan Leffler's answer:

Given the example script test.sh:

#!/bin/bash

LOG_FILE="./test.log"
echo -n >$LOG_FILE

exec 4<&1 5<&2 1>&2>&>(tee -a >(sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' > $LOG_FILE))


##### ##### #####
# Main

echo "starting execution"
printf "\n\n"

echo "color test:"
echo -e "\033[0;31mhello \033[0;32mworld\033[0m!"
printf "\n\n"

echo -e "\033[0;36mEnvironment:\033[0m\n  foo: cat\n  bar: dog\n  your wife: hot\n  fix: A/C"
echo -n "Before we get started. Is the above information correct?  "
read YES
echo -e "\n[READ] $YES" >> $LOG_FILE
YES=$(echo "$YES" | sed 's/^\s*//;s/\s*$//')
test ! "$(echo "$YES" | grep -iE '^y(es)?$')" && echo -e "\nExiting... :(" && exit
printf "\n\n"

#...some hundreds of lines of code later...

echo "Done!"


##### ##### #####
# End

exec 1<&4 4>&- 2<&5 5>&-

echo "Log File: $LOG_FILE"

1. The output to the terminal is as expected and there is no color escape codes/clutter in the log file as desired. However upon examining test.log, I do not see the [READ] ... (see line 21 of test.sh).

2. The log file [of my actual bash script] contains the line Log File: ... at the end of it even after closing the 4 and 5 fds. I was able to resolve the issue by putting a sleep 1 before the second exec - I assume there's a race condition or fd shenanigans to blame for it. Unfortunately for you guys, I am not able to reproduce this issue with test.sh but I'd be interested in any speculation anyone may have.

Churn

Note that \e[...m codes are specific to VT100/VT200/etc. and may not be the ones actually outputted by the program on a different type of $TERM. – Ewell

Answer 1

Consider using the pee program discussed in Is it possible to distribute stdin over parallel processes. It would allow you to send the log data through your sed script, while continuing to send the colours to the actual output.

One major advantage of this is that it would remove the 'execute sed once per line of log output'; that is really diabolical for performance (in terms of number of processes executed, if nothing else).

Shirlshirlee

excellent! thank you! exec 4<&1 5<&2 1>&2>&>(tee -a >(sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' > $LOG_FILE)) – Churn

Answer 2

I know it's not a perfect solution, but cat -v will make non visible chars like \x1B to be converted into visible form like ^[[1;34m. The output will be messy, but it will be ascii text at least.

Barrows

Answer 3

I use to do stuff like this by setting TERM=dumb before running my command. That pretty much removed any control characters except for tab, CR, and LF. I have no idea if this works for your situation, but it's worth a try. The problem is that you won't see color encodings on your terminal either since it's a dumb terminal.

You can also try either vis or cat (especially the -v parameter) and see if these do something for you. You'd simply put them in your pipeline like this:

exec 4<&1 5<&2 1>&2>&>(tee -a | cat -v | $LOG_FILE)

By the way, almost all terminal programs have an option to capture the input, and most clean it up for you. What platform are you on, and what type of terminal program are you using?

Montiel

Answer 4

You could attempt to use the -n option for read. It reads in n characters instead of waiting for a new line. You could set it to one. This would increase the number of iteration the code runs, but it would not wait for newlines.

From the man:

-n NCHARS read returns after reading NCHARS characters rather than waiting for a complete line of input.

Note: I have not tested this

Giese

Answer 5

You can use ANSIFilter to strip or transform console output with ANSI escape sequences.

See http://www.andre-simon.de/zip/download.html#ansifilter

Broca

Answer 6

Might not screen -L or the script commands be viable options instead of this exec loop?

Tonedeaf

can we turn off control characters with screen? – Escallop