How can I gzip standard in to a file and also print standard in to standard out?

Question

I want to execute a command, have the output of that command get gzip'd on the fly, and also echo/tee out the output of that command.

i.e., something like:

echo "hey hey, we're the monkees" | gzip --stdout > my_log.gz

Except when the line executes, I want to see this on standard out:

hey hey, we're the monkees

Vidovik

Answer 1

echo "hey hey, we're the monkees" | tee /dev/tty | gzip --stdout > my_log.gz

As pointed out in the comments, /dev/stdout might work better than /dev/tty in some circumstances.

Endurance

What's /dev/tty doing there? The original question wanted the output on standard output, not necessarily on the terminal. – Dominickdominie

/dev/tty is a synonym for the current terminal. The questioner used "standard out" in the customary manner to mean the the current terminal, rather than a more strict definition of the term. – Endurance

If it were indeed customary to use "standard output" to mean "current terminal", then that would be a custom likely to lead to much confusion! For this question, bash has /dev/stdout. – Dominickdominie

Under what circumstances would /dev/tty not produce the desired outcome where /dev/stdout would? – Endurance

When stdout goes to a file and not to the terminal. – Dominickdominie

This is just plain not what was asked -- what was asked for was a shell pipeline that would send the output to its stdout AND gzip it to a file. (What if I wanted to send the output on to yet another program -- say, less(1). – Growl

/dev/stdout didn't work on Solaris 10 using KSH, but /dev/tty did. – Electroscope

/dev/stdout will not work in any cases as it is applied after the |. – Petronille

@GarethRees you are right but practicality beats purity if you python -m this. you could serve the public with pointing it out in an answer - as opposed to editing the question, which latter would hide the fact that the customary is, arguably, commonplace. – Concomitant

Answer 2

Another way (assuming a shell like bash or zsh):

echo "hey hey, we're the monkees" | tee >(gzip --stdout > my_log.gz)

The admittedly strange >() syntax basically does the following:

• Create new FIFO (usually something in /tmp/)
• Execute command inside () and bind the FIFO to stdin on that subcommand
• Return FIFO filename to command line.

What tee ends up seeing, then, is something like:

tee /tmp/arjhaiX4

All gzip sees is its standard input.

For Bash, see man bash for details. It's in the section on redirection. For Zsh, see man zshexpn under the heading "Process Substitution."

As far as I can tell, the Korn Shell, variants of the classic Bourne Shell (including ash and dash), and the C Shell don't support this syntax.

Kef

Cool, I'm learning something myself here. Can you elaborate on what is happening there? – Chao

if I read it right, instead of given a file to tee, you've got it sending the copy as input to bracketed expression which writes the gzip output to another file. The uncompressed data leaves tee as normal on stdout – Chao

thanks for the edit, appreciate the extra info! Never knew about that FIFO syntax. – Chao

It came as a shock to me when I first learned it. But it lets you get away with very complex redirections, with several programs operating on the same input simultaneously. Of course, you can also do mkfifo by hand and run all of these commands in different consoles if you need to. – Kef

I've been doing Unix for 20 years, and I haven't wrapped my head around this new syntax yet. – Endurance

I'm using this for logging. I've found gzip buffers a lot and when I ctrl-c I lose everything. Ignoring SIGINT with >(trap '' INT; gzip --stdout > my_log.gz) seems to save it. – Delineation

@Delineation Nice! Indeed it seems that gzip buffers 16KB of compressed data before writing – Chondro

Answer 3

Have a nice cup of tee!

The tee command copies standard input to standard output and also to any files given as arguments. This is useful when you want not only to send some data down a pipe, but also to save a copy

As I'm having a slow afternoon, here's some gloriously illustrative ascii-art...

           +-----+                   +---+                  +-----+  
stdin ->   |cmd 1|    -> stdout ->   |tee|   ->  stdout  -> |cmd 2|
           +-----+                   +---+                  +-----+
                                       |
                                       v
                                     file

As greyfade demonstrates in another answer the 'file' need not be a regular file, but could be FIFO letting you pipe that tee'd output into a third command.

           +-----+                   +---+                  +-----+  
stdin ->   |cmd 1|    -> stdout ->   |tee|   ->  stdout  -> |cmd 2|
           +-----+                   +---+                  +-----+
                                       |
                                       v
                                     FIFO
                                       |
                                       v
                                    +-----+
                                    |cmd 3|
                                    +-----+

Chao

But I want to gzip the intermediary file on the fly. Is that possible only using tee? – Vidovik

yes, the other paul wrote a nice succinct answer while I messed about with an ascii art diagram :) – Chao

:-P I'm familiar with tee'ing to a file like you're describing. I just don't know how to gzip that file. if i do "echo foo | tee bar.log", I don't know how to make tee gzip 'bar.log', aside from the solution Paul Tomblin posted. – Vidovik

man, stackoverflow is awesome. I didn't even have to harass the office linux guru. Thanks all. – Vidovik

greyfade's excellent answer shows how you can do what you like with the tee'd output. – Chao

Yes, grayfade's answer does precisely what this one suggests, but with a very succinct syntax. This ones cmd 3 is written >(cmd 3) there. – Growl

Answer 4

Just to post a way that doesn't involve touching disk:

echo "hey hey, we're the monkees" | (exec 1>&3 && tee /proc/self/fd/3 | gzip --stdout > my_log.gz)

Petronille

What, you mean /tmp is on a disk ?!?!?!? – Growl

From a security perspective, it is. – Petronille