Using tee and grep to save certain lines of output to a file

Question

I want to see the entire output of my script but only save lines matching "vlim" to a new file. I've nearly figured it out but can't quite get it to work the way I want it to. I'm currently stuck between using:

python gmakemovie.test movie.cfg.test --overwrite | tee >(grep vlim) /output.txt
grep vlim output.txt > vlim.txt

or

python gmakemovie.test movie.cfg.test --overwrite | grep vlim | tee  /output.txt

The top option shows my entire output but also copies all of the output to output.txt. The bottom option copies only "vlim" but doesn't show the rest of my output, so I can't tell where I am in my script.

For clarity, this is what my output looks like:

imported pygad v0.4.32+ga1eacb4.dirty
from parameter file (fragmentary):
  snaps:      200 - 200
  width:      1000.0 [kpc]
  pixel:      500 x 500
  x - y:      x - y
  softening:  [  0.2    0.45   2.52  20.     0.2    0.2 ] [ckpc h_0**-1]
====================================================
get orientation from trace file "filepath":
  L:       [ 241.01309204  544.96875    -366.44366455] [1e+10 ckpc h_0**-1      Msol h_0**-1 km s**-1]
get first non-zero center from trace files
  from snapshot 200:
  center:  [ 36821.75390625  35120.65625     33730.06640625] [ckpc h_0**-1]
====================================================
run in serial
====================================================
read snapshot "filepath"
  z = 2.84615386294
  t = 2.33634681236 [Gyr]
get center and orientation from trace file "filepath":
  center:  [ 36821.75390625  35120.65625     33730.06640625] [ckpc h_0**-1]
  R200:    47.4815177362 [kpc]
center snapshot
orientate snapshot
render...
  axis 1: stars...
  im_lum = np.log10(im_lum)
vlim: [-6.59883562 -4.09629962]
  np.ndarray.__idiv__(self, x)
  axis 2: gas...
vlim: [-0.46031536  0.83438775]

and I want the output file to look like:

vlim: [-6.59883562 -4.09629962]
vlim: [-0.46031536  0.83438775]

I'm working in the terminal on my mac

Kisner

Answer 1

It woud be easiest to just

$ datagenerator | tee outfile-complete | grep 'pattern' >outfile-filtered &
$ less outfile-complete

If it's a long-running script, this will allow you to monitor the progress with less (use F in less to get it to act like tail -f) while the job is running as a background job.

EDIT: Actually, looking closer at your first command:

$ datagenerator | tee >( grep 'pattern' ) output

Only a tiny change is necessary to do what you intended:

$ datagenerator | tee >( grep 'pattern' >outfile-filtered ) output-complete

The grep in the sub-shell was writing to standard output. The change means that the filtered data goes to a file instead, and the complete output goes to the console and to output-complete

So now you have two solutions that does slightly different things.

Waspish

sorry if this is obvious, so I would set it up like: python gmakemovie movie.cfg.1 --overwrite | tee output_complete.txt | grep vlim output_filtered.txt and the grep would go through the complete.txt to find vlim? – Kisner

No, grep would not look at the complete output from the file, but from the standard output from tee. – Waspish

alright I must have done something wrong because all that did was clear both gmakemove.test and movie.cfg.test, produce two blank files (output_.txt), and added '--overwrite' and 'grep' to the directory I was working in.. – Kisner

Um... yes, that's wrong. I see now that you said "grep vlim output_filtered.txt" in your first comment. That should be grep vlim >output_filtered.txt. But none of that should have created any files called grep or otherwise. – Waspish

Phew, I was worried there for a while that I made a horrible mistake, so I re-ran all my tests three times to make sure, but all looked good on my side ;-) Good! – Waspish