Redirect stderr through grep -v in LSF batch job

Question

I'm using a library that generates a whole ton of output to stderr (and there is really no way to suppress the output directly in the code; it is ROOT's Minuit2 minimizer which is known for not having a way to suppress the output). I'm running batch jobs through the LSF system, and the error output files are so big that they exceed my disk quota. Erk.

When I run locally on a shell, I do:

python main.py 2> >( grep -v Minuit2 2>&1 )

to suppress the output, as is done here. This works great, but unfortunately I can't seem to get that or any variation of it to work when running on LSF. I think this is due to LSF not spawning the necessary subshell, but it's not clear.

I run on batch by passing LSF a submit script. The relevant line is:

python main.py $INPUT_FILE

which works great, aside from the aforementioned problem of gigantic error files.

When I try changing that line to

python main.py $INPUT_FILE 2> >( grep -v Minuit2 2>&1 )

I end up with

./singleSubmit.sh: line 16: syntax error near unexpected token `>'
./singleSubmit.sh: line 16: `python $MAIN $1 2> >( grep -v Minuit2 2>&1 )'

in the error log file.

Any idea how I could accomplish what I want, or why this is not working?

Thanks a ton!

Villainy

Oh, by the way, I checked the shell that LSF runs these commands in. It is tcsh, just the same as I use locally. – Villainy

Answer 1

The syntax you're using works in bash, not in csh/tcsh. Try changing the first line of your submission script to

#!/bin/bash

Pelton

Huh. I thought I tried that. Thank you so much for the help!! :) – Villainy

Out of curiosity, why does python main.py $INPUT_FILE 2> >( grep -v Minuit2 2>&1 ) work in interactive tcsh? Shouldn't I get the same error? – Villainy

That's a good question, because I don't think it should. I'm running tcsh interactively and can't get it to work: the error message I get is "Missing name for redirect" – Pelton

Yeah, turns out I was just being silly - I was actually running interactively in zsh. I get the same result as you for tcsh. Cheers! – Villainy