Redirect output to different directories for sun grid engine array jobs

Question

I'm running a lot of jobs with Sun Grid Engine. Since these are a jobs (~100000), I would like to use array jobs, which seems to be easier on the queue.

Another problem is that each jobs produces an stdout and stderr file, which I need to track error. If I define them in the qsub -t 1-100000 -o outputdir -e errordir I will end up having directories with 100000 files in them, which is too much.

Is there a way to have each job write the output file to a directory (say, a directory which consists of the first 2 characters of the job ID, which is random hex letters; or the job number modulu 1000, or something of that sort).

Thanks

Deathlike

Answer 1

I can't think of a good way to do this with qsub as there are no programmatic interfaces into the -o and -e options. There is, however, a way to accomplish what you want.

Run your qsub with -o and -e pointing to /dev/null. Make the command you run be some type of wrapper that redirects it's own stdout and stderr to files in whatever fashion you want (i.e., your broken down directory structure) before it execs the real job.

Guillen

You don't even need a wrapper. Assuming you're doing this via a jobscript, just append the command to run your program with &>> logs/xyz.log. You could even use environment variables like JOB_ID and SGE_TASK_ID to build the filename, and Bash supports basic arithmetic for the modulo. – Aventine