how to specify error log file and output file in qsub

Question

I have a qsub script as

#####----submit_job.sh---#####
    #!/bin/sh
    #$ -N job1
    #$ -t 1-100
    #$ -cwd
    SEEDFILE=/home/user1/data1
    SEED=$(sed -n -e "$SGE_TASK_ID p" $SEEDFILE)
    /home/user1/run.sh $SEED 

The problem is-- it puts all error and output files (job1.eJOBID & job1.oJOBID) to the same directory from where I am running qsub submit_job.sh while I want to save these file (output and error log file in same different place (specified as $SEED_output). I tried to change the line as

/home/user1/run.sh $SEED -o $SEED_output

But it didn't work. Any suggestion ?? How can I specify path and name of default output and error log file ??

Ritualist

I had in slurm #SBATCH --output="demo.%j.%N.out" #SBATCH --error="demo.%j.%N.err", how do I adapt it to qsub? – Maltese

related: community.openpbs.org/t/… How to include the job id and other info in the output file for qsub? – Maltese

Answer 1

Typically error and output files are given as pbs directives in the qsub script or as command line options to the qsub script like so:

#! /bin/bash
#PBS -q queue_name
#PBS -A account_name
#PBS -l nodes=12:ppn=12
#PBS -l walltime=18:00:00
#PBS -e /mypath/error.txt
#PBS -o /mypath/output.txt

or as a command line option to qsub like so:

qsub -o /mypath/output.txt -e /mypath/error.txt submit_job.sh

With the first option I don't think you can use a variable as the shell won't look at lines that are commented. Plus I think PBS deals with the commented lines before the shell would. If you know the path when you're invoking qsub, you could try the second option. Alternatively you could try simply redirecting the output and error in the script itself:

/home/user1/run.sh $SEED > ${SEED}/output.txt 2> ${SEED}/error.txt

The third option is probably the easiest. Output and error files might still be created in the run directory, though they'd likely be empty.

Vaccinia

I can use PBS variables like $PBS_JOBID in the #PBS -o line. – Janik

@Janik perhaps it depends on the version of the tools? I can't use PBS variables in the #PBS -o line (the variables aren't resolved). – Puttee

I had in slurm #SBATCH --output="demo.%j.%N.out" #SBATCH --error="demo.%j.%N.err", how do I adapt it to qsub? – Maltese

what if you want to include the jobid number and other info like that? – Maltese

The third option does indeed create an empty output file in the run directory; to deal with this I just passed #PBS -o /dev/null. – Skaggs

Answer 2

At first glance, you need brackets around your variable in the -o declaration.

/home/user1/run.sh $SEED -o ${SEED}_output

Otherwise bash is looking for a variable called ${SEED_output} which doesn't exist.

Vaccinia

Thanks for mentioning that, but I still have no luck :( I tried /home/user1/run.sh $SEED -o ${SEED}_output and nothing changed. Just to make things clear $SEED (or ${SEED}) is path to a directory. So i think output should save in file _output in the directory ${SEED} !!! – Ritualist

-o is passed to run.sh, so that won't work unless run.sh handles -o as a command line argument. – Vaccinia

I had in slurm #SBATCH --output="demo.%j.%N.out" #SBATCH --error="demo.%j.%N.err", how do I adapt it to qsub? – Maltese

what if you want to include the jobid number and other info like that? – Maltese

Answer 3

I ran into the same problem. And my workaround was to have:

#$ -N script-name
#$ -wd <path-to-project>/log 
cd ..

in the qsub script. Now the error and output file are saved in <path-to-project>/log and the script works in <path-to-project> At least for me that worked

Deadwood