Variable expansion in comments

Question

Is it possible to expand variables in comments inside a bash script?

I want to write a script to feed into SGE. The qsub syntax allows me to pass additional parameters to the grid engine using lines inside the bash script which begin with #$. For example,

#$ -q all.q
#$ -S /bin/bash
#$ -V
#$ -m beas
#$ -o run_20120103.out

What I want is that the -o parameter is dynamically set to a variable, say $1. So I would naively write

#$ -o run_${1}.out

However, since the line starts with a #, bash ignores it and the variable $1 is not expanded.

Any ideas? Some bash preprocessor? Some other way?

EDIT I just chose $1 as an example. It could just as well be $FOO or $BAR.

Evangelina

What is determining the value of ${1} ? Are you doing a for cycle to load different jobs on the cluster/grid? – Lessielessing

In that case you can simply call qsub with the option -o inside your loop instead of putting #$ -o inside the script fed into qsub. – Lessielessing

Answer 1

Variable expansion happens in shell memory, it doesn't affect the file. Therefore, it doesn't matter what bash expands.

Instead, you can probably generate the script to be run on the fly, with everything expanded in place:

cat << EOF | qsub [options] -
#$ -o run_$1.out
cmds
EOF

Aneroidograph

Can you explain exactly what this script does step by step? I tried testing it but as I have no idea what the behavior should be and I am not getting the behavior I expect it's quite useless – Copra

@Copra This script is not a complete or self contained example. It merely illustrates a syntactic structure for expanding variables in text before feeding it to qsub. – Aneroidograph

Could you expand it to be? I think this might be what I am looking for to expand a variable walltime for my jobs. There is no option like -o for walltime in qsub is there? That would make it so much easier :p Edit: I think I found it, -l can also be used as option to qsub, where I think I can pass the walltime. Testing now – Copra

Unfortunately I only know bash, not qsub. If you know how to invoke qsub and what the input should be, I'll be happy to help out with the bash required to make that happen. – Aneroidograph

The input for qsub is a bash job script :) – Copra

Well, if you know how to invoke qsub and what that bash job script should look like... – Aneroidograph

Answer 2

To be a bit more SGE specific than the accepted answer, just redirect the standard output of qsub and parse the job id. You can then use this to alter the job name via qalter.

Here's a sample of a script I used to submit and modify jobs like this.

 for a in $args;
 do 
    qsub submission_script $a 1> job.txt
    jid=$(cat job.txt | tr -dc '0-9' | cut -c -6)
    qalter $jid -N $a
 done

I'm certain that there are more idiomatic ways to do this, but this is a first go.

This approach allows submission_script to have it's own arguments. It also allows you to programmatically alter other job characteristics, not just the name.

Mallory

Answer 3

I am also using the qsub command in SGE to generate different output or error logs. For example, if I would like to include the TASK_ID in the script, instead of doing

#$ -o run_${TASK_ID}.out

Write it without the paretheses will do the trick:

#$ -o run_$TASK_ID.out

Fullerton

No, it doesn't do the trick. You get a file called run_$TASK_ID.out, the variable hasn't be replaced by its value – Leaden