I wanted to run a python script main.py
multiple times with different arguments through a sbatch_run.sh
script as in:
#!/bin/bash
#SBATCH --job-name=sbatch_run
#SBATCH --array=1-1000
#SBATCH --exclude=node047
arg1=10 #arg to be change during runs
arg2=12 #arg to be change during runs
python main.py $arg1 $arg2
The arguments are encoded in the bash file ran by sbatch. I was worried that if I ran sbatch_run.sh
multiple times one after the other but changing the value of arg1 and arg2 during each run, that it might cause errors in my runs. For example if I do:
sbatch sbatch_run.sh # with arg1=10 and arg2=12
and then immediately after I change sbatch_run.sh
but run the file again as in:
sbatch sbatch_run.sh # with arg1=69 and arg2=666
would case my runs to all run with the last one (i.e. arg1=69
and arg2=666
) instead of each run with its own arguments.
I know for sure that if I hard code the arguments in main.py and then run the same sbatch script but change the main.py it will run the last one. I was wondering if that is the case too if I change the sbatch_run.sh
script.
Just so you know, I did try this experiment, by running 1000 scripts, then some get queued and put a sleep command and then change the sbatch_run.sh
. It seems to not change what my run is, however, if I am wrong this is way too important to be wrong by accident and wanted to make sure I asked too.
For the record I ran:
#!/bin/bash
#SBATCH --job-name=ECHO
#SBATCH --array=1-1000
#SBATCH --exclude=node047
sleep 15
echo helloworld
echo 5
and then change the echo to echo 10 or echo byebyeworld.
main.py
example), make sure to have somewhere where the arguments don't change. For example, passing them directly in the bash script ran by slurm or a config file for each run, just make sure the correct config file is being ran and that you don't change it accidentally! – Constriction