How does one make sure that the python submission script in slurm is in the location from where the sbatch command was given?

Question

I have a python submission script that I run with sbatch using slurm:

sbatch batch.py

when I do this things do not work properly because I assume, the batch.py process does not inherit the right environment variables. Thus instead of running batch.py from where the sbatch command was done, its ran from somewhere else (/ I believe). I have managed to fix this by doing wrapping the python script with a bash script:

#!/usr/bin/env bash
cd path/to/scripts
python script.py

this temporary hack sort of works it seems though it seems that it avoids the question all together rather than addressing it. Does someone know how to fix this in a better way?

I know for example, that in docker the -w or -WORKDIR exists so that the docker container knows where its suppose to be at. I was wondering if something like that existed for slurm.

Neel

maybe: --workdir=<directory> Set the working directory of the batch script to directory before it is executed. The path can be specified as full path or relative path to the directory where the command is executed. – Neel

Have you given sbatch --get-user-env a look? It might be what you need. Otherwise if the environment variables are only added in your current terminal you might want to use sbatch --export to pass the variables you need explicitly. – Dunlop

not sure what it means to give --get-user-env a look, you mean take a look at the flag or is it a environment variable? – Neel

might be helpful: #39241532 – Neel

Answer 1

Slurm is designed to push the user's environment at submit time to the job, except for variables explicitly disabled by the user or the system administrator.

But the way the script is run is as follows: the script is copied on the master node of the allocation in a Slurm specific directory and run from there, with the $PWD set to the directory where the sbatch command was run.

You can see that with a simple script like this one:

$ cat t.sh
#!/bin/bash
#
#SBATCH --job-name=test_ms
#SBATCH --output=res_ms.txt

echo $PWD
dirname $(readlink -f "$0")

$ sbatch t.sh
Submitted batch job 1109631
$ cat res_ms.txt
/home/damienfrancois/
/var/spool/slurm/job1109631

One consequence is that Python scripts that import modules in the current directory fail to do so. The workaround is then to explicitly add sys.path.append(os.getcwd()) before the failing imports.

Amygdalin

ok, so according to you echoing $CWD in a bash submission script should give me something...but its currently giving me an empty .out file. – Neel

Sorry, not $CWD but $PWD. I corrected and update my answer – Amygdalin