How to import a local python module when using the sbatch command in SLURM

Question

I was using the cluster manager slurm and I was running a submission script with sbatch (with a python interpeter). The sbatch submission imported one of my modules called main_nn.py. The module is located in the same place as my submission directory, however, python fails to find it even though the file exists. I am having a hard time figuring it out why this is happening. My python file looks as follow:

#!/usr/bin/env python
#SBATCH --job-name=Python

print('hi')

import main_nn

however the output of my slurm dump file is:

hi
Traceback (most recent call last):
    File "/home/slurm/slurmd/job3223398/slurm_script", line6, in <module>
        import main_nn
ImportError: No module named main_nn

I tried checking if the module main_nn was in the current directory and it was there indeed. Thus, the first thing that seemed suspicious to me was that the error in the slurm file said the location of my script was at "/home/slurm/slurmd/job3223398/slurm_script" rather than at path_to_project. Thus I went ahead an added the line

os.system('pwd')

to see where my script was executing from and to my surprise it was executing at path_to_project rather than at "/home/slurm/slurmd/job3223398/slurm_script" which must mean that sbatch is doing something funky to executed a script at one location but make it think its at another. If this is the case how am I suppose to do an import in python where the module is in the same location as in my submission script? Am I forced to put it in a package and trick python to think its in a package/library?

Idiocrasy

rcc.uchicago.edu/docs/tutorials/kicp-tutorials/… – Idiocrasy

have you tried setting PYTHONPATH to your submission directory? – Vogeley

related: #46718999 – Idiocrasy

Answer 1

As Slurm copies the submission script to a specific location on the compute node to run it, your Python script will not find the modules that are in the submission directory.

But Slurm correctly sets the current working directory so you can explicitly add it to the python path with something like:

sys.path.append(os.getcwd()) 

near the beginning of your script.

Vogeley

I don't get it, why doesn't slurm just do this by itself or something? – Idiocrasy

This simply might have been overlooked by de development team. – Vogeley

is there a way to raise awareness to this issue to them? Seems to me its the type of issue that shouldn't even exist. – Idiocrasy