Running qsub with anaconda environment

Question

I have a program that usually runs inside a conda environmet in Linux, because I use it to manage my libraries, with this instructions:

source activate my_environment
python hello_world.py

How can I run hello_world.py in a high computer that works with PBS. Instructions explains to run adapting the code script.sh, shown below, and calling with the instruction qsub.

# script.sh
#!/bin/sh
#PBS -S /bin/sh
#PBS -N job_example
#PBS -l select=24
#PBS -j oe
cd $PBS_O_WORKDIR
mpiexec ./programa_mpi

How do I run hello_world.py with qsub using my anaconda environment?

Tymothy

Answer 1

You'll need to load the Python module before activating your environment and before running your script.

module load python3

cd $PBS_O_WORKDIR
source activate my_environment

mpiexec python hello_world.py

Check the documentation for your institution regarding their Python modules. At my institution, Anaconda was the environment module for Python3, so you could load it as I have shown.

Unpolitic

what do you mean by python module? – Clown

@Ghanem, I mean module load python3. On supercomputers, Python isn't enabled automatically, so you need to enable the version of Python or Matlab or whatever in your PBS script. Loading that module makes that version of Python your default when your Python script is run. – Unpolitic

Ok, @HS-nebula , could have a look here: https://stackoverflow.com/questions/56871054/force-shell-to-use-python-from-conda-variable-in-sungrid-engine , I have a similar issue. – Clown

@Ghanem can you reformat your comment so that it's a link? I can try, but I've never user SunGrid Engine. – Unpolitic

Answer 2

Unless it is in the environment by default, one also needs to "load" conda as well, inside the SGE(qsub) script (similar thing would hold for a slurm script too I assume). For example I had installed conda to the directory seen below in the SGE script, so I export the path (if conda is installed as a module on an HPCC, then simply load that instead):

#!/bin/bash
#$ -q compute
#$ -l compute
#$ -cwd
#$ -N name
#$ -j yes

export PATH=$HOME/miniconda3/bin:$PATH

source activate my_environment

environment function code...

Whitney

Answer 3

If your really need your conda environment you can activate it in the script

source ~/.bashrc #configures your shell to use conda activate
conda activate your_env_name

Cowslip

Answer 4

source ~/.bashrc

conda activate your_environment

cd $PBS_O_WORKDIR

Lyndes