run a python script in qsub

Question

I have a python script main_script.py that looks like this:

import os

Files = os.listdir(os.path.join(path, "."))
FilesNumber = len(Files)

for fileID in range (0,FilesNumber):
    filename = Files[fileID]

    # load file specified in filename and do stuff

basically it does the same kind of operations for each file in the variable Files

I would like to use qsub to parallelize the for loop.

Assuming that I have a txt file files.txt containing all the files names:

//mypath//pathfile1
//mypath//pathfile2
...
//mypath//pathfile100

how can I write the shell script that calls qsub and runs main_script.py I think that I would also need to adapt main_script.py but I do not know how...

The scheduler is Torque/Maui

Penumbra

Which scheduler are you using? The command qsub is used by several of them. – Bield

I should use Torque/Maui – Penumbra

Personal plug on this one: you can use fyrd (fyrd.science) to submit functions to torque or slurm from python, it is built for exactly this kind of thing. – Peony

Answer 1

One way to call any executable from a job script is to simply wrap it inside a bash script:

#/bin/bash

<full path to call executable>

If you name that script script.sh, and script.sh is executable, then you can execute:

qsub script.sh

and it will be submitted to the batch system. The gotchas - which you may well already know - are things like: if your executable isn't accessible from the compute node, then it won't be found when the job executes. The same is true for files that your script is using, so you'll want to make sure they're all located appropriately, usually a network-accessible filesystem.

If you wanted to directly submit the python script, you can add:

#!/usr/bin/python 

to the top (double-check that python is in /usr/bin on your system) and then you can directly qsub your python script. In your case,

qsub main_script.py

When submitted this way, the script no longer has to be in a network-accessible location, but the input files still do.

Romanic