How to find from where a job is submitted in SLURM?

Question

I submitted several jobs via SLURM to our school's HPC cluster. Because the shell scripts all have the same name, so the job names appear exactly the same. It looks like

[myUserName@rclogin06 ~]$ sacct -u myUserName
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
12577766         run.sh    general ourQueue_+          4    RUNNING      0:0 
12659777         run.sh    general ourQueue_+          8    RUNNING      0:0 
12675983         run.sh    general ourQueue_+         16    RUNNING      0:0 

How can I know from which directory a job is submitted so that I can differentiate the jobs?

Flossie

Answer 1

You can use the scontrol command to see the job details. $ scontrol show job <jobid>

For example, for a running job on our SLURM cluster:

$ scontrol show job 1665191
    JobId=1665191 Name=tasktest
    ...
    Shared=OK Contiguous=0 Licenses=(null) Network=(null)
    Command=/lustre/work/.../slurm_test/task.submit
    WorkDir=/lustre/work/.../slurm_test

You are looking for the last line, WorkDir.

Sweptwing

If you need the output log file, you can scontrol show job <JOBID> | grep StdOut. – Cochabamba

after you found the workdir, search for the job name or part of the job name in that dir. If it is sbatch file that has jobname in there you likely can find it quickly this way – Giulietta

Answer 2

The latest version of Slurm now offers that information through squeue with :

squeue --format "%Z"

that displays, according to the man page,

%Z    The job’s working directory.

Jacklight

Answer 3

In order to list the work directory of past jobs that are no longer accessible via squeue or scontrol, you can use sacct:

sacct -S 2020-08-10 -u myUserName --format "jobid,jobname%20,workdir%70"

Lists job id, job name and work directory of all jobs of user myUserName since August 10th, 2020.

Sanctus

I'm not seeing the workdir field as an option with sacct 2.6 on CentOS release 6.5 (Final) – Jeramey

@Jeramey Slurm 2.6 was released in 2013. Looks like the feature was added in slurm 17.11.0rc1 – Sanctus