Why use Slurm?¶
Within the SCG cluster, Slurm functions as a resource broker. Submitting a job to Slurm requests a set of CPU and memory resources. Slurm orders these requests and gives them a priority based on the cluster configuration and runs each job on the most appropriate available resource in the order that respects the job priority or, when possible, squeezes in short jobs via a backfill scheduler to harvest unused cpu time. Using Slurm allows many users to fairly share a set of computational resources with greatly reduced danger of negatively impacting anotehr persons jobs or work while also allowing the resources to be better and more fully utilized over time.
In addition to making it easier to effectively share a resource, Slurm also acts as a powerful tool for managing workflows. By encapsulating steps in job scripts it becomes possible to easily repeat and reuse workflows and having such job scripts adds to the documentation of the associated process.
What is a job script?¶
A job script is a normal unix shell script which optionally contains lines flagged such that the Slurm submission process can read arguments from them. A simple hello world job script using Bash would be
#!/bin/bash
# See `man sbatch` or https://slurm.schedmd.com/sbatch.html for descriptions
# of sbatch options.
#SBATCH --job-name=hello_world
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=interactive
#SBATCH --account=default
#SBATCH --time=1:00:00
echo 'Hello World!'Partition info¶
| Partition | Max mem | Max CPUs | Time limit |
|---|---|---|---|
| batch | - | - | 14 Days |
| nih_s10 | 97GB | 344 | 7 Days |
| interactive | 128GB | 16 | 24 Hours |
Submitting a job¶
Saving the script as hello_world.sh and submitting it to Slurm with
sbatch results in the job running
and producing an output file. The default output file is
slurm-JOB_ID.out located in the directory from which the job was submitted.
For example:
[griznog@smsx10srw-srcf-d15-37 jobs]$ sbatch hello_world.sh
Submitted batch job 6592914
[griznog@smsx10srw-srcf-d15-37 jobs]$ cat slurm-6592914.out
Hello World!The sbatch man page lists all sbatch options.
Managing Slurm Jobs¶
squeue¶
Once a job is submitted, it either immediately runs if resources are available
and there are no jobs ahead of it in the queue or it is queued and marked as
Pending. Pending and running jobs can be monitored with the
squeue command.
The default squeue output shows all jobs:
[griznog@smsx10srw-srcf-d15-37 jobs]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
6564781 interacti wrap crolle PD 0:00 1 (Resources)
6564782 interacti wrap crolle PD 0:00 1 (Priority)
6564783 interacti wrap crolle PD 0:00 1 (Priority)
6564784 interacti wrap crolle PD 0:00 1 (Priority)
6564785 interacti wrap crolle PD 0:00 1 (Priority)
6564786 interacti wrap crolle PD 0:00 1 (Priority)
6564787 interacti wrap crolle PD 0:00 1 (Priority)
6564788 interacti wrap crolle PD 0:00 1 (Priority)
6564789 interacti wrap crolle PD 0:00 1 (Priority)
6564790 interacti wrap crolle PD 0:00 1 (Priority)
6564791 interacti wrap crolle PD 0:00 1 (Priority)
6564792 interacti wrap crolle PD 0:00 1 (Priority)
6592902 dtn seqctr_s root PD 0:00 1 (BeginTime)
6564793 interacti wrap crolle PD 0:00 1 (Priority)
6564794 interacti wrap crolle PD 0:00 1 (Priority)
6564795 interacti wrap crolle PD 0:00 1 (Priority)
...Some useful squeue commands are:
| Command | Description |
|---|---|
squeue -u $USER |
Show only jobs owned by $USER. |
squeue -t pd |
Show only pending jobs. -t r to show only running jobs’ |
squeue -j JOBID |
Show details for job JOBID. |
squeue -j JOBID --format="%m" |
Use --format to show only the memory requested. |
scancel¶
The simplest method of cancelling a running job or job array task is to use scancel. For example:
[griznog@smsx10srw-srcf-d15-37 jobs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
6593409 interacti wrap griznog PD 0:00 1 (Priority)
[griznog@smsx10srw-srcf-d15-37 jobs]$ scancel 6593409
[griznog@smsx10srw-srcf-d15-37 jobs]$ squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
[griznog@smsx10srw-srcf-d15-37 jobs]$ The scancel man page lists the
options to scancel, many of which
are useful for selecting subsets of jobs to operate on.
scancel can also be used to send
specific signals to a jobs
processes. This can be useful for more advanced jobs which are designed to
perform different functions in response to receiving a signal, for instance,
applications that can perform a checkpoint might be triggered to do so manually
with a signal sent via scancel.
sinfo¶
The sinfo command can be used to list available partitions, show status and list node and partition configurations. To list partitions:
[griznog@smsx10srw-srcf-d15-37 jobs]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up 120-00:00: 22 mix dper730xd-srcf-d16-[01,03,05,07,09,11,13,15],dper7425-srcf-d15-13,sgisummit-frcf-111-[08,12,14,16,18,20,24,26,28,34,36,38],sgiuv20-rcf-111-32
batch* up 120-00:00: 3 alloc sgisummit-frcf-111-[10,22,30]
batch* up 120-00:00: 24 idle dper730xd-srcf-d16-[17,19,21,23,25,27,29,31,33,35,37,39],dper930-srcf-d15-05,dper7425-srcf-d15-[09,11,15,17,19,21,23,25,27,29,31]
interactive up 120-00:00: 2 drain* hppsl230s-rcf-412-01-l,hppsl230s-rcf-412-02-l
interactive up 120-00:00: 5 mix dper910-rcf-412-20,hppsl230s-rcf-412-01-r,hppsl230s-rcf-412-02-r,hppsl230s-rcf-412-03-l,hppsl230s-rcf-412-03-r
interactive up 120-00:00: 18 idle hppsl230s-rcf-412-04-l,hppsl230s-rcf-412-04-r,hppsl230s-rcf-412-05-l,hppsl230s-rcf-412-05-r,hppsl230s-rcf-412-06-l,hppsl230s-rcf-412-06-r,hppsl230s-rcf-412-07-l,hppsl230s-rcf-412-07-r,hppsl230s-rcf-412-08-l,hppsl230s-rcf-412-08-r,hppsl230s-rcf-412-09-l,hppsl230s-rcf-412-09-r,hppsl230s-rcf-412-10-l,hppsl230s-rcf-412-10-r,hppsl230s-rcf-412-11-l,hppsl230s-rcf-412-11-r,hppsl230s-rcf-412-12-l,hppsl230s-rcf-412-12-r
nih_s10 up 4-00:00:00 1 mix sgiuv300-srcf-d10-01
nih_s10_gpu up 4-00:00:00 1 mix sgiuv300-srcf-d10-01
dtn up 120-00:00: 2 idle cfxs2600gz-rcf-114-[06,08]
apps up 120-00:00: 1 idle dper7425-srcf-d10-37sacct¶
The sacct command can be used to retrieve infomration about jobs that have completed. For example:
[griznog@smsx10srw-srcf-d15-37 jobs]$ sacct -j 6593409
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
6593409 wrap interacti+ default 1 CANCELLED+ 0:0
6593409.bat+ batch default 1 CANCELLED 0:15
6593409.ext+ extern default 1 COMPLETED 0:0 When troubleshooting jobs, the ExitCode can be useful to determine how the job ended as it shows the exit code of the job script and the signal which caused the process to terminate. For the example above, the job was cancelled and it’s script was killed by signal 15 with the job script returning 0.
The sacct command can return many details about jobs, consult the sacct man page for more information about options and values that can be retrieved for jobs.
More information¶
For more information about additional Slurm commands, see the Slurm documentation.