Transition from PBSPro to Slurm (rzcluster)
Studierende | Beschäftigte | Einrichtungen | stud. Gruppen |
---|
Dear rzcluster users,
as of 7th December 2016, we will deploy a new batch system - Slurm - on the rzcluster, which means that all rzcluster users need to adapt their batch scripts accordingly to be able to process batch jobs after this date. From user perspective, the transition from PBSPro to Slurm should be a rather straightforward process, however it goes along with some new environmental variables, commands and job parameters.
To assist users, we give below a summary of the most important changes.
If you encounter problems during the conversion of your scripts or have further questions, please do not hesitate to contact us directly or to send a message to our ticket system: hpcsupport@rz.uni-kiel.de.
Best regards,
your HPC-Support Team
PS, for a full documentation of Slurm you may visit the official Slurm website.
PBSPro vs Slurm: environment variables
Environment variable | PBSPro | Slurm |
---|---|---|
Job ID | $PBS_JOBID |
$SLURM_JOBID |
Job name | $PBS_JOBNAME |
$SLURM_JOB_NAME |
Directory from which the job has been submitted | $PBS_O_WORKDIR |
$SLURM_SUBMIT_DIR |
Node list | $PBS_NODELIST |
$SLURM_NODELIST |
- Note, that in Slurm it is not necessary to explicitly change into the directory named
$SLURM_SUBMIT_DIR
, as jobs will automatically use the directory from which they have been submitted. - During job submission in Slurm, all environment variables which have been defined (so far) will be transfered to the batch job. While this is convenient at first glance, it can lead to the fact that the same job script behaves differently depending on the environment variables set. For this reason it is advisable to set the following directive:
#SBATCH --export=NONE
.
PBSPro vs Slurm: commands
Command | PBSPro | Slurm |
---|---|---|
Submit a batch job | qsub <jobscript> |
sbatch <jobscript> |
Delete or terminate a batch job | qdel <jobid> |
scancel <jobid> |
List all jobs in the system | qstat |
squeue |
List own jobs | qstat -u <userid> |
squeue -u <userid> |
Show status of a specific job | qstat <jobid> |
squeue -j <jobid> |
Show details of a specific job | qstat -f <jobid> |
scontrol show job <jobid> |
- To start an interactive run equivalent to
qsub -I
, please see the examples on the rzcluster page (under topic: batch processing). - Note, that in Slurm all output is written to its destination already during job execution. Thus, a command (equivalent to
qpeek <jobid>
) in order to monitor the job's stdout progress is obsolete. - Detailed resource information about a running job is available in Slurm via the command
sstat -j <jobid>
.
PBSPro vs Slurm: job parameters
- Like in PBSPro, a Slurm batch script begins with the
#!/bin/bash
directive on the first line. The subsequent lines then contain the directive#SBATCH
followed by a resource request or other job information. For specific examples and templates see the next section.
Job parameter | PBSPro | Slurm |
---|---|---|
Batch script directive | #PBS |
#SBATCH |
Queue/partition | -q=<queue> |
--partition=<partition> or -p <partition> |
Job name | -N=<jobname> |
--job-name=<jobname> or -J <jobname> |
Stdout file | -o=<filename> |
--output=<filename> or -o <filename> |
Stderr file | -e=<filename> |
--error=<filename> or -e <filename> |
Stdout and stderr into the same file | -j oe |
--output=<filename> or -o <filename> (default if no --error is specified) |
Number of nodes | -l select=<n> |
--nodes=<n> or -N <n> |
Number of (MPI) processes per node | -l mpiprocs=<m> |
--tasks-per-node=<m> |
Number of CPUs per node or task | -l ncpus=<m> |
--cpus-per-task=<m> or -c <m> |
Main memory (in MB) | -l mem=<memory> |
--mem=<memory> |
... in GB | --mem=1G |
|
Walltime | -l walltime=[hh:mm:ss] |
--time=[hh:mm:ss] or -t [hh:mm:ss] |
Never requeue the job | -r n |
--no-requeue |
Email address | -M <address> |
--mail-user=<address> |
Email notifications | -m b |
--mail-type=BEGIN |
-m e |
--mail-type=END |
|
-m a |
--mail-type=FAIL |
|
-m abe |
--mail-type=ALL |
Slurm script templates
- Example for a serial calculation:
#!/bin/bash #SBATCH --job-name=test #SBATCH --output=test.out #SBATCH --error=test.err #SBATCH --nodes=1 #SBATCH --tasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=1000 #SBATCH --time=01:00:00 #SBATCH --partition=small export OMP_NUM_THREADS=1 ./test.ex
- Example for a parallel multi-node MPI calculation:
#!/bin/bash #SBATCH --job-name=test #SBATCH --output=test.out #SBATCH --error=test.err #SBATCH --nodes=2 #SBATCH --tasks-per-node=12 #SBATCH --cpus-per-task=1 #SBATCH --mem=10000 #SBATCH --time=01:00:00 #SBATCH --partition=small export OMP_NUM_THREADS=1 module load intelmpi16.0.0 mpirun -np 24 ./test.ex
- Example for a parallel multi-node hybrid (MPI+OpenMP) calculation:
#!/bin/bash #SBATCH --job-name=test #SBATCH --output=test.out #SBATCH --error=test.err #SBATCH --nodes=2 #SBATCH --tasks-per-node=6 #SBATCH --cpus-per-task=2 #SBATCH --mem=10000 #SBATCH --time=01:00:00 #SBATCH --partition=small export OMP_NUM_THREADS=2 module load intelmpi16.0.0 mpirun -np 12 ./test.ex
- Example for a job array:
#!/bin/bash #SBATCH --job-name=test #SBATCH --array 0-9 #SBATCH --output test-%A_%a.out #SBATCH --nodes=1 #SBATCH --tasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=1000 #SBATCH --time=00:10:00 #SBATCH --partition=small echo "Hi, I am step $SLURM_ARRAY_TASK_ID in the array job $SLURM_ARRAY_JOB_ID"
Support and Consulting
HPC-Support-Team: hpcsupport@rz.uni-kiel.de
Responsible contact persons at the Computing Centre:
Please see HPC-Support and Consulting.