Transition from PBSPro to Slurm (rzcluster)

no/neinStudierende yes/jaBeschäftigte yes/jaEinrichtungen no/neinstud. Gruppen

Dear rzcluster users,

as of 7th December 2016, we will deploy a new batch system - Slurm -  on the rzcluster, which means that all rzcluster users need to adapt their batch scripts accordingly to be able to process batch jobs after this date. From user perspective, the transition from PBSPro to Slurm should be a rather straightforward process, however it goes along with some new environmental variables, commands and job parameters.

To assist users, we give below a summary of the most important changes.

If you encounter problems during the conversion of your scripts or have further questions, please do not hesitate to contact us directly or to send a message to our ticket system: hpcsupport@rz.uni-kiel.de.

Best regards,

your HPC-Support Team

PS, for a full documentation of Slurm you may visit the official Slurm website.


 

PBSPro vs Slurm: environment variables

 

Environment variable PBSPro Slurm
Job ID $PBS_JOBID $SLURM_JOBID
Job name $PBS_JOBNAME $SLURM_JOB_NAME
Directory from which the job has been submitted $PBS_O_WORKDIR $SLURM_SUBMIT_DIR
Node list $PBS_NODELIST $SLURM_NODELIST

 

  • Note, that in Slurm it is not necessary to explicitly change into the directory named $SLURM_SUBMIT_DIR , as jobs will automatically use the directory from which they have been submitted.
  • During job submission in Slurm, all environment variables which have been defined (so far) will be transfered to the batch job. While this is convenient at first glance, it can lead to the fact that the same job script behaves differently depending on the environment variables set. For this reason it is advisable to set the following directive: #SBATCH --export=NONE .

 

PBSPro vs Slurm: commands

 

Command PBSPro Slurm
Submit a batch job qsub <jobscript> sbatch <jobscript>
Delete or terminate a batch job qdel <jobid> scancel <jobid>
List all jobs in the system qstat squeue
List own jobs qstat -u <userid> squeue -u <userid>
Show status of a specific job qstat <jobid> squeue -j <jobid>
Show details of a specific job qstat -f <jobid> scontrol show job <jobid>

 

  • To start an interactive run equivalent to qsub -I , please see the examples on the rzcluster page (under topic: batch processing).
  • Note, that in Slurm all output is written to its destination already during job execution. Thus, a command (equivalent to qpeek <jobid> ) in order to monitor the job's stdout progress is obsolete.
  • Detailed resource information about a running job is available in Slurm via the command sstat -j <jobid> .

 

PBSPro vs Slurm: job parameters

 

  • Like in PBSPro, a Slurm batch script begins with the #!/bin/bash directive on the first line. The subsequent lines then contain the directive #SBATCH followed by a resource request or other job information. For specific examples and templates see the next section.

 

Job parameter PBSPro Slurm
Batch script directive #PBS #SBATCH
Queue/partition -q=<queue> --partition=<partition> or -p <partition>
Job name -N=<jobname> --job-name=<jobname> or -J <jobname>
Stdout file -o=<filename> --output=<filename> or -o <filename>
Stderr file -e=<filename> --error=<filename> or -e <filename>
Stdout and stderr into the same file -j oe --output=<filename> or -o <filename> (default if no --error is specified)
Number of nodes -l select=<n> --nodes=<n> or -N <n>
Number of (MPI) processes per node -l mpiprocs=<m> --tasks-per-node=<m>
Number of CPUs per node or task -l ncpus=<m> --cpus-per-task=<m> or -c <m>
Main memory (in MB) -l mem=<memory> --mem=<memory>
... in GB   --mem=1G
Walltime -l walltime=[hh:mm:ss] --time=[hh:mm:ss] or -t [hh:mm:ss]
Never requeue the job -r n --no-requeue
Email address -M <address> --mail-user=<address>
Email notifications -m b --mail-type=BEGIN
  -m e --mail-type=END
  -m a --mail-type=FAIL
  -m abe --mail-type=ALL

 

Slurm script templates

  • Example for a serial calculation:
    #!/bin/bash
    #SBATCH --job-name=test
    #SBATCH --output=test.out
    #SBATCH --error=test.err
    #SBATCH --nodes=1
    #SBATCH --tasks-per-node=1
    #SBATCH --cpus-per-task=1
    #SBATCH --mem=1000
    #SBATCH --time=01:00:00
    #SBATCH --partition=small
    
    export OMP_NUM_THREADS=1
    ./test.ex
  • Example for a parallel multi-node MPI calculation:
    #!/bin/bash
    #SBATCH --job-name=test
    #SBATCH --output=test.out
    #SBATCH --error=test.err
    #SBATCH --nodes=2
    #SBATCH --tasks-per-node=12
    #SBATCH --cpus-per-task=1
    #SBATCH --mem=10000
    #SBATCH --time=01:00:00
    #SBATCH --partition=small
    
    export OMP_NUM_THREADS=1
    module load intelmpi16.0.0
    mpirun -np 24 ./test.ex
    
  • Example for a parallel multi-node hybrid (MPI+OpenMP) calculation:
    #!/bin/bash
    #SBATCH --job-name=test
    #SBATCH --output=test.out
    #SBATCH --error=test.err
    #SBATCH --nodes=2
    #SBATCH --tasks-per-node=6
    #SBATCH --cpus-per-task=2
    #SBATCH --mem=10000
    #SBATCH --time=01:00:00
    #SBATCH --partition=small
    
    export OMP_NUM_THREADS=2
    module load intelmpi16.0.0
    mpirun -np 12 ./test.ex
  • Example for a job array:
    #!/bin/bash
    #SBATCH --job-name=test
    #SBATCH --array 0-9
    #SBATCH --output test-%A_%a.out
    #SBATCH --nodes=1
    #SBATCH --tasks-per-node=1
    #SBATCH --cpus-per-task=1
    #SBATCH --mem=1000
    #SBATCH --time=00:10:00
    #SBATCH --partition=small
    
    echo "Hi, I am step $SLURM_ARRAY_TASK_ID in the array job $SLURM_ARRAY_JOB_ID"

 


Support and Consulting

HPC-Support-Team: hpcsupport@rz.uni-kiel.de
Responsible contact persons at the Computing Centre:
Please see HPC-Support and Consulting.