Cascade Lake Nodes¶

The Cascade Lake upgrade to CSD3 entered general service in November 2020. All users are encouraged to make use of these nodes.

Slurm parition¶

The Cascade Lake nodes are named according to the scheme cpu-p-[1-672].
The Cascade Lake nodes are in cclake and cclake-himem Slurm partitions. Your existing -CPU projects will be able to submit jobs to these.
The cclake nodes have 56 cpus (1 cpu = 1 core), and 192 GiB of RAM. This means that they have 3410 MiB per cpu.
The cclake-himem nodes have 56 cpus (1 cpu = 1 core), and 384 GiB of RAM. This means that they have 6840 MiB per cpu.
The Cascade Lake nodes are interconnected by Mellanox HDR100 Infiniband.
The Cascade Lake nodes are running Rocky Linux 8, which is a rebuild of Red Hat Enterprise Linux 8 (RHEL8).

Recommendations for running on cclake¶

Since the cpu-p nodes are running Rocky8, you will want to recompile your applications.

It is possible to log into a Cascade Lake login nodes with compatible OS/software by logging into login-cascadelake.hpc.cam.ac.uk, which will land you on one of the login-p-* nodes.

Alternatively, request an interactive node using sintr:

sintr -A yourproject-cpu -N1 -n4 -t 1:00:00 -p cclake

The per-job wallclock time limits are 36 hours and 12 hours for SL1/2 and SL3 respectively.

The per-job, per-user cpu limits are now 4256 and 448 cpus for SL1/2 and SL3 respectively.

These limits should be regarded as provisional and may be revised.

Default submission script for cclake¶

You should find a symbolic link to a Cascade Lake default job submission script modified for the cclake nodes in your home directory, called:

slurm_submit.peta4-cclake

This is set up for MPI jobs using cclake, but can be modified for other types of job. If you prefer to modify your existing job scripts, please see the following sections for guidance.

Jobs not using MPI and requiring no more than 3410 MiB per cpu¶

In this case you should be able to simply add the cclake partition to your sbatch directive, e.g.:

#SBATCH -p cclake

will submit a job able to run on the first nodes available in the cclake partition.

Jobs requiring more than 3410 MiB per cpu¶

In the case of larger memory requirements, it is most efficient to submit instead to the cclake-himem partition which will allocate 6840 MiB per cpu:

#SBATCH -p cclake-himem

If this amount of memory per cpu is insufficient you will need to specify either the –mem= or –cpus-per-task= directive, in addition to the -p directive, in order to make sure you have enough memory at run time. Note that in this case, Slurm will satisfy the memory requirement by allocating (and charging for) more cpus if necessary. E.g.:

#SBATCH -p cclake-himem
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem=8000

In the above example we are requesting 8000 MiB of memory per node, but only one task. Slurm will usually allocate one cpu per task, but here, because it enforces 6840 MiB per cpu for cclake-himem, it will allocate 2 cpus to the single task in order that the job will have the 8000 MiB which it claims it requires. Note that this increases the number of cpu core hours consumed by the job and hence the charge. Also note that since each cpu receives 6840 MiB by default anyway, the user would lose nothing by requesting 13000 MiB instead of 8000 MiB here.

Jobs requiring MPI¶

We recommend using Intel MPI 2020 on Cascade Lake. There are other, related changes to the default environment seen by jobs running on the cclake nodes. If you wish to recompile or test in the cascade lake environment, the simplest option is to login directly to one of the login-cascadelake nodes, i.e. to login-cascadelake.hpc.cam.ac.uk. Alternatively, request an interactive node and work on a cclake node directly.