Zenith Quick Start Guide =========== *These new nodes entered Early Access service in August 2026* .. important:: * This is a service that is in **early access** and not yet available for projects. Please keep an eye on official channels of communication from the Cambridge RCS team for further information. * During the Early Access phase, this page will be updated and improved regularly, if you notice something not quite right, please contact support. Getting Access ------ As mentioned above, this is in early access and so you will be invited to the relevant early access project via our Cambridge Access Portal, **Waldur**, please follow the guidance found in `User Dawn Access `_ as the procedure is the same for Zenith projects. First Time Accessing Zenith ------- MultiFactor Authentication (MFA) is required, as with all our services. If you are already a user of our services (CSD3, Dawn, Storage, or AIRR) you will already have the appropriate MFA setup. Otherwise, it is strongly recommended that you read through the `MultiFactor Authentication (MFA) `_ user documentation to assist with the setup of your MFA. Login ------- All Zenith users should initially use the following login nodes to access Zenith resources (Zenith-specific login nodes will be forthcoming): :: ssh @login-dawn.hpc.cam.ac.uk The **first time** you login, you will be asked to check that the host key fingerprints are correct. Please check that the fingerprints reported match those on the page `CSD3 Host Keys `_ *before* responding *yes* (NB not *y*). You will then be asked for your *TOTP Verification code*; this is the MFA step mentioned above. If you cannot connect to the above login nodes, first check that you have received and completed your Zenith project invite, as described above, and have received confirmation that you have been added to the project. If you have checked and received confirmation, please contact support@hpc.cam.ac.uk for further assistance. Once you have logged in, you will have immediate access to your storage, but for development work you will likely want to request an interactive job using Slurm, as detailed below. This is because the login nodes are currently a different architecture to the Zenith nodes themselves, and also to lessen load on the login nodes. Filesystems -------------- Please see `here `_ for a summary of available filesystems and the rules governing them. Hardware -------- Each Zenith (MI355X) node consists of: - 8x AMD Instinct MI355X GPUs. - 2x AMD EPYC 9575F processors (64 CPU cores per processor). - 288 GiB of GPU memory per GPU. - 64 TB/s aggregate GPU memory bandwidth per node. Slurm Partition --------------- The MI355X (gpu-u-N) nodes are in the **mi355x** Slurm partition. Zenith Slurm projects follow the usual CSD3 naming convention for GPU projects and follow the pattern ``NAME-ZENITH-GPU``, containing units of GPU hours. Modules ------- To load the Zenith-specific modules run: :: module purge module load rhel9/mi355x/base This will make available modules tailored to the Zenith architecture. In particular there are modules for the AMD compilers: :: module load rocm The ``rocm`` module gives access to the ``amdclang``, ``amdclang++``, and ``amdflang`` compilers. These are strongly recommended for CPU codes, and essential for GPU codes. The GCC compiler suite is also available via the ``gcc`` module. For MPI codes, there is the ``openmpi`` module. This is tied to the ``rocm`` compiler suite. If you believe you need MPI with the GCC suite, please contact support. For work with python, use either the ``python`` module and use python virtual environments or the ``miniforge3`` module and use conda environments. Recommendations for running on Zenith ------- To run an interactive job on Zenith, use the ``sint`` command, e.g. to request 16 CPU cores on 1 node, with 1 GPU for 10 minutes run: :: sint -p mi355x -N1 -n1 -c16 --gres=gpu:1 -A NAME-ZENITH-GPU -t 00:10:00 Note that you will be billed for Zenith GPU hours on interactive jobs. When compiling codes to run using the Zenith GPUs you should specify the particular AMD architecture by passing the ``--offload-arch=gfx950`` flag to the AMD compiler you are using. You may also use ``--offload-arch=native`` when compiling on a node. Problems -------- Please contact support@hpc.cam.ac.uk if you encounter any problems or have any questions