Application Development Environment

The application development environment on CSD3 is primarily controlled through the modules environment. By loading and switching modules you control the compilers, libraries and software available.

This means that for compiling on CSD3 you typically set the compiler you wish to use using the appropriate modules, then load all the required library modules (e.g. numerical libraries, IO format libraries).

Additionally, if you are compiling parallel applications using MPI (or SHMEM, etc.) then you will need to load one of the MPI environments and use the appropriate compiler wrapper scripts.

By default, all users on CSD3 start with no modules loaded.

Basic usage of the module command on CSD3 is covered below. For full documentation please see:

Note: The modules provided by the Spack package manager behave differently to those usually encountered in Linux environments. In particular, each module has the versions of dependency libraries hardcoded using RPATH. More information is provided below. You can identify Spack modules as they have a random string of 7 characters at the end of their name, e.g.: fftw-3.3.5-intel-17.0.2-dxt2dzn.

Using the modules environment

Information on the available modules

Finding out which modules (and hence which compilers, libraries and software) are available on the system is performed using the module avail command:

user@system:~> module avail
...

This will list all the names and versions of the modules available on the service. Not all of them may work in your account though due to, for example, licencing restrictions. You will notice that for many modules we have more than one version, each of which is identified by a version number. One of these versions is the default. As the service develops the default version will change.

You can list all the modules of a particular type by providing an argument to the module avail command. For example, to list all available versions of the Intel Compiler type:

[user@login0 ~]$ module avail intel-compilers

--------------------------------------- /lustre/sw/modulefiles ---------------------------------------
intel-compilers-16/16.0.2.181 intel-compilers-16/16.0.3.210

If you want more info on any of the modules, you can use the module help command:

[user@login0 ~]$ module help mpt

----------- Module Specific Help for 'mpt/2.14' -------------------

The SGI Message Passing Toolkit (MPT) is an optimized MPI
implementation for SGI systems and clusters.  See the
MPI(1) man page and the MPT User's Guide for more
information.

The simple module list command will give the names of the modules and their versions you have presently loaded in your envionment:

[user@login0 ~]$ module list
Currently Loaded Modulefiles:
1) mpt/2.14                        3) intel-fc-16/16.0.3.210
2) intel-cc-16/16.0.3.210          4) intel-compilers-16/16.0.3.210

Loading, unloading and swapping modules

To load a module to use module add or module load. For example, to load the intel-compilers-17 into the development environment:

module load intel-compilers-17

This will load the default version of the intel commpilers Library. If you need a specfic version of the module, you can add more information:

module load intel-compilers-17/17.0.2.174

will load version 16.0.3.210 for you, regardless of the default. If you want to clean up, module remove will remove a loaded module:

module remove intel-compilers-17

(or module rm intel-compilers-17 or module unload intel-compilers-17) will unload what ever version of intel-compilers-17 (even if it is not the default) you might have loaded. There are many situations in which you might want to change the presently loaded version to a different one, such as trying the latest version which is not yet the default or using a legacy version to keep compatibility with old data. This can be achieved most easily by using “module swap oldmodule newmodule”.

Suppose you have loaded version 16.0.2.181, say, of intel-compilers-16, the following command will change to version 16.0.3.210:

module swap intel-compilers-16 intel-compilers-16/16.0.2.181

Available Compiler Suites

Note: As CSD3 uses dynamic linking by default you will generally also need to load any modules you used to compile your code in your job submission script when you run your code.

Intel Compiler Suite

The Intel compiler suite is accessed by loading the intel-compilers-* module. For example:

module load intel-compilers-17

Once you have loaded the module, the compilers are available as:

  • ifort - Fortran
  • icc - C
  • icpc - C++

C++ with Intel Compilers

Intel compilers rely on GCC C++ headers and libraries to support more recent C++11 features. If you are using Intel compilers to compile C++ on CSD3 you should also load the gcc/5.4.0 module to have access to the correct C++ files:

::
module load gcc/5.4.0

Note: You will also need to load this module in your job submission scripts when running code compiled in this way.

GCC Compiler Suite

The GCC compiler suite is accessed by loading the gcc module. For example:

module load gcc

Once you have loaded the module, the compilers are available as:

  • gfortran - Fortran
  • gcc - C
  • g++ - C++

PGI Compiler Suite

The Portland Group (PGI) compilers are available under the pgi modules. For example:

module load pgi/2017

Once you have loaded the module, the compilers are available as:

  • pgfortran - Fortran
  • pgcc - C
  • pg++ - C++

Compiling MPI codes

There are two prefered MPI libraries currently available on CSD3:

  • Intel MPI
  • OpenMPI

Using Intel MPI

To compile MPI code with Intel MPI, using any compiler, first load the intel/bundles/complib/2020.2 module (which e.g. on the cclake CPU nodes is loaded by default as part of the rhel7/default-ccl module):

module load intel/bundles/complib/2020.2

This makes the compiler wrapper scripts available to you. To change the underlying compiler, use the I_MPI_ environment variables:

Language Intel (default) GCC PGI
Fortran I_MPI_F90=ifort I_MPI_F90=gfortran I_MPI_F90=pgfortran
C++ I_MPI_CXX=icpc I_MPI_CXX=g++ I_MPI_CXX=pg++
C I_MPI_CC=icc I_MPI_CC=gcc I_MPI_CC=pgcc

Further details on using the different compiler suites with Intel MPI are available in the following sections.

Compiler Information and Options

The manual pages for the different compiler suites are available:

GCC
Fortran man gfortran , C/C++ man gcc
Intel
Fortran man ifort , C/C++ man icc

Useful compiler options

Whilst difference codes will benefit from compiler optimisations in different ways, for reasonable performance, at least initially, we suggest the following compiler options:

Intel
-O2
GNU
-O2 -ftree-vectorize -funroll-loops -ffast-math

To target the specific hardware on CSD3 use the following options:

Partition Intel GCC
cclake -xCASCADELAKE -march=cascadelake
icelake -xICELAKE-SERVER -march=icelake-server
sapphire -xSAPPHIRERAPIDS -march=sapphirereapids
ampere   -march=znver3

Alternatively, login to a machine with the same architevture that you will be running on and use

Intel
-xHost
GNU
-march=native

When you have a application that you are happy is working correctly and has reasonable performance you may wish to investigate some more aggressive compiler optimisations. Below is a list of some further optimisations that you can try on your application (Note: these optimisations may result in incorrect output for programs that depend on an exact implementation of IEEE or ISO rules/specifications for math functions):

Intel
-fast
GNU
-Ofast -funroll-loops

Vectorisation, which is one of the important compiler optimisations for any modern Intel hardware, is enabled by default as follows:

Intel
At -O2 and above
GNU
At -O3 and above or when using -ftree-vectorize

To promote integer and real variables from four to eight byte precision for FORTRAN codes the following compiler flags can be used:

Intel
-real-size 64 -integer-size 64 -xAVX (Sometimes the Intel compiler incorrectly generates AVX2 instructions if the -real-size 64 or -r8 options are set. Using the -xAVX option prevents this.)
GNU
-freal-4-real-8 -finteger-4-integer-8

Using static linking/libraries

By default, executables are built using shared/dynamic libraries (that is, libraries which are loaded at run-time as and when needed by the application) when using the wrapper scripts.

An application compiled this way to use shared/dynamic libraries will use the default version of the library installed on the system (just like any other Linux executable), even if the system modules were set differently at compile time. This means that the application may potentially be using slightly different object code each time the application runs as the defaults may change. This is usually the desired behaviour for many applications as any fixes or improvements to the default linked libraries are used without having to recompile the application, however some users may feel this is not the desired behaviour for their applications.

Alternatively, applications can be compiled to use static libraries (i.e. all of the object code of referenced libraries are contained in the executable file). This has the advantage that once an executable is created, whenever it is run in the future, it will always use the same object code (within the limit of changing runtime environemnt). However, executables compiled with static libraries have the potential disadvantage that when multiple instances are running simultaneously multiple copies of the libraries used are held in memory. This can lead to large amounts of memory being used to hold the executable and not application data.

To create an application that uses static libraries you must pass an extra flag during compilation, -Bstatic.

Use the UNIX command ldd exe_file to check whether you are using an executable that depends on shared libraries. This utility will also report the shared libraries this executable will use if it has been dynamically linked.