OpenMM

From the OpenMM website http://openmm.org/

OpenMM is a high performance toolkit for molecular simulation. Use it as a library, or as an application. We include extensive language bindings for Python, C, C++, and even Fortran. The code is open source and actively maintained on Github, licensed under MIT and LGPL. Part of the Omnia suite of tools for predictive biomolecular simulation.

Installing OpenMM on CSD3

Below are instructions of how to install openmm in a miniconda virtual environment. Part of the installation instructions are located at http://docs.openmm.org/latest/userguide/application.html#installing-openmm. One important point to note is that the current version of OpenMM requires CUDA version 9.2, but on CSD3 the default CUDA version is 8.0.

Connect to a Wilkes2 login node with:

ssh your-username@login-gpu.hpc.cam.ac.uk

Load the required modules with:

module unload cuda/8.0
module load cuda/9.2
module load miniconda3-4.5.4-gcc-5.4.0-hivczbz

Create a Conda virtual environment and install OpenMM with:

conda create -p openmm-env
source activate openmm-env/
conda install -c omnia -c conda-forge openmm

Test openmm in the login node with:

python -m simtk.testInstallation

All the test should succeed except the CUDA one because there is no NVIDIA GPU on the login node.

Running OpenMM on CSD3

To submit an OpenMM job please use the following submission script:

#! Which project should be charged (NB Wilkes2 projects end in '-GPU'):
#SBATCH -A CHANGE_TO_YOUR_ACCOUNT-GPU
#! How many whole nodes should be allocated?
#SBATCH --nodes=1
#! How many (MPI) tasks will there be in total?
#! Note probably this should not exceed the total number of GPUs in use.
#SBATCH --ntasks=1
#! Specify the number of GPUs per node (between 1 and 4; must be 4 if nodes>1).
#! Note that the job submission script will enforce no more than 3 cpus per GPU.
#SBATCH --gres=gpu:1
#! How much wallclock time will be required?
#SBATCH --time=00:10:00
#! What types of email messages do you wish to receive?
#SBATCH --mail-type=FAIL
#! Uncomment this to prevent the job from being requeued (e.g. if
#! interrupted by node failure or system downtime):
##SBATCH --no-requeue

#! Do not change:
#SBATCH -p pascal

#! sbatch directives end here (put any additional directives above this line)

#! Notes:
#! Charging is determined by GPU number*walltime.

#! Number of nodes and tasks per node allocated by SLURM (do not change):
numnodes=$SLURM_JOB_NUM_NODES
numtasks=$SLURM_NTASKS
mpi_tasks_per_node=$(echo "$SLURM_TASKS_PER_NODE" | sed -e  's/^\([0-9][0-9]*\).*$/\1/')
#! ############################################################
#! Modify the settings below to specify the application's environment, location
#! and launch method:

#! Optionally modify the environment seen by the application
#! (note that SLURM reproduces the environment at submission irrespective of ~/.bashrc):
. /etc/profile.d/modules.sh                # Leave this line (enables the module command)
module purge                               # Removes all modules still loaded
module load rhel7/default-gpu              # REQUIRED - loads the basic environment

module unload cuda/8.0
module load cuda/9.2
module load miniconda3-4.5.4-gcc-5.4.0-hivczbz

source activate openmm-env/
python -m simtk.testInstallation
conda deactivate

The output should be the following (times might be slightly different):

There are 4 Platforms available:

1 Reference - Successfully computed forces
2 CPU - Successfully computed forces
3 CUDA - Successfully computed forces
4 OpenCL - Successfully computed forces

Median difference in forces between platforms:

Reference vs. CPU: 6.29882e-06
Reference vs. CUDA: 6.73039e-06
CPU vs. CUDA: 7.76279e-07
Reference vs. OpenCL: 6.75426e-06
CPU vs. OpenCL: 8.19109e-07
CUDA vs. OpenCL: 2.19595e-07

The slurm submit script can be submitted to the queue with sbatch. This will run on one node using one core up to a limit of 10 minutes. To run on more cores the -n option should be adjusted and the timelimit can be changed with -t.