Benchmarking: Gromacs

Prerequisites

CUDA >=3.0

CMake 2.6

Python 2.7

Built on CentOS 5.7 with GCC 4.1.2 20080704. For MPI mpi/mvapich2-interconnects-gnu was used.

Notes adapted from http://www.gromacs.org/Downloads/Installation_Instructions/GPUs

Installing pre-requisites

CentOS 5.7 does not ship with CMake out of the box, thankfully it is in the repositories.

yum install cmake

The installed and latest version of python was 2.4.3

Obtain the source code from http://python.org/getit/. The following will install Python to /usr/local/bin/python rather than the existing location /usr/bin/python

tar xzvf Python-2.7.2.tgz
cd Python-2.7.2
./configure
make
make install

Build Gromacs for MPI

FFTW and Gromacs must be built with the same precision tpyes

FFTW

tar xzvf fftw-3.3.tar.gz

#For single precision
./configure --enable-threads --enable-float --enable-sse --prefix=/shared/fftw

#For double precision
./configure --enable-threads --prefix=/shared/fftw

make
make install

Gromacs

tar xzvf gromacs-4.5.5.tar.gz
mkdir -p /shared/gromacs/build
mkdir /shared/gromacs/lib

cd /shared/gromacs/build

module load mpi/mvapich2-interconnects-gnu

export FFTW_LOCATION=/shared/fftw/bin

#For single precision
cmake -DCMAKE_INSTALL_PREFIX=/shared/gromacs -DLIB_INSTALL_DIR=/shared/gromacs/lib -DSOVERSION=6  -D FFTW3F_INCLUDE_DIR=/shared/fftw/include -D FFTW3F_LIBRARIES=/shared/fftw/lib/libfftw3f.a -DCMAKE_CXX_COMPILER=/opt/mpich2/gnu/bin/mpicxx -DCMAKE_C_COMPILER=/opt/mpich2/gnu/bin/mpicc -DGMX_MPI=ON -DGMX_PREFER_STATIC_LIBS=ON /shared/gromacs-4.5.5

#For double precision - Currently not working
cmake -DCMAKE_INSTALL_PREFIX=/shared/gromacs -DLIB_INSTALL_DIR=/shared/gromacs/lib -DSOVERSION=6  -D FFTW3F_INCLUDE_DIR=/shared/fftw/include -D FFTW3F_LIBRARIES=/shared/fftw/lib/libfftw3.a -DCMAKE_CXX_COMPILER=/opt/mpich2/gnu/bin/mpicxx -DCMAKE_C_COMPILER=/opt/mpich2/gnu/bin/mpicc -DGMX_MPI=ON -DGMX_PREFER_STATIC_LIBS=ON /shared/gromacs-4.5.5

make
make install-mdrun

Build Gromacs for GPU

OpenMM

Note: Ensure CUDA environment variables are set before proceeding

Before installing Gromacs for GPU we need to install OpenMM

mkdir -p /shared/OpenMM/lib/plugins

#As we are installing in a non standard location we need to set some variables
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/shared/OpenMM/lib
export OPENMM_PLUGIN_DIR=/shared/OpenMM/lib/plugins
export OPENMM_ROOT_DIR=/shared/OpenMM

unzip OpenMM4.0-Linux64.zip
./install.sh

Gromacs

Note: Parallel runs are not supported in the GPU build of Gromacs 4.5

To build Gromacs for GPU

tar xzvf gromacs-4.5.5.tar.gz
mkdir -p /shared/gromacs/build
mkdir /shared/gromacs/lib

cd /shared/gromacs/build

cmake -DGMX_OPENMM=ON -DCMAKE_INSTALL_PREFIX=/shared/gromacs -DLIB_INSTALL_DIR=/shared/gromacs/lib -DSOVERSION=6 /shared/gromacs-4.5.5

make mdrun
make install-mdrun

Benchmarking

Benchmarks can be obtained from http://www.gromacs.org/@api/deki/files/128/=gromacs-gpubench-dhfr.tar.gz

For MPI

mkdir /shared/gromacs/benchmarks
tar xzvf -C /shared/gromacs/bechmarks gromacs-gpubench-dhfr.tar.gz
cd /shared/gromacs/benchmarks/dhfr/GPU/dhfr-impl-1nm.bench
mpirun -np 8 -machinefile ~/nodes /shared/gromacs/bin/mdrun-gpu

For GPU

mkdir /shared/gromacs/benchmarks
tar xzvf -C /shared/gromacs/bechmarks gromacs-gpubench-dhfr.tar.gz
cd /shared/gromacs/benchmarks/dhfr/GPU/dhfr-impl-1nm.bench
/shared/gromacs/bin/mdrun-gpu -device "OpenMM:force-device=yes" #I experinced an error stating Tesla M2090
                                                                #was not supported even though it is listed
                                                                #as a supported device

You can specifiy whch card to run on with the DeviceID switch. Usefull if you need to run on a certain card or want to start another instance.

mdrun-gpu -device "DeviceID=1" #Deaults to 0