David: Created page with "Prerequisites CUDA >=3.0 CMake 2.6 Python 2.7 Built on CentOS 5.7 with GCC 4.1.2 20080704. For MPI mpi/mvapich2-interconnects-gnu was used. Notes adapted from http://www..."

2012-10-12T15:25:19Z

Created page with "Prerequisites CUDA >=3.0 CMake 2.6 Python 2.7 Built on CentOS 5.7 with GCC 4.1.2 20080704. For MPI mpi/mvapich2-interconnects-gnu was used. Notes adapted from http://www..."

New page

Prerequisites

CUDA >=3.0

CMake 2.6

Python 2.7

Built on CentOS 5.7 with GCC 4.1.2 20080704. For MPI mpi/mvapich2-interconnects-gnu was used.

Notes adapted from http://www.gromacs.org/Downloads/Installation_Instructions/GPUs

==== Installing pre-requisites ====

CentOS 5.7 does not ship with CMake out of the box, thankfully it is in the repositories.

<syntaxhighlight>
yum install cmake
</syntaxhighlight>

The installed and latest version of python was 2.4.3

Obtain the source code from http://python.org/getit/. The following will install Python to /usr/local/bin/python rather than the existing location /usr/bin/python

<syntaxhighlight>
tar xzvf Python-2.7.2.tgz
cd Python-2.7.2
./configure
make
make install
</syntaxhighlight>

====Build Gromacs for MPI ====

FFTW and Gromacs must be built with the same precision tpyes

===== FFTW =====

<syntaxhighlight>
tar xzvf fftw-3.3.tar.gz

#For single precision
./configure --enable-threads --enable-float --enable-sse --prefix=/shared/fftw

#For double precision
./configure --enable-threads --prefix=/shared/fftw

make
make install
</syntaxhighlight>
===== Gromacs =====

<syntaxhighlight>
tar xzvf gromacs-4.5.5.tar.gz
mkdir -p /shared/gromacs/build
mkdir /shared/gromacs/lib

cd /shared/gromacs/build

module load mpi/mvapich2-interconnects-gnu

export FFTW_LOCATION=/shared/fftw/bin

#For single precision
cmake -DCMAKE_INSTALL_PREFIX=/shared/gromacs -DLIB_INSTALL_DIR=/shared/gromacs/lib -DSOVERSION=6 -D FFTW3F_INCLUDE_DIR=/shared/fftw/include -D FFTW3F_LIBRARIES=/shared/fftw/lib/libfftw3f.a -DCMAKE_CXX_COMPILER=/opt/mpich2/gnu/bin/mpicxx -DCMAKE_C_COMPILER=/opt/mpich2/gnu/bin/mpicc -DGMX_MPI=ON -DGMX_PREFER_STATIC_LIBS=ON /shared/gromacs-4.5.5

#For double precision - Currently not working
cmake -DCMAKE_INSTALL_PREFIX=/shared/gromacs -DLIB_INSTALL_DIR=/shared/gromacs/lib -DSOVERSION=6 -D FFTW3F_INCLUDE_DIR=/shared/fftw/include -D FFTW3F_LIBRARIES=/shared/fftw/lib/libfftw3.a -DCMAKE_CXX_COMPILER=/opt/mpich2/gnu/bin/mpicxx -DCMAKE_C_COMPILER=/opt/mpich2/gnu/bin/mpicc -DGMX_MPI=ON -DGMX_PREFER_STATIC_LIBS=ON /shared/gromacs-4.5.5

make
make install-mdrun
</syntaxhighlight>

==== Build Gromacs for GPU ====

===== OpenMM =====

Note: Ensure CUDA environment variables are set before proceeding

Before installing Gromacs for GPU we need to install OpenMM

<syntaxhighlight>
mkdir -p /shared/OpenMM/lib/plugins

#As we are installing in a non standard location we need to set some variables
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/shared/OpenMM/lib
export OPENMM_PLUGIN_DIR=/shared/OpenMM/lib/plugins
export OPENMM_ROOT_DIR=/shared/OpenMM

unzip OpenMM4.0-Linux64.zip
./install.sh
</syntaxhighlight>

===== Gromacs =====

Note: Parallel runs are not supported in the GPU build of Gromacs 4.5

To build Gromacs for GPU

<syntaxhighlight>
tar xzvf gromacs-4.5.5.tar.gz
mkdir -p /shared/gromacs/build
mkdir /shared/gromacs/lib

cd /shared/gromacs/build

cmake -DGMX_OPENMM=ON -DCMAKE_INSTALL_PREFIX=/shared/gromacs -DLIB_INSTALL_DIR=/shared/gromacs/lib -DSOVERSION=6 /shared/gromacs-4.5.5

make mdrun
make install-mdrun
</syntaxhighlight>

===== Benchmarking =====

Benchmarks can be obtained from http://www.gromacs.org/@api/deki/files/128/=gromacs-gpubench-dhfr.tar.gz

For MPI

<syntaxhighlight>
mkdir /shared/gromacs/benchmarks
tar xzvf -C /shared/gromacs/bechmarks gromacs-gpubench-dhfr.tar.gz
cd /shared/gromacs/benchmarks/dhfr/GPU/dhfr-impl-1nm.bench
mpirun -np 8 -machinefile ~/nodes /shared/gromacs/bin/mdrun-gpu
</syntaxhighlight>

For GPU

<syntaxhighlight>
mkdir /shared/gromacs/benchmarks
tar xzvf -C /shared/gromacs/bechmarks gromacs-gpubench-dhfr.tar.gz
cd /shared/gromacs/benchmarks/dhfr/GPU/dhfr-impl-1nm.bench
/shared/gromacs/bin/mdrun-gpu -device "OpenMM:force-device=yes" #I experinced an error stating Tesla M2090
#was not supported even though it is listed
#as a supported device
</syntaxhighlight>

You can specifiy whch card to run on with the DeviceID switch. Usefull if you need to run on a certain card or want to start another instance.

<syntaxhighlight>
mdrun-gpu -device "DeviceID=1" #Deaults to 0
</syntaxhighlight>

Benchmarking: Gromacs - Revision history

David: Created page with "Prerequisites CUDA >=3.0 CMake 2.6 Python 2.7 Built on CentOS 5.7 with GCC 4.1.2 20080704. For MPI mpi/mvapich2-interconnects-gnu was used. Notes adapted from http://www..."