Benchmarking: HPL (optimised) with OpenMPI and Vanilla Centos

Install OpenMPI

Optionally rebuid OpenMPI ()

  yum install openmpi openmpi-devel

Setup the user environment

# add to the end of your ~/.bashrc
export PATH=$PATH:/usr/lib64/openmpi/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/openmpi/lib

Build ATLAS

NOTE" Ensure throttling is disabled as ATLAS will give out

/etc/init.d/cpuspeed stop 

# may also have to apply this patch for ATLAS 3.10.1
patch -p0 CONFIG/src/probe_arch.c << EOF
@@ -238,8 +238,7 @@ int main(int nargs, char **args)
       printf("CPU MHZ=%d\n",
              ProbeOneInt(OS, asmd, targ, "-m", "CPU MHZ=", &sure));
    if (flags & Pthrottle)
-      printf("CPU THROTTLE=%d\n",
-             ProbeOneInt(OS, asmd, targ, "-t", "CPU THROTTLE=", &sure));
+      printf("CPU THROTTLE=0\n");
    if (flags & P64)
    {
       if (asmd == gas_x86_64)
EOF

Download From: http://sourceforge.net/projects/math-atlas/files/Stable/3.10.1/atlas3.10.1.tar.bz2/download

  wget http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2
  tar jxvf atlas3.10.1.tar.bz2 
  mkdir atlas3.10.1_build
  cd atlas3.10.1_build/
  /root/scratch/ATLAS/configure --prefix=/opt/atlas/3.10.1
  make 
  make check

Get Linpack

Latest versions are available from : http://www.netlib.org/benchmark/hpl/

  wget http://www.netlib.org/benchmark/hpl/hpl-2.1.tar.gz
  tar zxvf hpl-2.1.tar.gz

Setup the Makefile

  cd hpl-2.1
  cp setup/Make.Linux_PII_CBLAS Make.Linux_JB

Edit the Makefile

# Change the following
[root@localhost hpl-2.1]# diff setup/Make.Linux_PII_CBLAS Make.Linux_JB
64c64
< ARCH         = Linux_PII_CBLAS
---
> ARCH         = Linux_JB
70c70
< TOPdir       = $(HOME)/hpl
---
> TOPdir       = $(HOME)/scratch/hpl-2.1
84,86c84,86
< MPdir        = /usr/local/mpi
< MPinc        = -I$(MPdir)/include
< MPlib        = $(MPdir)/lib/libmpich.a
---
> MPdir        = /usr/lib64/openmpi
> MPinc        = -I/usr/include/openmpi-x86_64
> MPlib        = $(MPdir)/lib/libmpi.so
95c95
< LAdir        = $(HOME)/netlib/ARCHIVES/Linux_PII
---
> LAdir        = /usr/lib64/atlas-sse3/
176c176
< LINKER       = /usr/bin/g77
---
> LINKER       = mpif90

Build

  make arch=Linux_JB

Run (check the run works ok with the default HPL.dat file - which is for 4 core)

  cd bin/Linux_JB
  mpirun -np 4 ./xhpl

Edit the HPL to optimise (to be filled in)