Benchmarking: HPL CUDA Accelerated for Linux64

Requirements

- CUDA 5.0
- GNU Compilers
- Intel MKL
- OpenMPI (Configured from computenode and compiled with GNU compilers)
- Tesla GPUs

Download

Download CUDA Accelerated Linpack for Linux64 (hpl-2.0_FERMI_v15.gz) from NVIDIA Developer Portal

untar hpl-2.0_FERMI_v15.gz
cd hpl-2.0_FERMI_v15

Edit Make.CUDA

# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
TOPdir = /shared/apps/linpack/hpl-2.0_FERMI_v15
.
.
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS) -----------------------------
# ----------------------------------------------------------------------
LAdir        = /shared/apps/intel/mkl/lib/intel64
LAinc        = -I/shared/apps/cuda/cuda-5.0/include
LAlib = -L$(TOPdir)/src/cuda -ldgemm \
        -L/shared/apps/cuda/cuda-5.0/lib64 -lcublas -lcuda -lcudart \
        -L$(LAdir) -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core
.
.
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
CC      = mpicc
CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall -fopenmp

Environment

This is to make sure that you don't have any issues in compiling HPL on your system because they have hard-coded in the /usr/local/cuda/ directory as the default directory in all of their makefiles. You can go through them all and change it, but it is just easier to create a link to where you have it installed.

cd /usr/local
ln -s /shared/apps/cuda/cuda-5.0 cuda

Build

module load cuda/5.0
module load openmpi/1.6.3-gpu
make arch=CUDA

Modulefile

File location: /shared/modulefiles/gpu/linpack
File name: hpl-2.0_FERMI_v15

#%Module 1.0
#
prereq  cuda/5.0
prereq  openmpi/1.6.3-gpu
prepend-path  PATH      /shared/apps/linpack/hpl-2.0_FERMI_v15/bin/CUDA
prepend-path  LD_LIBRARY_PATH           /shared/apps/intel/mkl/lib/intel64

Run (from GPU node)

Load necessary modules

module load cuda/5.0
module load openmpi/1.6.3-gpu
module load gpu/linpack/hpl-2.0_FERMI_v15

Copy HPL.dat file to local folder and make changes if required.

cp /shared/apps/linpack/hpl-2.0_FERMI_v15/bin/CUDA/HPL.dat .

Run linpack

mpirun -np 2 run_linpack