Benchmarking: ROME CPUs

Benchmarking AMD ROME and How

THought i would write up how i got to the results i had - Matthew Hole

Where the tar file for AMD HPL is stored

\\10.0.0.222\software\AMD\HPL\AMD_HPL.tgz

BIOS settings

• Advanced > ACPI Settings > NUMA Nodes Per Sockets > NPS4
• Advanced > CPU Configuration >SMT Control > Disabled
• Advanced > NB Configuration > Determinism Control > Manual
• Advanced > NB Configuration > Determinism Slider > Power
• Advanced > NB Configuration > cTDP Control > Manual
• Advanced > NB Configuration > cTDP > 240
• Advanced > NB Configuration > Package Power Limit Control > Manual
• Advanced > NB Configuration > Package Power Limit > 240
• Advanced > NB Configuration > Memory Configuration > Memory Clock > 3200Mhz

Dependencies required

To compile openmpi tools you will always need gcc etc, safe time with development tools.

I tested on kernel 4.4.188-1.el7.elrepo to get the results i did with the 7702,7502

yum update -y
yum group install "Development Tools" -y
yum install install vim screen htop epel-release -y
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm 
yum --enablerepo=elrepo-kernel install kernel-lt 
wget https://download.open-mpi.org/release/open-mpi/v2.1/openmpi-2.1.5.tar.gz

Install ICC from Intel`s parallel Studio

add the bin path to bash

source /opt/intel/bin/compilervars.sh intel64

if you wish for this to be persistent between reboots add the above source line to the end of bashrc "vi ~/.bashrc"

compile openmpi 2.1.5

Navigate to the directory thats created after you decompress the openmpi tgz file.

Run the commands below to compile openmpi.

./configure --prefix=/usr/lib64/openmpi-2.1.5
make all install

Using AMD HPL

Set cpupower to performance (if you reboot the system you will need to rerun this command).

cpupower frequency-set -g performance

Would suggest installing screen.

untar AMD_HPL

tar xvf AMD_HPL.tgz

Change directory to "HPL" that will have been created from untar`ing.

Within HPL you will find a number of files but the most important ones will be:

"run_hpl_ccx.sh" and "HPL.dat"

within run_hpl_ccx.sh you may have to edit the absolute location of mpirun since it may not match your system.

You may see dependency errors (libmpi.so.20 missing) if you do not reference openmpi 2.1.5 with it claiming missing lib

open HPL.dat and edit the Ns value to be inline with your memory capacity.

Using HPL calulator to work out what this value should be relative to the system your testi=ng.

Appfiles for AMD HPL =

2x 7302 set for 4 numa

-np 1 ./xhpl_ccx.sh 0-3 4
-np 1 ./xhpl_ccx.sh 4-7 4
-np 1 ./xhpl_ccx.sh 8-11 4
-np 1 ./xhpl_ccx.sh 12-15 4
-np 1 ./xhpl_ccx.sh 16-19 4
-np 1 ./xhpl_ccx.sh 20-23 4
-np 1 ./xhpl_ccx.sh 24-27 4
-np 1 ./xhpl_ccx.sh 28-31 4
-np 1 ./xhpl_ccx.sh 32-35 4
-np 1 ./xhpl_ccx.sh 36-39 4
-np 1 ./xhpl_ccx.sh 40-43 4
-np 1 ./xhpl_ccx.sh 44-47 4
-np 1 ./xhpl_ccx.sh 48-51 4
-np 1 ./xhpl_ccx.sh 52-55 4
-np 1 ./xhpl_ccx.sh 56-59 4
-np 1 ./xhpl_ccx.sh 60-63 4

2x 7402 set for 4 numa

-np 1 ./xhpl_ccx.sh 0-3 4
-np 1 ./xhpl_ccx.sh 4-7 4
-np 1 ./xhpl_ccx.sh 8-11 4
-np 1 ./xhpl_ccx.sh 12-15 4
-np 1 ./xhpl_ccx.sh 16-19 4
-np 1 ./xhpl_ccx.sh 20-23 4
-np 1 ./xhpl_ccx.sh 24-27 4
-np 1 ./xhpl_ccx.sh 28-31 4

Getting HPL to loop for 3 runs (or more)

The absolute path to "run_hpl_ccx.sh" may change depending on where you untar AMD_HPL.tgz so just check that the path is correct here for you.

#! /bin/bash
for i in {1..3}; do /root/HPL/run_hpl_ccx.sh; done