Benchmarking: Stream (Memory Bandwidth)

STREAM: The STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels.

• Note: Ensure power saving features are disabled, we need max clock speed to prevent fluctuations in performance:
  • /etc/init.d/cpuspeed stop

Get the source

• Main STREAM website: http://www.cs.virginia.edu/stream/
• Pull the latest copy of STREAM from:

  # (v 5.10 at the time of edit)
  wget http://www.cs.virginia.edu/stream/FTP/Code/stream.c

Compile

• Can use either Intel or GCC to build/compile
• Ensure you build for multi-threaded runs (-fopenmp (gcc) -openmp (icc)
• For large array sizes, include -mcmodel=medium
• Noticed best performance using Intel ICC

Intel

  icc -O3 -static -openmp stream.c -o stream_icc

GCC

• GCC typically gave the worst performance in the limited tests we performed (probably better optimisation flags required, but not well documented for STREAM)

  gcc -O3 -fopenmp stream.c -o stream_gcc

Open 64

• Below are optimisations for the AMD 6300 Arch

  opencc -march=bdver1 -mp -Ofast -LNO:simd=2 -WOPT:sib=on  \
     -LNO:prefetch=2:pf2=0 -CG:use_prefetchnta=on -LNO:prefetch_ahead=4 -DSTREAM_ARRAY_SIZE=30000000 \
     -DNTIMES=30 -DOFFSET=1840 stream.c -o stream_occ

Run

• Vary the number of threads used by using: export OMP_NUM_THREADS=32

Intel

  export OMP_NUM_THREADS=16
  export KMP_AFFINITY=compact
  ./stream_icc

GCC

  export GOMP_CPU_AFFINITY="0 1 2 ..."
  ./stream_gcc

Open64

• Below is the recommend best for AMD 6300 arch
• Peak memory bandwidth is achieved when STREAM is run on three cores of each NUMA node. For example, the following run shows that the same system is capable of achieving STREAM 5% better than when using all cores.

  # assuming 32 core system
  export O64_OMP_AFFINITY=”TRUE”
  export O64_OMP_AFFINITY_MAP=”2,4,6,10,12,14,18,20,22,26,28,30”
  export OMP_NUM_THREADS=12
  ./stream

Results

Numascale 4x AMD 6380 Dual-Socket Servers

  # build
  opencc -Ofast -march=bdver2 -mp -mcmodel=medium -LNO:simd=2 -WOPT:sib=on -LNO:prefetch=2:pf2=0 \
              -CG:use_prefetchnta=on -LNO:prefetch_ahead=4 -DSTREAM_ARRAY_SIZE=1500000000ULL stream.c -o stream_c.exe.open64.33g
  # run (on all 128 cores)
  OMP_NUM_THREADS=128 O64_OMP_AFFINITY=true O64_OMP_AFFINITY_MAP=$(seq -s, 0 1 127) ./stream_c.exe.open64.33g

  Function    Best Rate MB/s  Avg time     Min time     Max time
  Copy:          213159.1     0.118355     0.112592     0.143757
  Scale:         211838.0     0.126541     0.113294     0.153623
  Add:           199420.7     0.206908     0.180523     0.310094
  Triad:         200751.6     0.189908     0.179326     0.209648

  # run on every second core (one per memory controller)
  OMP_NUM_THREADS=64 O64_OMP_AFFINITY=true O64_OMP_AFFINITY_MAP=$(seq -s, 0 2 127) ./stream_c.exe.open64.33g 
  
  Function    Best Rate MB/s  Avg time     Min time     Max time
  Copy:          221879.3     0.110662     0.108167     0.113518
  Scale:         234276.7     0.108583     0.102443     0.121964
  Add:           213480.2     0.171661     0.168634     0.180814
  Triad:         215921.8     0.170627     0.166727     0.177991