Exablaze: Setup and Benchmark ExaNIC

= Check the PCI NIC locality

• Ensure that we are pinning the processes to the cores with the PCI controller

[root@hft2 ~]# cat /sys/class/net/eth5/device/local_cpulist 
8-15

• Make sure the boot args specify isolcpus for the correct cores to isolated from the kernel scheduler.

title CentOS (2.6.32-431.20.3.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-431.20.3.el6.x86_64 ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll pcie_aspm=performance mce=ignore_ce isolcpus=3,5,10,12
        initrd /initramfs-2.6.32-431.20.3.el6.x86_64.img

• Check this at run time with:

[root@hft2 ~]# cat /proc/cmdline 
ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=133M@0M rd_LVM_LV=VolGroup/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll pcie_aspm=performance mce=ignore_ce isolcpus=3,5,10,12

• Then pin the tasks on server/client to the correct cores (in this instance 12, as our range was 8-15, 12 just selected at random)

# on the server side
[root@hft2 ~]# exasock taskset -c 12 sockperf sr
sockperf: == version #2.5.241 == 
sockperf: [SERVER] listen on:
[ 0] IP = 0.0.0.0         PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 2827] using recvfrom() to block on socket(s)

• Then on the client side

# on the client side
[root@hft1 ~]# exasock taskset -c 12 sockperf pp -i 192.168.1.8 -t 5 -m 12
sockperf: == version #2.5.241 == 
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 192.168.1.8     PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: Starting test...
sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=5.100 sec; SentMessages=2437259; ReceivedMessages=2437258
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=5.000 sec; SentMessages=2391308; ReceivedMessages=2391308
sockperf: ====> avg-lat=  1.034 (std-dev=0.019)
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 1.034 usec
sockperf: Total 2391308 observations; each percentile contains 23913.08 observations
sockperf: ---> <MAX> observation =   11.844
sockperf: ---> percentile  99.99 =    1.501
sockperf: ---> percentile  99.90 =    1.377
sockperf: ---> percentile  99.50 =    1.067
sockperf: ---> percentile  99.00 =    1.053
sockperf: ---> percentile  95.00 =    1.045
sockperf: ---> percentile  90.00 =    1.041
sockperf: ---> percentile  75.00 =    1.037
sockperf: ---> percentile  50.00 =    1.033
sockperf: ---> percentile  25.00 =    1.029
sockperf: ---> <MIN> observation =    1.013