Solarflare SFN7122F
Onload Install
System requierments
- Centos with full updates
- Development tools
Installation
scripts/onload_install
onload_tool reloadSolar Flare Tests
Pre requisites
- centos uptodate
- Development tools
- two servers connected by direct cable
- Solarflare interfaces configured and pingbale
- OpenOnload, netperf and sfnettest installed
netperf install
netperf can be downloaded from http://www.netperf.org/netperf/
tar -zxvf netperf-<version>.tar.gz
cd netperf-<version>
./configure
make install
sfnettest install
sfnettest can be downloaded from http://www.openonload.org
tar -zxvf sfnettest-<version>.tgz
cd sfnettest-<version>/src
makepre test Config
Isolate the CPUs from, SMP balancing by adding teh following to the kernel line in boot.conf:
isolcpus=<comma separated cpu list>Stop the services:
service cpuspeed stop
ice irqbalance stop
service iptables stop
ethtool -C eth<N> rx-usecs 0 adaptive-rx offCheck the BIOS settings so make sure that power saving is disabled
Steer the Interrupts to a CPU core used for the benchmark:
# cat /proc/interrupts | grep eth2
33: 0 0 0 0 IR-PCI-MSI-edge eth2-0
34: 0 0 0 0 IR-PCI-MSI-edge eth2-1
# echo 1 > /proc/irq/33/smp_affinity
# echo 2 > /proc/irq/34/smp_affinity
Latency Tests
Run sfnt-pingpong on both servers:
[root@system-1]# onload --profile=latency taskset -c 3 ./sfnt-pingpong
[root@system-2]# onload --profile=latency taskset -c 3 ./sfnt-pingpong \
--affinity "3;3" udp <system1-ip>
Latency Results
# size mean min median max %ile stddev iter
0 2503 2154 2487 18825 2624 69 596000
1 2502 2223 2487 17217 2612 62 596000
2 2503 2176 2487 18864 2616 59 596000
4 2503 2168 2487 13717 2618 58 596000
8 2505 2187 2490 14830 2623 57 596000
16 2499 2229 2479 11746 2603 58 597000
32 2539 2267 2518 9174 2688 62 588000
64 2569 2085 2546 7050 2712 59 581000
128 2693 2382 2678 7312 2857 71 554000
256 2849 2566 2833 7247 3019 70 524000
512 3004 2714 2986 7648 3180 75 497000
1024 3698 3318 3704 8298 3872 86 404000
1472 4191 3862 4173 10687 4386 86 357000
1473 10016 8907 9509 14944 13021 1059 150000
2048 10425 9258 9872 15001 13615 1168 144000
4096 13375 12061 13005 16923 15385 856 112000
8192 21197 18881 20913 29207 24453 1130 71000
16384 36503 33472 36381 42301 39757 1177 42000
32768 65624 61013 65539 71654 69432 1516 23000Netperf Tests
To accelerate an application with the Onload user-level transport, simply invoke the application on the command line as normal, prepended with "onload".
ie. Instead of:
netperf -t TCP_RR -H myserver
do this:
onload netperf -t TCP_RR -H myserver
and tuned for best latency:
onload -p latency netperf -t TCP_RR -H myserver
Netperf Results
| Netperf | 12,333.00 |
| Onload Netperf | 68004.42 |
| Onload best latency | 97869.02 |
Iperf Results
Serial Runs
| Iperf | 1 | 9.42 |
| Onload iperf | 1 | 9.41 |
| Onload best latency | 1 | 9.41 |
Parallel Runs
| Iperf | 10 | 9.42 |
| Onload iperf | 10 | 9.41 |
| Onload best latency | 10 | 9.41 |
sqmr
Solarflare results on the left. Ethernet on the right. Onload fails with MPI.
# msgsize iters time msgs/sec MiB/sec msgs/sec MiB/se # msgsize iters time msgs/sec MiB/sec msgs/sec MiB/se
0 4096 0.11 75280.97 0.00 75280.97 0.0 | 0 4096 0.11 75387.01 0.00 75387.01 0.0
1 3277 0.08 81653.50 0.08 81653.50 0.0 | 1 3277 0.08 81618.83 0.08 81618.83 0.0
2 2622 0.06 81574.19 0.16 81574.19 0.1 | 2 2622 0.06 81633.83 0.16 81633.83 0.1
4 2098 0.05 81634.33 0.31 81634.33 0.3 | 4 2098 0.05 81637.36 0.31 81637.36 0.3
8 1679 0.04 81589.99 0.62 81589.99 0.6 | 8 1679 0.04 81657.64 0.62 81657.64 0.6
16 1344 0.03 81632.68 1.25 81632.68 1.2 | 16 1344 0.03 81635.04 1.25 81635.04 1.2
32 1076 0.03 81635.79 2.49 81635.79 2.4 | 32 1076 0.03 81632.83 2.49 81632.83 2.4
64 861 0.02 81716.47 4.99 81716.47 4.9 | 64 861 0.02 81591.84 4.98 81591.84 4.9
128 689 0.02 81528.96 9.95 81528.96 9.9 | 128 689 0.02 81687.97 9.97 81687.97 9.9
256 552 0.01 81633.76 19.93 81633.76 19.9 | 256 552 0.01 81626.56 19.93 81626.56 19.9
512 442 0.01 76950.12 37.57 76950.12 37.5 512 442 0.01 76950.12 37.57 76950.12 37.5
1024 354 0.02 42666.20 41.67 42666.20 41.6 | 1024 354 0.02 42661.29 41.66 42661.29 41.6
2048 284 0.03 16303.61 31.84 16303.61 31.8 | 2048 284 0.03 16234.61 31.71 16234.61 31.7
4096 228 0.03 16716.80 65.30 16716.80 65.3 | 4096 228 0.03 16785.61 65.57 16785.61 65.5
8192 183 0.01 24906.15 194.58 24906.15 194.5 | 8192 183 0.02 16558.07 129.36 16558.07 129.3
16384 147 0.02 12678.91 198.11 12678.91 198.1 | 16384 147 0.02 12699.28 198.43 12699.28 198.4
32768 118 0.03 9290.67 290.33 9290.67 290.3 | 32768 118 0.03 9189.67 287.18 9189.67 287.1
65536 95 0.07 2743.08 171.44 2743.08 171.4 | 65536 95 0.07 2725.74 170.36 2725.74 170.3
131072 76 0.08 1813.43 226.68 1813.43 226.6 | 131072 76 0.08 1792.37 224.05 1792.37 224.0
262144 61 0.09 1314.43 328.61 1314.43 328.6 | 262144 61 0.10 1258.62 314.65 1258.62 314.6
524288 49 0.13 726.65 363.32 726.65 363.3 | 524288 49 0.13 729.33 364.67 729.33 364.6
1048576 40 0.21 387.84 387.84 387.84 387.8 | 1048576 40 0.41 196.70 196.70 196.70 196.7
2097152 32 0.55 116.48 232.95 116.48 232.9 | 2097152 32 0.76 84.01 168.02 84.01 168.0
4194304 26 0.51 102.45 409.79 102.45 409.7 | 4194304 26 0.89 58.12 232.50 58.12 232.5