Solarflare SFN8522 - Revision history

Vipul at 12:33, 22 June 2016

2016-06-22T12:33:11Z

Vipul at 12:30, 22 June 2016

2016-06-22T12:30:19Z

Vipul: Changed formatting to add single processor info

2016-06-22T12:11:39Z

Changed formatting to add single processor info

Vipul: Created page with "== Onload Install== === System requierments === * Centos with full updates * Development tools === Installation === scripts/onload_install onload_tool reload
2016-06-20T09:53:58Z

Created page with "== Onload Install== === System requierments === * Centos with full updates * Development tools === Installation === <syntaxhighlight> scripts/onload_install onload_tool reload </bash..."

New page
== Onload Install==

=== System requierments ===

* Centos with full updates
* Development tools

=== Installation ===

<syntaxhighlight>
scripts/onload_install
onload_tool reload
</syntaxhighlight>

== Solar Flare Tests ==

===Pre requisites ===

* centos uptodate
* Development tools

* two servers connected by direct cable
* Solarflare interfaces configured and pingbale
* OpenOnload, netperf and sfnettest installed

=== netperf install ===

netperf can be downloaded from http://www.netperf.org/netperf/

<syntaxhighlight>
tar -zxvf netperf-<version>.tar.gz
cd netperf-<version>
./configure
make install
</syntaxhighlight>

=== sfnettest install ===

sfnettest can be downloaded from http://www.openonload.org

<syntaxhighlight>
tar -zxvf sfnettest-<version>.tgz
cd sfnettest-<version>/src
make
</syntaxhighlight>

=== pre test Config ===

Isolate the CPUs from, SMP balancing by adding teh following to the kernel line in boot.conf:

<syntaxhighlight>
isolcpus=<comma separated cpu list>
</syntaxhighlight>

Stop the services:

<syntaxhighlight>
service cpuspeed stop
ice irqbalance stop
service iptables stop
ethtool -C eth<N> rx-usecs 0 adaptive-rx off
</syntaxhighlight>

Check the BIOS settings so make sure that power saving is disabled

Steer the Interrupts to a CPU core used for the benchmark:

<syntaxhighlight>
# cat /proc/interrupts | grep eth2
33: 0 0 0 0 IR-PCI-MSI-edge eth2-0
34: 0 0 0 0 IR-PCI-MSI-edge eth2-1

# echo 1 > /proc/irq/33/smp_affinity
# echo 2 > /proc/irq/34/smp_affinity
</syntaxhighlight>

== Latency Tests ==

Run sfnt-pingpong on both servers:

<syntaxhighlight>
[root@system-1]# onload --profile=latency taskset -c 3 ./sfnt-pingpong
[root@system-2]# onload --profile=latency taskset -c 3 ./sfnt-pingpong \
--affinity "3;3" udp <system1-ip>
</syntaxhighlight>

=== Latency Results ===

<syntaxhighlight>
size mean min median max %ile stddev iter
0 1675 1553 1668 923157 1760 977 890000
1 1674 1543 1668 9182 1754 43 891000
2 1674 1549 1668 8899 1756 41 891000
4 1673 1539 1667 4036 1752 40 892000
8 1673 1554 1667 11701 1751 40 892000
16 1670 1531 1665 3991 1753 36 893000
32 1704 1576 1699 4001 1785 39 875000
64 1732 1641 1727 11127 1810 37 861000
128 1841 1734 1837 4406 1936 44 811000
256 2026 1902 2023 4446 2126 48 737000
512 2264 2143 2260 4819 2365 48 660000
1024 2881 2769 2876 5350 2992 56 519000
1472 3439 3324 3433 5993 3557 57 435000
</syntaxhighlight>

== Netperf Tests ==

To accelerate an application with the Onload user-level transport, simply invoke the application on the command line as normal, prepended with "onload".

ie. Instead of:

netperf -t TCP_RR -H myserver

do this:

onload netperf -t TCP_RR -H myserver
<syntaxhighlight>
oo:netperf[3420]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [0]
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.20 () port 0 AF_INET : first burst 0
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec

16384 87380 1 1 10.00 67595.62
16384 87380
</syntaxhighlight>

and tuned for best latency:

onload -p latency netperf -t TCP_RR -H myserver

<syntaxhighlight>
oo:netperf[3494]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [5]
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.20 () port 0 AF_INET : first burst 0
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec

16384 87380 1 1 10.00 78216.20
16384 87380

</syntaxhighlight>
== sqmr ==

← Older revision		Revision as of 12:30, 22 June 2016
Line 1:		Line 1:
		+	== Single Processor==
		+
		+	=== Latency Results ===
		+
		+	* affinity : 3;3 (3 cores used on both server and client)
		+	<syntaxhighlight>
		+	size mean min median max %ile stddev iter
		+	0 1327 1250 1279 710001 1512 715 1000000
		+	1 1275 1250 1274 2916 1298 22 1000000
		+	2 1276 1251 1275 7708 1299 21 1000000
		+	4 1276 1252 1275 3716 1299 21 1000000
		+	8 1276 1250 1275 3101 1299 20 1000000
		+	16 1275 1251 1274 2910 1298 22 1000000
		+	32 1302 1274 1300 2970 1328 24 1000000
		+	64 1331 1304 1330 2980 1357 21 1000000
		+	128 1405 1376 1404 3259 1431 24 1000000
		+	256 1553 1523 1552 3319 1586 24 961000
		+	512 1692 1663 1690 10515 1732 29 882000
		+	1024 2288 2254 2287 4082 2337 24 653000
		+	1472 2800 2764 2798 6632 2860 34 534000
		+
		+	</syntaxhighlight>
		+
		+	* affinity : 2;2 (2 cores used on both server and client)
		+	<syntaxhighlight>
		+	size mean min median max %ile stddev iter
		+	0 1320 1248 1274 72090 1504 151 1000000
		+	1 1270 1246 1269 72127 1293 139 1000000
		+	2 1270 1247 1269 72740 1293 99 1000000
		+	4 1270 1247 1269 72026 1293 99 1000000
		+	8 1270 1245 1269 71577 1293 121 1000000
		+	16 1269 1244 1267 72496 1291 100 1000000
		+	32 1292 1265 1290 72119 1317 100 1000000
		+	64 1321 1296 1319 71428 1346 138 1000000
		+	128 1399 1375 1398 72579 1427 100 1000000
		+	256 1546 1519 1544 72845 1579 143 965000
		+	512 1689 1663 1687 71901 1729 105 884000
		+	1024 2288 2252 2285 72943 2334 171 653000
		+	1472 2802 2768 2800 72621 2858 136 534000
		+
		+
		+	</syntaxhighlight>
		+
		+	* affinity : 1;1 (1 core used on both server and client)
		+	<syntaxhighlight>
		+	size mean min median max %ile stddev iter
		+	0 1338 1262 1290 136203 1535 222 1000000
		+	1 1283 1258 1282 133695 1306 133 1000000
		+	2 1280 1255 1279 134346 1302 134 1000000
		+	4 1280 1255 1279 132151 1302 185 1000000
		+	8 1280 1254 1279 131724 1302 131 1000000
		+	16 1285 1260 1284 132821 1307 133 1000000
		+	32 1309 1281 1308 132820 1333 186 1000000
		+	64 1338 1312 1337 132945 1364 133 1000000
		+	128 1418 1391 1417 132810 1445 133 1000000
		+	256 1556 1529 1555 132146 1590 134 959000
		+	512 1700 1670 1698 133154 1740 199 878000
		+	1024 2292 2259 2290 133706 2343 165 652000
		+	1472 2808 2770 2805 132879 2868 252 533000
		+
		+	</syntaxhighlight>
		+
		+	=== Netperf Tests ===
		+
		+	* onload netperf -t TCP_RR -H myserver
		+	<syntaxhighlight>
		+	oo:netperf[3420]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [0]
		+	MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.20 () port 0 AF_INET : first burst 0
		+	Local /Remote
		+	Socket Size Request Resp. Elapsed Trans.
		+	Send Recv Size Size Time Rate
		+	bytes Bytes bytes bytes secs. per sec
		+
		+	16384 87380 1 1 10.00 67595.62
		+	16384 87380
		+	</syntaxhighlight>
		+
		+	* onload -p latency netperf -t TCP_RR -H myserver
		+
		+	<syntaxhighlight>
		+	oo:netperf[3494]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [5]
		+	MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.20 () port 0 AF_INET : first burst 0
		+	Local /Remote
		+	Socket Size Request Resp. Elapsed Trans.
		+	Send Recv Size Size Time Rate
		+	bytes Bytes bytes bytes secs. per sec
		+
		+	16384 87380 1 1 10.00 78216.20
		+	16384 87380
		+
		+	</syntaxhighlight>
		+
		+
		+
	== Dual Processor==		== Dual Processor==

@@ Line 65: / Line 65: @@
   *  onload netperf -t TCP_RR -H myserver
 <syntaxhighlight>
-oo:netperf[3420]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [0]
+oo:netperf[13975]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [7]
-MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.20 () port 0 AF_INET : first burst 0
+MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.10 () port 0 AF_INET : first burst 0
 Local /Remote
 Socket Size   Request  Resp.   Elapsed  Trans.
@@ Line 72: / Line 72: @@
 bytes  Bytes  bytes    bytes   secs.    per sec
-  87380  1        1       10.00    67595.62
+  87380  1        1       10.00    97391.17
   87380
-</syntaxhighlight>
   * onload -p latency netperf -t TCP_RR -H myserver
 <syntaxhighlight>
-oo:netperf[3494]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [5]
+oo:netperf[13993]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [1]
-MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.20 () port 0 AF_INET : first burst 0
+MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.10 () port 0 AF_INET : first burst 0
 Local /Remote
 Socket Size   Request  Resp.   Elapsed  Trans.
@@ Line 86: / Line 86: @@
 bytes  Bytes  bytes    bytes   secs.    per sec
-  87380  1        1       10.00    78216.20
+  87380  1        1       10.00    149533.29
   87380
 </syntaxhighlight>

@@ Line 1: / Line 1: @@
-== Onload Install==
+== Dual Processor==
 === Latency Results ===
 <syntaxhighlight>
 	size	mean	min	median	max	%ile	stddev	iter
@@ Line 108: / Line 21: @@
 </syntaxhighlight>
-== Netperf Tests ==
+=== Netperf Tests ===
-do this:
+ *  onload netperf -t TCP_RR -H myserver
 <syntaxhighlight>
 oo:netperf[3420]: Using OpenOnload 201604-8000preview Copyright 2006-2016 Solarflare Communications, 2002-2005 Level 5 Networks [0]
@@ Line 131: / Line 36: @@
 </syntaxhighlight>
-and tuned for best latency:
+ * onload -p latency netperf -t TCP_RR -H myserver
 <syntaxhighlight>
@@ Line 147: / Line 50: @@
 </syntaxhighlight>