http://wiki.define-technology.com/mediawiki-1.35.0/index.php?action=history&feed=atom&title=Solarflare_Latency_Testing_for_JSE_on_vScaler Solarflare Latency Testing for JSE on vScaler - Revision history 2026-05-04T16:10:37Z Revision history for this page on the wiki MediaWiki 1.35.0 http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Solarflare_Latency_Testing_for_JSE_on_vScaler&diff=29644&oldid=prev David: Created page with "<syntaxhighlight> Setting up SFC # install latest driver yum -y groupinstall 'Development tools' yum -y install kernel-devel wget numactl rpmbuild --rebuild ./sfc-4.15.2.1001-1.src.rp..." 2019-12-20T12:47:17Z <p>Created page with &quot;&lt;syntaxhighlight&gt; Setting up SFC # install latest driver yum -y groupinstall &#039;Development tools&#039; yum -y install kernel-devel wget numactl rpmbuild --rebuild ./sfc-4.15.2.1001-1.src.rp...&quot;</p> <p><b>New page</b></p><div>&lt;syntaxhighlight&gt;<br /> Setting up SFC <br /> # install latest driver<br /> yum -y groupinstall &#039;Development tools&#039; <br /> yum -y install kernel-devel wget numactl <br /> rpmbuild --rebuild ./sfc-4.15.2.1001-1.src.rpm <br /> rpm -Uvh /root/rpmbuild/RPMS/x86_64/kernel-module-sfc-RHEL7-3.10.0-957.21.3.el7.x86_64-4.15.2.1001-1.x86_64.rpm<br /> <br /> [root@solarflare01 Solarflare]# modinfo sfc | grep version: <br /> version: 4.15.2.1001<br /> <br /> [root@solarflare01 Solarflare]# ethtool -i enp139s0f0 | grep version <br /> version: 4.15.2.1001<br /> <br /> (drivers were detected ok on centos 7.6)<br /> rpm -ivh sfutils-7.5.4.1001-1.x86_64.rpm <br /> <br /> # hmm not sure above is needed - now i get to install openonload and it covers the driver update as well <br /> # remove the installed driver from above<br /> rpm -e &#039;kernel-module-sfc-RHEL7-3.10.0-957.21.3.el7.x86_64-4.15.2.1001-1.x86_64&#039;<br /> rmmod sfc<br /> <br /> # install openonload <br /> wget https://openonload.org/download/openonload-201811-u1.tgz<br /> tar zxvf openonload-201811-u1.tgz <br /> ./openonload-201811-u1/scripts/onload_install wget https://www.openonload.org/download/sfnettest/sfnettest-1.5.0.tgz<br /> <br /> # unload for pci passthrough<br /> ./onload_tool unload <br /> <br /> # get the sfntest too<br /> wget https://www.openonload.org/download/sfnettest/sfnettest-1.5.0.tgz<br /> cd sfnettest-1.5.0/src<br /> make <br /> <br /> # note to check and see what firmware is running: just sfboot<br /> # to factory reset: sfboot -i enp139s0f0 --factory-reset (x2 card)<br /> <br /> # change the firmware to full feature <br /> [root@solarflare01 Solarflare]# sfboot --adapter=enp139s0f0 firmware-variant=full-feature<br /> <br /> # set the low latency firmware <br /> [root@solarflare01 ~]# sfboot --adapter=enp139s0f0 firmware-variant=ultra-low-latency<br /> <br /> # note: cold power cycle just run<br /> ipmitool power cycle <br /> <br /> # tuning <br /> # what numa node is the interface connected to<br /> cat /sys/class/net/enp139s0f0/device/numa_node<br /> <br /> # what cores are in that numa node <br /> <br /> cat /sys/devices/system/node/node1/cpulist <br /> <br /> # run latency tests (based on ultra-low-latency profile)<br /> <br /> # on one node run<br /> onload --profile=latency taskset -c 8 ./sfnt-pingpong<br /> <br /> # then on the other node<br /> [root@solarflare02 src]# onload --profile=latency taskset -c 8 ./sfnt-pingpong --affinity &quot;1;1&quot; tcp 192.168.10.5<br /> oo:sfnt-pingpong[21133]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [0]<br /> # cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.5<br /> # version: 1.5.0<br /> # src: 8dc3b027d85b28bedf9fd731362e4968<br /> # date: Fri 19 Jul 14:27:10 BST 2019<br /> # uname: Linux solarflare02.testing 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux<br /> # cpu: model name : Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz<br /> # lspci: 05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 8b:00.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)<br /> # lspci: 8b:00.1 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)<br /> # enp139s0f0: driver: sfc<br /> # enp139s0f0: version: 4.15.3.1011<br /> # enp139s0f0: bus-info: 0000:8b:00.0<br /> # enp139s0f1: driver: sfc<br /> # enp139s0f1: version: 4.15.3.1011<br /> # enp139s0f1: bus-info: 0000:8b:00.1<br /> # enp5s0f0: driver: igb<br /> # enp5s0f0: version: 5.4.0-k<br /> # enp5s0f0: bus-info: 0000:05:00.0<br /> # enp5s0f1: driver: igb<br /> # enp5s0f1: version: 5.4.0-k<br /> # enp5s0f1: bus-info: 0000:05:00.1<br /> # ram: MemTotal: 65693408 kB<br /> # tsc_hz: 2099969430<br /> # LD_PRELOAD=libonload.so<br /> # onload_version=201811-u1<br /> # EF_TCP_FASTSTART_INIT=0<br /> # EF_POLL_USEC=100000<br /> # EF_TCP_FASTSTART_IDLE=0<br /> # server LD_PRELOAD=libonload.so<br /> # percentile=99<br /> #<br /> # size mean min median max %ile stddev iter<br /> 1 1768 1676 1719 50581 2166 158 842000<br /> 2 1729 1680 1720 21333 1839 84 862000<br /> 4 1730 1681 1720 20046 1844 92 861000<br /> 8 1731 1682 1722 20967 1850 92 861000<br /> 16 1739 1684 1730 19944 1867 89 857000<br /> 32 1752 1694 1740 10220 1877 90 851000<br /> 64 1801 1745 1791 21847 1931 97 828000<br /> 128 1893 1824 1875 14862 2029 98 788000<br /> 256 2074 1930 2069 27097 2236 111 719000<br /> 512 2300 2176 2289 11293 2469 103 649000<br /> 1024 2606 2475 2585 23790 2919 141 573000<br /> 2048 3857 3521 3829 13734 4398 191 388000<br /> 4096 4664 4401 4622 22394 5181 186 321000<br /> 8192 6547 6214 6491 15511 7508 240 229000<br /> 16384 9997 9540 9946 28038 11050 293 150000<br /> 32768 16476 15478 16419 34062 18176 432 91000<br /> 65536 29156 27571 29151 54446 31152 647 52000<br /> <br /> <br /> <br /> # redo with the standard full feature firmware <br /> <br /> $ sfboot firmware-variant=full-feature<br /> $ ipmitool power cycle <br /> <br /> [root@solarflare01 src]# sfboot | grep -i variant <br /> Firmware variant Full feature / virtualization<br /> Firmware variant Full feature / virtualization<br /> <br /> <br /> # size mean min median max %ile stddev iter<br /> 1 2248 2022 2071 4200847 3239 6349 663000<br /> 2 2209 2022 2075 889552 3158 3298 675000<br /> 4 2207 2018 2078 865345 3132 3096 676000<br /> 8 2199 2035 2093 4827169 2692 6848 678000<br /> 16 2241 2045 2092 219934 3027 3539 666000<br /> 32 2230 2038 2102 4414673 3319 6302 669000<br /> 64 2244 2064 2128 4589940 3179 6427 665000<br /> 128 2269 2105 2170 185278 3305 1587 658000<br /> 256 2522 2220 2366 4359659 4037 6899 592000<br /> 512 2588 2329 2460 1322388 3893 3480 577000<br /> 1024 2822 2529 2614 4219115 8031 6869 530000<br /> 2048 3973 3566 3742 4108063 8391 7989 377000<br /> 4096 5064 4480 4745 270765 11188 4253 296000<br /> 8192 7103 6283 6700 4363507 13410 11234 211000<br /> 16384 11054 9628 10243 1432292 18087 10285 136000<br /> 32768 18228 15645 16971 4225292 34181 16490 83000<br /> 65536 30909 27770 29450 4078629 66407 22086 49000<br /> <br /> <br /> <br /> <br /> # lets setup 8 PF (physical functions) with solarflare NICs for SR-IOV <br /> <br /> sfboot switch-mode=pfiov pf-count=4<br /> <br /> grubby --update-kernel=ALL --args=&quot;intel_iommu=on&quot;<br /> grubby --info=ALL<br /> <br /> # reboot<br /> <br /> # results within a VM <br /> [root@sfn8 src]# onload --profile=latency ./sfnt-pingpong --affinity &quot;1;1&quot; tcp 192.168.10.1<br /> onload: Note: Disabling CTPIO cut-through because only adapters running at 10GbE benefit from it<br /> oo:sfnt-pingpong[20195]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [0]<br /> # cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.1<br /> # version: 1.5.0<br /> # src: 8dc3b027d85b28bedf9fd731362e4968<br /> # date: Tue Jul 30 23:26:52 UTC 2019<br /> # uname: Linux sfn8.novalocal 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux<br /> # cpu: model name : Intel Core Processor =191aa(Broadwell)<br /> sh: /sbin/lspci: No such file or directory<br /> # eth0: driver: virtio_net<br /> # eth0: version: 1.0.0<br /> # eth0: bus-info: 0000:00:03.0<br /> # eth1: driver: sfc<br /> # eth1: version: 4.15.3.1011<br /> # eth1: bus-info: 0000:00:05.0<br /> # ram: MemTotal: 3880088 kB <br /> # tsc_hz: 2099966200<br /> # LD_PRELOAD=libonload.so<br /> # onload_version=201811-u1<br /> # EF_TCP_FASTSTART_INIT=0<br /> # EF_CTPIO_SWITCH_BYPASS=1<br /> # EF_POLL_USEC=100000<br /> # EF_TCP_FASTSTART_IDLE=0<br /> # EF_EPOLL_MT_SAFE=1<br /> # server LD_PRELOAD=libonload.so <br /> # percentile=99<br /> #<br /> # size mean min median max %ile stddev iter<br /> 1 1942 1725 1833 273704 4923 802 766000<br /> 2 1900 1727 1816 657833 4651 1035 783000<br /> 4 1891 1735 1817 402257 4585 849 787000<br /> 8 1860 1666 1803 223816 4601 775 800000<br /> 16 1785 1638 1718 773823 4202 1102 834000<br /> 32 1836 1661 1745 157788 4279 649 811000<br /> 64 1903 1717 1789 425864 4582 897 782000<br /> 128 2052 1860 1954 537661 5154 978 726000<br /> 256 2247 1994 2147 75286 5532 771 663000<br /> 512 2541 2240 2405 307319 6814 939 587000<br /> 1024 2642 2335 2545 24229 5824 701 565000<br /> 2048 3539 3210 3433 2107848 6482 3312 422000<br /> 4096 4558 4139 4400 443807 8669 1140 328000<br /> 8192 7038 6009 6652 2194215 13286 4967 213000<br /> 16384 10645 9503 10182 291544 18314 1944 141000<br /> 32768 15752 14786 15096 502919 22343 2251 96000<br /> 65536 29551 26566 28286 352787 40923 3319 51000<br /> <br /> # lets get the ID&#039;s of the devices to pass-through <br /> [root@solarflare02 ~]# lspci -nn | grep -i sol<br /> 8b:00.0 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> 8b:00.1 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> 8b:00.2 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> 8b:00.3 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> 8b:00.4 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> 8b:00.5 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> 8b:00.6 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> 8b:00.7 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)<br /> <br /> # Nova conf<br /> (virtualenv) [root@solarflare01 ~]# cat /etc/kolla/config/nova.conf <br /> [filter_scheduler]<br /> enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter<br /> available_filters=nova.scheduler.filters.all_filters<br /> <br /> [pci]<br /> passthrough_whitelist={&quot;vendor_id&quot;:&quot;1924&quot;,&quot;product_id&quot;:&quot;0b03&quot;}<br /> alias={&quot;name&quot;:&quot;SolarFlareNIC&quot;,&quot;vendor_id&quot;:&quot;1924&quot;,&quot;product_id&quot;:&quot;0b03&quot;}<br /> <br /> # reconfigure <br /> kolla-ansible -i &lt;blah&gt; reconfigure<br /> <br /> # flavor setup <br /> openstack flavor create --public --ram 4096 --disk 10 --vcpus 4 m1.large.sfn<br /> openstack flavor set m1.large.sfn --property pci_passthrough:alias=&#039;SolarFlareNIC:1&#039;<br /> <br /> # Get centos<br /> wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1905.qcow2.xz<br /> unxz CentOS-7-x86_64-GenericCloud-1905.qcow2.xz<br /> openstack image create --public --file ./CentOS-7-x86_64-GenericCloud-1905.qcow2 --disk-format qcow2 centos7<br /> <br /> # spin up VM<br /> openstack server create --image centos7 --flavor m1.large.sfn --key-name mykey --network demo-net sfn1<br /> <br /> openstack floating ip create public1<br /> openstack floating ip list<br /> <br /> openstack server add floating ip sfn1 172.28.125.151 <br /> openstack server list<br /> # repeat for all <br /> <br /> # mac addresses for solarflare01<br /> 22: enp139s0f1: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:51 brd ff:ff:ff:ff:ff:ff<br /> 23: enp139s0f2: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:52 brd ff:ff:ff:ff:ff:ff ** sfn1<br /> 24: enp139s0f3: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:53 brd ff:ff:ff:ff:ff:ff<br /> 25: enp139s0f4: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:54 brd ff:ff:ff:ff:ff:ff<br /> 26: enp139s0f5: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:55 brd ff:ff:ff:ff:ff:ff<br /> 56: enp139s0f6: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:56 brd ff:ff:ff:ff:ff:ff<br /> 57: enp139s0f7: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:57 brd ff:ff:ff:ff:ff:ff<br /> <br /> # mac addresses for solarflare02<br /> 2: enp139s0f0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc mq state UP group default qlen 1000<br /> link/ether 00:0f:53:65:13:10 brd ff:ff:ff:ff:ff:ff **<br /> 4: enp139s0f1: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:11 brd ff:ff:ff:ff:ff:ff<br /> 5: enp139s0f2: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:12 brd ff:ff:ff:ff:ff:ff ** sfn8<br /> 6: enp139s0f3: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:13 brd ff:ff:ff:ff:ff:ff<br /> 7: enp139s0f4: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:14 brd ff:ff:ff:ff:ff:ff<br /> 8: enp139s0f5: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:15 brd ff:ff:ff:ff:ff:ff<br /> 49: enp139s0f6: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:16 brd ff:ff:ff:ff:ff:ff<br /> 50: enp139s0f7: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN group default qlen 1000<br /> link/ether 00:0f:53:65:13:17 brd ff:ff:ff:ff:ff:ff<br /> <br /> <br /> # updated on the LL2 cascade lake box (based on SFC9220 prev)<br /> [root@solarflare02-ll2 src]# onload --profile=latency taskset -c 12 ./sfnt-pingpong --affinity &quot;1;1&quot; tcp 192.168.10.5<br /> oo:sfnt-pingpong[27963]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [0]<br /> # cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.5<br /> # version: 1.5.0<br /> # src: 8dc3b027d85b28bedf9fd731362e4968<br /> # date: Thu 1 Aug 16:00:30 BST 2019<br /> # uname: Linux solarflare02-ll2.internal 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux<br /> # cpu: model name : Intel(R) Xeon(R) Gold 6246 CPU @ 3.30GHz<br /> # lspci: 3b:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 3b:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 3b:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 3b:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 5e:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)<br /> # lspci: 5e:00.1 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)<br /> # lspci: 86:00.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)<br /> # lspci: 86:00.1 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)<br /> # enp134s0f0: driver: sfc<br /> # enp134s0f0: version: 4.15.3.1011<br /> # enp134s0f0: bus-info: 0000:86:00.0<br /> # enp134s0f1: driver: sfc<br /> # enp134s0f1: version: 4.15.3.1011<br /> # enp134s0f1: bus-info: 0000:86:00.1<br /> # enp59s0f0: driver: igb<br /> # enp59s0f0: version: 5.4.0-k<br /> # enp59s0f0: bus-info: 0000:3b:00.0<br /> # enp59s0f1: driver: igb<br /> # enp59s0f1: version: 5.4.0-k<br /> # enp59s0f1: bus-info: 0000:3b:00.1<br /> # enp59s0f2: driver: igb<br /> # enp59s0f2: version: 5.4.0-k<br /> # enp59s0f2: bus-info: 0000:3b:00.2<br /> # enp59s0f3: driver: igb<br /> # enp59s0f3: version: 5.4.0-k<br /> # enp59s0f3: bus-info: 0000:3b:00.3<br /> # enp94s0f0: driver: sfc<br /> # enp94s0f0: version: 4.15.3.1011<br /> # enp94s0f0: bus-info: 0000:5e:00.0<br /> # enp94s0f1: driver: sfc<br /> # enp94s0f1: version: 4.15.3.1011<br /> # enp94s0f1: bus-info: 0000:5e:00.1<br /> # ram: MemTotal: 196463288 kB<br /> # tsc_hz: 3366144400<br /> # LD_PRELOAD=libonload.so<br /> # onload_version=201811-u1<br /> # EF_TCP_FASTSTART_INIT=0<br /> # EF_POLL_USEC=100000<br /> # EF_TCP_FASTSTART_IDLE=0<br /> # server LD_PRELOAD=libonload.so<br /> # percentile=99<br /> #<br /> # size mean min median max %ile stddev iter<br /> 1 1426 1398 1421 39777 1582 67 1000000<br /> 2 1428 1396 1422 18082 1584 45 1000000<br /> 4 1428 1398 1422 18566 1582 47 1000000<br /> 8 1433 1405 1427 5724 1592 43 1000000<br /> 16 1439 1412 1434 17545 1581 42 1000000<br /> 32 1455 1428 1450 6005 1601 35 1000000<br /> 64 1497 1466 1492 17158 1603 41 996000<br /> 128 1574 1541 1570 18502 1655 38 947000<br /> 256 1716 1668 1714 8920 1806 35 869000<br /> 512 1849 1800 1846 15524 1910 38 807000<br /> 1024 2441 2355 2433 22335 2626 76 612000<br /> 2048 3829 3688 3818 26569 4055 79 391000<br /> 4096 5664 5554 5650 14221 5865 60 265000<br /> 8192 9451 9321 9439 17686 9679 73 159000<br /> 16384 16701 16540 16684 19869 16972 84 90000<br /> 32768 30656 30489 30644 44469 30973 111 49000<br /> 65536 58556 58389 58540 60230 58895 93 26000<br /> <br /> # ll2 with XtremeScale SFC9250 (however on a 10GB cable, not 25GB)<br /> [root@solarflare02-ll2 src]# onload --profile=latency taskset -c 12 ./sfnt-pingpong --affinity &quot;1;1&quot; tcp 192.168.25.5<br /> oo:sfnt-pingpong[28262]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [4]<br /> # cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.25.5<br /> # version: 1.5.0<br /> # src: 8dc3b027d85b28bedf9fd731362e4968<br /> # date: Thu 1 Aug 16:23:39 BST 2019<br /> # uname: Linux solarflare02-ll2.internal 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux<br /> # cpu: model name : Intel(R) Xeon(R) Gold 6246 CPU @ 3.30GHz<br /> # lspci: 3b:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 3b:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 3b:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 3b:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br /> # lspci: 5e:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)<br /> # lspci: 5e:00.1 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)<br /> # lspci: 86:00.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)<br /> # lspci: 86:00.1 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)<br /> # enp134s0f0: driver: sfc<br /> # enp134s0f0: version: 4.15.3.1011<br /> # enp134s0f0: bus-info: 0000:86:00.0<br /> # enp134s0f1: driver: sfc<br /> # enp134s0f1: version: 4.15.3.1011<br /> # enp134s0f1: bus-info: 0000:86:00.1<br /> # enp59s0f0: driver: igb<br /> # enp59s0f0: version: 5.4.0-k<br /> # enp59s0f0: bus-info: 0000:3b:00.0<br /> # enp59s0f1: driver: igb<br /> # enp59s0f1: version: 5.4.0-k<br /> # enp59s0f1: bus-info: 0000:3b:00.1<br /> # enp59s0f2: driver: igb<br /> # enp59s0f2: version: 5.4.0-k<br /> # enp59s0f2: bus-info: 0000:3b:00.2<br /> # enp59s0f3: driver: igb<br /> # enp59s0f3: version: 5.4.0-k<br /> # enp59s0f3: bus-info: 0000:3b:00.3<br /> # enp94s0f0: driver: sfc<br /> # enp94s0f0: version: 4.15.3.1011<br /> # enp94s0f0: bus-info: 0000:5e:00.0<br /> # enp94s0f1: driver: sfc<br /> # enp94s0f1: version: 4.15.3.1011<br /> # enp94s0f1: bus-info: 0000:5e:00.1<br /> # ram: MemTotal: 196463288 kB<br /> # tsc_hz: 3366141000<br /> # LD_PRELOAD=libonload.so<br /> # onload_version=201811-u1<br /> # EF_TCP_FASTSTART_INIT=0<br /> # EF_POLL_USEC=100000<br /> # EF_TCP_FASTSTART_IDLE=0<br /> # server LD_PRELOAD=libonload.so<br /> # percentile=99<br /> #<br /> # size mean min median max %ile stddev iter<br /> 1 1384 1314 1376 30745 1551 68 1000000<br /> 2 1386 1319 1377 16738 1550 54 1000000<br /> 4 1389 1325 1381 14021 1554 51 1000000<br /> 8 1395 1334 1388 5473 1560 50 1000000<br /> 16 1403 1340 1395 25656 1571 61 1000000<br /> 32 1421 1361 1412 8571 1594 53 1000000<br /> 64 1473 1412 1465 8125 1648 48 1000000<br /> 128 1581 1516 1572 17270 1771 55 943000<br /> 256 1736 1626 1728 15769 1943 63 860000<br /> 512 2111 1974 2103 5844 2332 65 707000<br /> 1024 2762 2636 2748 16447 3018 84 541000<br /> 2048 4118 3941 4098 17379 4424 95 364000<br /> 4096 6006 5837 5990 19417 6298 92 250000<br /> 8192 9835 9589 9817 20922 10207 125 153000<br /> 16384 16972 16732 16952 31324 17381 136 89000<br /> 32768 31007 30723 30990 41647 31396 139 49000<br /> 65536 58915 58605 58891 71188 59351 160 26000<br /> <br /> # again with a 25GB cable (slower!) <br /> <br /> <br /> # Results of two HFT nodes VM (same flavor as above) <br /> [root@sfn10 src]# onload --profile=latency taskset -c 0 ./sfnt-pingpong --affinity &quot;1;1&quot; tcp 192.168.10.2<br /> oo:sfnt-pingpong[20567]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [5]<br /> # cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.2<br /> # version: 1.5.0<br /> # src: 8dc3b027d85b28bedf9fd731362e4968<br /> # date: Tue 20 Aug 23:00:19 UTC 2019<br /> # uname: Linux sfn10 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux<br /> # cpu: model name : Intel Xeon Processor (Skylake, IBRS)<br /> # lspci: 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device<br /> # lspci: 00:05.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)<br /> # ens5: driver: sfc<br /> # ens5: version: 4.15.3.1011<br /> # ens5: bus-info: 0000:00:05.0<br /> # eth1: driver: virtio_net<br /> # eth1: version: 1.0.0<br /> # eth1: bus-info: 0000:00:03.0<br /> # ram: MemTotal: 3880072 kB<br /> # tsc_hz: 3366133080<br /> # LD_PRELOAD=libonload.so<br /> # onload_version=201811-u1<br /> # EF_TCP_FASTSTART_INIT=0<br /> # EF_POLL_USEC=100000<br /> # EF_TCP_FASTSTART_IDLE=0<br /> # server LD_PRELOAD=libonload.so<br /> # percentile=99<br /> #<br /> # size mean min median max %ile stddev iter<br /> 1 1572 1444 1503 291943 2593 388 947000<br /> 2 1663 1577 1630 292976 2911 415 896000<br /> 4 1661 1578 1631 194620 2859 321 897000<br /> 8 1668 1577 1637 271314 2890 366 893000<br /> 16 1533 1453 1486 64385 2538 207 971000<br /> 32 1526 1468 1499 240736 2461 380 976000<br /> 64 1569 1508 1543 238612 2468 335 949000<br /> 128 1651 1586 1623 52374 2899 229 903000<br /> 256 1794 1717 1771 293475 2779 434 831000<br /> 512 1923 1827 1903 316119 2793 456 776000<br /> 1024 2417 2346 2394 52744 3193 197 618000<br /> 2048 3798 3650 3766 291841 4781 617 394000<br /> 4096 5677 5519 5643 70912 6694 335 264000<br /> 8192 9431 9249 9384 826605 10561 2068 159000<br /> 16384 16706 16502 16665 269437 17901 939 90000<br /> 32768 30649 30414 30603 181373 32231 793 49000<br /> 65536 58755 58383 58683 296877 60411 1645 26000<br /> [david@Davids-MacBook-Pro-4 notes]$ <br /> <br /> &lt;/syntaxhighlight&gt;</div> David