Solarflare SFN7122F

Onload Install

System requierments

Centos with full updates
Development tools

Installation

scripts/onload_install
onload_tool reload

Solar Flare Tests

Pre requisites

centos uptodate
Development tools

two servers connected by direct cable
Solarflare interfaces configured and pingbale
OpenOnload, netperf and sfnettest installed

netperf install

netperf can be downloaded from http://www.netperf.org/netperf/

tar -zxvf netperf-<version>.tar.gz
cd netperf-<version>
./configure
make install

sfnettest install

sfnettest can be downloaded from http://www.openonload.org

tar -zxvf sfnettest-<version>.tgz
cd sfnettest-<version>/src
make

pre test Config

Isolate the CPUs from, SMP balancing by adding teh following to the kernel line in boot.conf:

isolcpus=<comma separated cpu list>

Stop the services:

service cpuspeed stop
ice irqbalance stop
service iptables stop
ethtool -C eth<N> rx-usecs 0 adaptive-rx off

Check the BIOS settings so make sure that power saving is disabled

Steer the Interrupts to a CPU core used for the benchmark:

# cat /proc/interrupts | grep eth2
33: 0 0 0 0 IR-PCI-MSI-edge eth2-0
34: 0 0 0 0 IR-PCI-MSI-edge eth2-1

# echo 1 > /proc/irq/33/smp_affinity
# echo 2 > /proc/irq/34/smp_affinity

Latency Tests

Run sfnt-pingpong on both servers:

[root@system-1]# onload --profile=latency taskset -c 3 ./sfnt-pingpong
[root@system-2]# onload --profile=latency taskset -c 3 ./sfnt-pingpong \
--affinity "3;3" udp <system1-ip>

Latency Results

               size        mean       min        median    max       %ile      stddev    iter
                0         2503       2154       2487      18825     2624       69        596000
                1         2502       2223       2487      17217     2612       62        596000
                2         2503       2176       2487      18864     2616       59        596000
                4         2503       2168       2487      13717     2618       58        596000
                8         2505       2187       2490      14830     2623       57        596000
                16        2499       2229       2479      11746     2603       58        597000
                32        2539       2267       2518      9174      2688       62        588000
                64        2569       2085       2546      7050      2712       59        581000
                128       2693       2382       2678      7312      2857       71        554000
                256       2849       2566       2833      7247      3019       70        524000
                512       3004       2714       2986      7648      3180       75        497000
                1024      3698       3318       3704      8298      3872       86        404000
                1472      4191       3862       4173      10687     4386       86        357000
                1473      10016      8907       9509      14944     13021      1059      150000
                2048      10425      9258       9872      15001     13615      1168      144000
                4096      13375      12061      13005     16923     15385      856       112000
                8192      21197      18881      20913     29207     24453      1130      71000
                16384     36503      33472      36381     42301     39757      1177      42000
                32768     65624      61013     65539      71654     69432      1516      23000

Netperf Tests

To accelerate an application with the Onload user-level transport, simply invoke the application on the command line as normal, prepended with "onload".

ie. Instead of:

  netperf -t TCP_RR -H myserver

do this:

  onload netperf -t TCP_RR -H myserver

and tuned for best latency:

  onload -p latency netperf -t TCP_RR -H myserver

sqmr

Solarflare results on the left. Ethernet on the right. Onload fails with MPI.

# msgsize iters time     msgs/sec MiB/sec     msgs/sec MiB/se   # msgsize iters time     msgs/sec MiB/sec     msgs/sec MiB/se
       0  4096  0.11     75280.97    0.00     75280.97    0.0 |        0  4096  0.11     75387.01    0.00     75387.01    0.0
       1  3277  0.08     81653.50    0.08     81653.50    0.0 |        1  3277  0.08     81618.83    0.08     81618.83    0.0
       2  2622  0.06     81574.19    0.16     81574.19    0.1 |        2  2622  0.06     81633.83    0.16     81633.83    0.1
       4  2098  0.05     81634.33    0.31     81634.33    0.3 |        4  2098  0.05     81637.36    0.31     81637.36    0.3
       8  1679  0.04     81589.99    0.62     81589.99    0.6 |        8  1679  0.04     81657.64    0.62     81657.64    0.6
      16  1344  0.03     81632.68    1.25     81632.68    1.2 |       16  1344  0.03     81635.04    1.25     81635.04    1.2
      32  1076  0.03     81635.79    2.49     81635.79    2.4 |       32  1076  0.03     81632.83    2.49     81632.83    2.4
      64   861  0.02     81716.47    4.99     81716.47    4.9 |       64   861  0.02     81591.84    4.98     81591.84    4.9
     128   689  0.02     81528.96    9.95     81528.96    9.9 |      128   689  0.02     81687.97    9.97     81687.97    9.9
     256   552  0.01     81633.76   19.93     81633.76   19.9 |      256   552  0.01     81626.56   19.93     81626.56   19.9
     512   442  0.01     76950.12   37.57     76950.12   37.5        512   442  0.01     76950.12   37.57     76950.12   37.5
    1024   354  0.02     42666.20   41.67     42666.20   41.6 |     1024   354  0.02     42661.29   41.66     42661.29   41.6
    2048   284  0.03     16303.61   31.84     16303.61   31.8 |     2048   284  0.03     16234.61   31.71     16234.61   31.7
    4096   228  0.03     16716.80   65.30     16716.80   65.3 |     4096   228  0.03     16785.61   65.57     16785.61   65.5
    8192   183  0.01     24906.15  194.58     24906.15  194.5 |     8192   183  0.02     16558.07  129.36     16558.07  129.3
   16384   147  0.02     12678.91  198.11     12678.91  198.1 |    16384   147  0.02     12699.28  198.43     12699.28  198.4
   32768   118  0.03      9290.67  290.33      9290.67  290.3 |    32768   118  0.03      9189.67  287.18      9189.67  287.1
   65536    95  0.07      2743.08  171.44      2743.08  171.4 |    65536    95  0.07      2725.74  170.36      2725.74  170.3
  131072    76  0.08      1813.43  226.68      1813.43  226.6 |   131072    76  0.08      1792.37  224.05      1792.37  224.0
  262144    61  0.09      1314.43  328.61      1314.43  328.6 |   262144    61  0.10      1258.62  314.65      1258.62  314.6
  524288    49  0.13       726.65  363.32       726.65  363.3 |   524288    49  0.13       729.33  364.67       729.33  364.6
 1048576    40  0.21       387.84  387.84       387.84  387.8 |  1048576    40  0.41       196.70  196.70       196.70  196.7
 2097152    32  0.55       116.48  232.95       116.48  232.9 |  2097152    32  0.76        84.01  168.02        84.01  168.0
 4194304    26  0.51       102.45  409.79       102.45  409.7 |  4194304    26  0.89        58.12  232.50        58.12  232.5