Difference between revisions of "Solarflare SFN7122F"

From Define Wiki
Jump to navigation Jump to search
Line 11: Line 11:
 
scripts/onload_install
 
scripts/onload_install
 
onload_tool reload
 
onload_tool reload
 +
</syntaxhighlight>
 +
 +
== Solar Flare Tests ==
 +
 +
===Pre requisites ===
 +
 +
* centos uptodate
 +
* Development tools
 +
 +
* two servers connected by direct cable
 +
* Solarflare interfaces configured and pingbale
 +
* OpenOnload, netperf and sfnettest installed
 +
 +
 +
=== netperf install ===
 +
 +
netperf can be downloaded from http://www.netperf.org/netperf/
 +
 +
<syntaxhighlight>
 +
tar -zxvf netperf-<version>.tar.gz
 +
cd netperf-<version>
 +
./configure
 +
make install
 +
</syntaxhighlight>
 +
 +
 +
=== sfnettest install ===
 +
 +
sfnettest can be downloaded from http://www.openonload.org
 +
 +
<syntaxhighlight>
 +
tar -zxvf sfnettest-<version>.tgz
 +
cd sfnettest-<version>/src
 +
make
 +
</syntaxhighlight>
 +
 +
=== pre test Config ===
 +
 +
Isolate the CPUs from, SMP balancing by adding teh following to the kernel line in boot.conf:
 +
 +
<syntaxhighlight>
 +
isolcpus=<comma separated cpu list>
 +
</syntaxhighlight>
 +
 +
Stop the services:
 +
 +
<syntaxhighlight>
 +
service cpuspeed stop
 +
ice irqbalance stop
 +
service iptables stop
 +
ethtool -C eth<N> rx-usecs 0 adaptive-rx off
 +
</syntaxhighlight>
 +
 +
Check the BIOS settings so make sure that power saving is disabled
 +
 +
Steer the Interrupts to a CPU core used for the benchmark:
 +
 +
<syntaxhighlight>
 +
# cat /proc/interrupts | grep eth2
 +
33: 0 0 0 0 IR-PCI-MSI-edge eth2-0
 +
34: 0 0 0 0 IR-PCI-MSI-edge eth2-1
 +
 +
# echo 1 > /proc/irq/33/smp_affinity
 +
# echo 2 > /proc/irq/34/smp_affinity
 +
</syntaxhighlight>
 +
 +
 +
== Latency Tests ==
 +
 +
Run sfnt-pingpong on both servers:
 +
 +
<syntaxhighlight>
 +
[root@system-1]# onload --profile=latency taskset -c 3 ./sfnt-pingpong
 +
[root@system-2]# onload --profile=latency taskset -c 3 ./sfnt-pingpong \
 +
--affinity "3;3" udp <system1-ip>
 +
</syntaxhighlight>
 +
 +
 +
=== Latency Results ===
 +
 +
<syntaxhighlight>
 +
#            size        mean  min        median                max      %ile        stddev                  iter
 +
                0              2503      2154      2487                      18825    2624      69                          596000
 +
                1              2502      2223      2487                      17217  2612      62                          596000
 +
                2              2503      2176      2487                      18864  2616      59                          596000
 +
                4              2503      2168      2487                      13717  2618      58                          596000
 +
                8              2505      2187      2490                      14830  2623      57                          596000
 +
                16          2499      2229      2479                      11746  2603      58                          597000
 +
                32          2539      2267      2518                      9174      2688      62                          588000
 +
                64          2569      2085      2546                      7050      2712      59                          581000
 +
                128        2693      2382      2678                      7312      2857      71                          554000
 +
                256        2849      2566      2833                      7247      3019      70                          524000
 +
                512        3004      2714      2986                      7648      3180      75                          497000
 +
                1024      3698      3318      3704                      8298      3872      86                          404000
 +
                1472      4191      3862      4173                      10687  4386      86                          357000
 +
                1473      10016  8907      9509                      14944  13021  1059                      150000
 +
                2048      10425  9258      9872                      15001  13615  1168                      144000
 +
                4096      13375  12061  13005                    16923  15385  856                        112000
 +
                8192      21197  18881  20913                    29207  24453  1130                      71000
 +
                16384  36503  33472  36381                    42301  39757  1177                      42000
 +
                32768  65624  61013  65539                    71654  69432  1516                      23000
 +
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 28: Line 130:
  
 
   onload -p latency netperf -t TCP_RR -H myserver
 
   onload -p latency netperf -t TCP_RR -H myserver
−
 
  
 
== Netperf Results ==
 
== Netperf Results ==

Revision as of 13:30, 23 August 2013

Onload Install

System requierments

  • Centos with full updates
  • Development tools

Installation

scripts/onload_install
onload_tool reload

Solar Flare Tests

Pre requisites

  • centos uptodate
  • Development tools
  • two servers connected by direct cable
  • Solarflare interfaces configured and pingbale
  • OpenOnload, netperf and sfnettest installed


netperf install

netperf can be downloaded from http://www.netperf.org/netperf/ 

tar -zxvf netperf-<version>.tar.gz
cd netperf-<version>
./configure
make install


sfnettest install

sfnettest can be downloaded from http://www.openonload.org 

tar -zxvf sfnettest-<version>.tgz
cd sfnettest-<version>/src
make

pre test Config

Isolate the CPUs from, SMP balancing by adding teh following to the kernel line in boot.conf: 

isolcpus=<comma separated cpu list>

Stop the services: 

service cpuspeed stop
ice irqbalance stop
service iptables stop
ethtool -C eth<N> rx-usecs 0 adaptive-rx off

Check the BIOS settings so make sure that power saving is disabled

Steer the Interrupts to a CPU core used for the benchmark: 

# cat /proc/interrupts | grep eth2
33: 0 0 0 0 IR-PCI-MSI-edge eth2-0
34: 0 0 0 0 IR-PCI-MSI-edge eth2-1

# echo 1 > /proc/irq/33/smp_affinity
# echo 2 > /proc/irq/34/smp_affinity


Latency Tests

Run sfnt-pingpong on both servers: 

[root@system-1]# onload --profile=latency taskset -c 3 ./sfnt-pingpong
[root@system-2]# onload --profile=latency taskset -c 3 ./sfnt-pingpong \
--affinity "3;3" udp <system1-ip>


Latency Results

#             size        mean   min        median                 max       %ile        stddev                  iter
                0              2503       2154       2487                       18825     2624      69                           596000
                1              2502       2223       2487                       17217   2612       62                           596000
                2              2503       2176       2487                       18864   2616       59                           596000
                4              2503       2168       2487                       13717   2618       58                           596000
                8              2505       2187       2490                       14830   2623       57                           596000
                16           2499       2229       2479                       11746   2603       58                           597000
                32           2539       2267       2518                       9174       2688       62                           588000
                64           2569       2085       2546                       7050       2712       59                           581000
                128         2693       2382       2678                       7312       2857       71                           554000
                256         2849       2566       2833                       7247       3019       70                           524000
                512         3004       2714       2986                       7648       3180       75                           497000
                1024       3698       3318       3704                       8298       3872       86                           404000
                1472       4191       3862       4173                       10687   4386       86                           357000
                1473       10016   8907       9509                       14944   13021   1059                       150000
                2048       10425   9258       9872                       15001   13615   1168                       144000
                4096       13375   12061   13005                    16923   15385   856                         112000
                8192       21197   18881   20913                    29207   24453   1130                       71000
                16384   36503   33472   36381                    42301   39757   1177                       42000
                32768   65624   61013   65539                    71654   69432   1516                       23000

Netperf Tests

To accelerate an application with the Onload user-level transport, simply invoke the application on the command line as normal, prepended with "onload".

ie. Instead of: 

  netperf -t TCP_RR -H myserver

do this: 

  onload netperf -t TCP_RR -H myserver

and tuned for best latency: 

  onload -p latency netperf -t TCP_RR -H myserver

Netperf Results

Netperf 12,333.00
Onload Netperf 68004.42
Onload best latency 97869.02

Iperf Results

Serial Runs

Iperf 1 9.42
Onload iperf 1 9.41
Onload best latency 1 9.41

Parallel Runs

Iperf 10 9.42
Onload iperf 10 9.41
Onload best latency 10 9.41


sqmr

Solarflare results on the left. Ethernet on the right. Onload fails with MPI. 

# msgsize iters time     msgs/sec MiB/sec     msgs/sec MiB/se   # msgsize iters time     msgs/sec MiB/sec     msgs/sec MiB/se
       0  4096  0.11     75280.97    0.00     75280.97    0.0 |        0  4096  0.11     75387.01    0.00     75387.01    0.0
       1  3277  0.08     81653.50    0.08     81653.50    0.0 |        1  3277  0.08     81618.83    0.08     81618.83    0.0
       2  2622  0.06     81574.19    0.16     81574.19    0.1 |        2  2622  0.06     81633.83    0.16     81633.83    0.1
       4  2098  0.05     81634.33    0.31     81634.33    0.3 |        4  2098  0.05     81637.36    0.31     81637.36    0.3
       8  1679  0.04     81589.99    0.62     81589.99    0.6 |        8  1679  0.04     81657.64    0.62     81657.64    0.6
      16  1344  0.03     81632.68    1.25     81632.68    1.2 |       16  1344  0.03     81635.04    1.25     81635.04    1.2
      32  1076  0.03     81635.79    2.49     81635.79    2.4 |       32  1076  0.03     81632.83    2.49     81632.83    2.4
      64   861  0.02     81716.47    4.99     81716.47    4.9 |       64   861  0.02     81591.84    4.98     81591.84    4.9
     128   689  0.02     81528.96    9.95     81528.96    9.9 |      128   689  0.02     81687.97    9.97     81687.97    9.9
     256   552  0.01     81633.76   19.93     81633.76   19.9 |      256   552  0.01     81626.56   19.93     81626.56   19.9
     512   442  0.01     76950.12   37.57     76950.12   37.5        512   442  0.01     76950.12   37.57     76950.12   37.5
    1024   354  0.02     42666.20   41.67     42666.20   41.6 |     1024   354  0.02     42661.29   41.66     42661.29   41.6
    2048   284  0.03     16303.61   31.84     16303.61   31.8 |     2048   284  0.03     16234.61   31.71     16234.61   31.7
    4096   228  0.03     16716.80   65.30     16716.80   65.3 |     4096   228  0.03     16785.61   65.57     16785.61   65.5
    8192   183  0.01     24906.15  194.58     24906.15  194.5 |     8192   183  0.02     16558.07  129.36     16558.07  129.3
   16384   147  0.02     12678.91  198.11     12678.91  198.1 |    16384   147  0.02     12699.28  198.43     12699.28  198.4
   32768   118  0.03      9290.67  290.33      9290.67  290.3 |    32768   118  0.03      9189.67  287.18      9189.67  287.1
   65536    95  0.07      2743.08  171.44      2743.08  171.4 |    65536    95  0.07      2725.74  170.36      2725.74  170.3
  131072    76  0.08      1813.43  226.68      1813.43  226.6 |   131072    76  0.08      1792.37  224.05      1792.37  224.0
  262144    61  0.09      1314.43  328.61      1314.43  328.6 |   262144    61  0.10      1258.62  314.65      1258.62  314.6
  524288    49  0.13       726.65  363.32       726.65  363.3 |   524288    49  0.13       729.33  364.67       729.33  364.6
 1048576    40  0.21       387.84  387.84       387.84  387.8 |  1048576    40  0.41       196.70  196.70       196.70  196.7
 2097152    32  0.55       116.48  232.95       116.48  232.9 |  2097152    32  0.76        84.01  168.02        84.01  168.0
 4194304    26  0.51       102.45  409.79       102.45  409.7 |  4194304    26  0.89        58.12  232.50        58.12  232.5
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Solarflare_SFN7122F&oldid=3269"