Difference between revisions of "FhGFS Benchmarking"

From Define Wiki
Jump to navigation Jump to search
 
(28 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
== fhgfs-net ==
+
== Network Performance ==
 +
=== Iperf ===
 +
Iperf will validate the efficient bandwidth between every machines, the real network connection speed.
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# fhgfs-net
+
root@cloud32:~# iperf -s
 +
------------------------------------------------------------
 +
Server listening on TCP port 5001
 +
TCP window size: 85.3 KByte (default)
 +
------------------------------------------------------------
 +
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
 +
[ ID] Interval      Transfer    Bandwidth
 +
[ 4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
 +
------------------------------------------------------------
 +
Client connecting to 172.28.12.32, TCP port 5001
 +
TCP window size: 21.6 KByte (default)
 +
------------------------------------------------------------
 +
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
 +
[ ID] Interval      Transfer    Bandwidth
 +
[  3]  0.0- 1.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  1.0- 2.0 sec  172 MBytes  1.44 Gbits/sec
 +
[  3]  2.0- 3.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  3.0- 4.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  4.0- 5.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  5.0- 6.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  6.0- 7.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  7.0- 8.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  8.0- 9.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3] 9.0-10.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
=== Netcat ===
 +
Netcat will determine the current bandwidth for the writes.
 +
<syntaxhighlight>
 +
root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
 +
Listening on [0.0.0.0] (family 0, port 2222)
 +
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
 +
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
 +
^C3884852+0 records in
 +
3884851+0 records out
 +
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s
  
−
mgmt_nodes
+
real   0m45.953s
−
=============
+
user   0m3.719s
−
Blade9 [ID: 1]
+
sys    1m2.406s
−
   Connections: TCP: 1 (101.101.101.9:8008);
+
</syntaxhighlight>
−
 
 
−
meta_nodes
 
−
=============
 
−
Blade9 [ID: 11201]
 
−
   Connections: TCP: 1 (101.101.101.9:8005);
 
  
−
storage_nodes
+
== Multi Stream Throughput ==
−
=============
+
<div style="border: 1px #888888 solid; background-color: #e57f52;padding: 3px;">'''Eight clients, filesize of ~1GB per storage server''' </div>
−
Blade4 [ID: 3855]
+
=== Two Storage Servers ===
−
  Connections: TCP: 1 (101.101.101.4:8003);
+
==== Tuning ====
−
Blade6 [ID: 17264]
+
Striping across two servers
−
  Connections: TCP: 1 (101.101.101.6:8003);
 
−
Blade10 [ID: 18046]
 
−
  Connections: TCP: 1 (101.101.101.10:8003);
 
−
Blade5 [ID: 32274]
 
−
  Connections: TCP: 1 (101.101.101.5:8003);
 
−
Blade7 [ID: 41907]
 
−
  Connections: TCP: 1 (101.101.101.7:8003);
 
−
Blade8 [ID: 56479]
 
−
  Connections: TCP: 1 (101.101.101.8:8003);
 
−
Blade3 [ID: 60659]
 
−
  Connections: TCP: 1 (101.101.101.3:8003);
 
  
−
</syntaxhighlight>
 
−
== fhgfs-check-servers ==
 
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# fhgfs-check-servers
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
−
Management
+
Mount: '/mnt/fhgfs'; Path: ''
−
==========
+
Chunksize: 524288
−
Blade9 [ID: 1]: reachable
+
Number of storage targets: 2
−
 
 
−
Metadata
 
−
==========
 
−
Blade9 [ID: 11201]: reachable
 
  
−
Storage
+
root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
−
==========
+
Path:
−
Blade4 [ID: 3855]: reachable
+
Mount: /mnt/fhgfs
−
Blade6 [ID: 17264]: reachable
+
EntryID: root
−
Blade10 [ID: 18046]: reachable
+
Metadata node: cloud32 [ID: 13708]
−
Blade5 [ID: 32274]: reachable
+
Stripe pattern details:
−
Blade7 [ID: 41907]: reachable
+
+ Type: RAID0
−
Blade8 [ID: 56479]: reachable
+
+ Chunksize: 512K
−
Blade3 [ID: 60659]: reachable
+
+ Number of storage targets: desired: 2
 
</syntaxhighlight>
 
</syntaxhighlight>
−
== fhgfs-df ==
 
−
<syntaxhighlight>
 
−
[root@blade1 C]# fhgfs-df
 
−
METADATA SERVERS:
 
−
TargetID        Pool      Total        Free      ITotal      IFree
 
−
========        ====      =====        ====      ======      =====
 
−
  11201    [normal]      49.1GB      46.6GB        3.1M        3.0M
 
  
−
STORAGE TARGETS:
+
==== Test ====
−
TargetID        Pool      Total        Free      ITotal      IFree
 
−
========        ====      =====        ====      ======      =====
 
−
    9039      [low]      49.1GB      45.1GB        3.1M        3.1M
 
−
  23710      [low]      49.1GB      44.5GB        3.1M        3.0M
 
−
  26221      [low]      49.1GB      44.5GB        3.1M        3.1M
 
−
  34234      [low]      49.1GB      45.7GB        3.1M        3.1M
 
−
  36548      [low]      49.1GB      45.4GB        3.1M        3.0M
 
−
  57799      [low]      49.1GB      45.1GB        3.1M        3.1M
 
−
  63163      [low]      49.1GB      45.1GB        3.1M        3.1M
 
−
</syntaxhighlight>
 
−
== IOR ==
 
−
=== Test 1 ===
 
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# ./IOR
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
==== Result ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
Run began: Tue Aug  6 07:46:08 2013
 
−
Command line used: ./IOR
 
−
Machine: Linux blade1
 
−
 
 
Summary:
 
Summary:
 
         api                = POSIX
 
         api                = POSIX
−
         test filename      = testFile
+
         test filename      = /mnt/fhgfs/test.ior
−
         access            = single-shared-file
+
         access            = file-per-process
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
−
         ordering inter file= no tasks offsets
+
         ordering inter file=constant task offsets = 1
−
         clients            = 1 (1 per node)
+
         clients            = 8 (1 per node)
−
         repetitions        = 1
+
         repetitions        = 5
−
         xfersize          = 262144 bytes
+
         xfersize          = 524288 bytes
−
         blocksize          = 1 MiB
+
         blocksize          = 256 MiB
−
         aggregate filesize = 1 MiB
+
         aggregate filesize = 2 GiB
  
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
write        531.93     531.93     531.93     0.00    2127.74    2127.74    2127.74     0.00  0.00188   EXCEL
+
write        133.09     129.90     131.16     1.17    266.18    259.79      262.32     2.34  15.61596   EXCEL
−
read         1326.47    1326.47     1326.47     0.00    5305.89    5305.89     5305.89     0.00   0.00075   EXCEL
+
read         340.31     321.88     334.40      6.63    680.62     643.76     668.79    13.25   6.12693   EXCEL
 +
 
 +
Max Write: 133.09 MiB/sec (139.56 MB/sec)
 +
Max Read:  340.31 MiB/sec (356.84 MB/sec)
 +
</syntaxhighlight>
  
−
Max Write: 531.93 MiB/sec (557.77 MB/sec)
+
=== Four Storage Servers ===
−
Max Read:  1326.47 MiB/sec (1390.91 MB/sec)
+
==== Tuning ====
 +
Striping across four storage servers
  
−
Run finished: Tue Aug  6 07:46:08 2013
+
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 4
 +
 
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 4
 
</syntaxhighlight>
 
</syntaxhighlight>
−
=== Test 2 ===
+
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfile
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 +
==== Result ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
Run began: Tue Aug  6 07:35:35 2013
 
−
Command line used: ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfile
 
−
Machine: Linux blade1 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64
 
−
Using synchronized MPI timer
 
−
Start time skew across all tasks: 0.00 sec
 
−
Path: /mnt/fhgfs
 
−
FS: 343.6 GiB  Used FS: 6.6%  Inodes: 0.0 Mi  Used Inodes: -nan%
 
−
Participating tasks: 1
 
−
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
 
−
task 0 on blade1
 
−
 
 
Summary:
 
Summary:
 
         api                = POSIX
 
         api                = POSIX
−
         test filename      = /mnt/fhgfs/iorfile
+
         test filename      = /mnt/fhgfs/test.ior
 
         access            = file-per-process
 
         access            = file-per-process
−
        pattern            = strided (574 segments)
 
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
 
         ordering inter file=constant task offsets = 1
 
         ordering inter file=constant task offsets = 1
−
         clients            = 1 (1 per node)
+
         clients            = 8 (1 per node)
−
         repetitions        = 4
+
         repetitions        = 5
−
         xfersize          = 262144 bytes
+
         xfersize          = 524288 bytes
−
         blocksize          = 10 MiB
+
         blocksize          = 512 MiB
−
         aggregate filesize = 5.61 GiB
+
         aggregate filesize = 4 GiB
  
−
Using Time Stamp 1375788940 (0x5200df8c) for Data Signature
+
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
−
Commencing write performance test.
+
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
Tue Aug  6 07:35:40 2013
+
write         264.27    252.20      258.31      4.50    528.53    504.40      516.63      9.00  15.86156  EXCEL
 +
read          600.25    534.51      564.42    22.51    1200.49    1069.03    1128.85    45.03  7.26843  EXCEL
  
−
access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)  close(s) total(s)  iter
+
Max Write: 264.27 MiB/sec (277.10 MB/sec)
−
------    ---------  ---------- --------- --------  --------  --------  --------  ----
+
Max Read: 600.25 MiB/sec (629.40 MB/sec)
−
write    532.48    10240      256.00    0.000449  10.78      0.000363  10.78      0    XXCEL
+
</syntaxhighlight>
−
Verifying contents of the file(s) just written.
 
−
Tue Aug  6 07:35:56 2013
 
  
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
+
=== Six Storage Servers ===
−
Commencing read performance test.
+
==== Tuning ====
−
Tue Aug  6 07:36:14 2013
+
Striping across six storage servers.
  
−
read      475.34    10240      256.00    0.001303  12.07      0.000385  12.08      0    XXCEL
+
<syntaxhighlight>
−
Using Time Stamp 1375788986 (0x5200dfba) for Data Signature
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
−
Commencing write performance test.
+
Mount: '/mnt/fhgfs'; Path: ''
−
Tue Aug  6 07:36:26 2013
+
Chunksize: 524288
 +
Number of storage targets: 6
  
−
write    270.84    10240      256.00    0.238056  20.96      0.000310  21.19      1    XXCEL
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
−
Verifying contents of the file(s) just written.
+
Path:
−
Tue Aug  6 07:36:47 2013
+
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 6
 +
</syntaxhighlight>
 +
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
  
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
+
==== Result ====
−
Commencing read performance test.
+
<syntaxhighlight>
−
Tue Aug  6 07:37:04 2013
+
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 768 MiB
 +
        aggregate filesize = 6 GiB
  
−
read     496.80     10240     256.00     0.001265   11.55     0.000352  11.55      1   XXCEL
+
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
−
Using Time Stamp 1375789036 (0x5200dfec) for Data Signature
+
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
Commencing write performance test.
+
write        360.60    335.26      344.56     8.61    721.20     670.53     689.13     17.23  17.84209   EXCEL
−
Tue Aug 6 07:37:16 2013
+
read          629.99    596.70     607.71    11.88    1259.99   1193.40    1215.43    23.77 10.11383  EXCEL
  
−
write    217.72    10240      256.00    0.270669  26.09      0.000319  26.36      2    XXCEL
+
Max Write: 360.60 MiB/sec (378.11 MB/sec)
−
Verifying contents of the file(s) just written.
+
Max Read:  629.99 MiB/sec (660.60 MB/sec)
−
Tue Aug  6 07:37:42 2013
+
</syntaxhighlight>
 +
=== Eight Storage Servers ===
 +
==== Tuning ====
 +
Striping across eight storage servers.
 +
<syntaxhighlight>
 +
Number of nodes: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 8
 +
</syntaxhighlight>
 +
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
  
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
+
==== Result ====
−
Commencing read performance test.
+
<syntaxhighlight>
−
Tue Aug  6 07:38:00 2013
+
Summary:
−
 
+
        api                = POSIX
−
read      497.94    10240      256.00    0.001282  11.53      0.000351  11.53      2    XXCEL
+
        test filename     = /mnt/fhgfs/test.ior
−
Using Time Stamp 1375789091 (0x5200e023) for Data Signature
+
        access            = file-per-process
−
Commencing write performance test.
+
        ordering in a file = sequential offsets
−
Tue Aug  6 07:38:11 2013
+
        ordering inter file=constant task offsets = 1
−
 
+
        clients            = 8 (1 per node)
−
write    219.26    10240      256.00    0.235869  25.94      0.000313  26.18     3    XXCEL
+
        repetitions        = 5
−
Verifying contents of the file(s) just written.
+
        xfersize          = 524288 bytes
−
Tue Aug  6 07:38:37 2013
+
        blocksize          = 1 GiB
−
 
+
        aggregate filesize = 8 GiB
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
 
−
Commencing read performance test.
 
−
Tue Aug  6 07:38:55 2013
 
−
 
 
−
read      468.33    10240      256.00    0.001103  12.25      0.000376  12.26      3    XXCEL
 
−
Operation  Max (MiB) Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)  Op grep #Tasks tPN reps  fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize
 
  
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
write        532.48     217.72     310.07    130.17      3.71      1.52        2.16     0.91 21.12902   1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
+
write        371.61     357.65     366.43      4.82    743.21    715.30      732.85     9.63 22.36045   EXCEL
−
read          497.94     468.33     484.60     13.01      3.47      3.26        3.38      0.09 11.85335   1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
+
read          648.27     564.96     602.26     27.39    1296.53    1129.92    1204.52    54.78 13.62983   EXCEL
−
 
 
−
Max Write: 532.48 MiB/sec (558.35 MB/sec)
 
−
Max Read:  497.94 MiB/sec (522.12 MB/sec)
 
  
−
Run finished: Tue Aug 6 07:39:07 2013
+
Max Write: 371.61 MiB/sec (389.66 MB/sec)
 +
Max Read:  648.27 MiB/sec (679.76 MB/sec)
 
</syntaxhighlight>
 
</syntaxhighlight>

Latest revision as of 09:59, 7 February 2014

Network Performance

Iperf

Iperf will validate the efficient bandwidth between every machines, the real network connection speed. 

root@cloud32:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
------------------------------------------------------------
Client connecting to 172.28.12.32, TCP port 5001
TCP window size: 21.6 KByte (default)
------------------------------------------------------------
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  1.0- 2.0 sec   172 MBytes  1.44 Gbits/sec
[  3]  2.0- 3.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  3.0- 4.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  4.0- 5.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  5.0- 6.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  6.0- 7.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  7.0- 8.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  8.0- 9.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  9.0-10.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec

Netcat

Netcat will determine the current bandwidth for the writes. 

root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
Listening on [0.0.0.0] (family 0, port 2222)
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
^C3884852+0 records in
3884851+0 records out
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s

real    0m45.953s
user    0m3.719s
sys     1m2.406s

Multi Stream Throughput

Eight clients, filesize of ~1GB per storage server

Two Storage Servers

Tuning

Striping across two servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 2

root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 2

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 256 MiB
        aggregate filesize = 2 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         133.09     129.90      131.16      1.17     266.18     259.79      262.32      2.34  15.61596   EXCEL
read          340.31     321.88      334.40      6.63     680.62     643.76      668.79     13.25   6.12693   EXCEL

Max Write: 133.09 MiB/sec (139.56 MB/sec)
Max Read:  340.31 MiB/sec (356.84 MB/sec)

Four Storage Servers

Tuning

Striping across four storage servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 4

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 4

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 512 MiB
        aggregate filesize = 4 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         264.27     252.20      258.31      4.50     528.53     504.40      516.63      9.00  15.86156   EXCEL
read          600.25     534.51      564.42     22.51    1200.49    1069.03     1128.85     45.03   7.26843   EXCEL

Max Write: 264.27 MiB/sec (277.10 MB/sec)
Max Read:  600.25 MiB/sec (629.40 MB/sec)

Six Storage Servers

Tuning

Striping across six storage servers. 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 6

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 6

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 768 MiB
        aggregate filesize = 6 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         360.60     335.26      344.56      8.61     721.20     670.53      689.13     17.23  17.84209   EXCEL
read          629.99     596.70      607.71     11.88    1259.99    1193.40     1215.43     23.77  10.11383   EXCEL

Max Write: 360.60 MiB/sec (378.11 MB/sec)
Max Read:  629.99 MiB/sec (660.60 MB/sec)

Eight Storage Servers

Tuning

Striping across eight storage servers. 

Number of nodes: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 8

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 1 GiB
        aggregate filesize = 8 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         371.61     357.65      366.43      4.82     743.21     715.30      732.85      9.63  22.36045   EXCEL
read          648.27     564.96      602.26     27.39    1296.53    1129.92     1204.52     54.78  13.62983   EXCEL

Max Write: 371.61 MiB/sec (389.66 MB/sec)
Max Read:  648.27 MiB/sec (679.76 MB/sec)
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=FhGFS_Benchmarking&oldid=3956"