Difference between revisions of "FhGFS Benchmarking"

From Define Wiki
Jump to navigation Jump to search
 
(24 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
= Test System 1 =
+
== Network Performance ==
−
== fhgfs-net ==
+
=== Iperf ===
 +
Iperf will validate the efficient bandwidth between every machines, the real network connection speed.
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# fhgfs-net
+
root@cloud32:~# iperf -s
 +
------------------------------------------------------------
 +
Server listening on TCP port 5001
 +
TCP window size: 85.3 KByte (default)
 +
------------------------------------------------------------
 +
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
 +
[ ID] Interval      Transfer    Bandwidth
 +
[ 4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
 +
------------------------------------------------------------
 +
Client connecting to 172.28.12.32, TCP port 5001
 +
TCP window size: 21.6 KByte (default)
 +
------------------------------------------------------------
 +
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
 +
[ ID] Interval      Transfer    Bandwidth
 +
[  3]  0.0- 1.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  1.0- 2.0 sec  172 MBytes  1.44 Gbits/sec
 +
[  3]  2.0- 3.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  3.0- 4.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  4.0- 5.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  5.0- 6.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  6.0- 7.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  7.0- 8.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  8.0- 9.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3] 9.0-10.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
=== Netcat ===
 +
Netcat will determine the current bandwidth for the writes.
 +
<syntaxhighlight>
 +
root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
 +
Listening on [0.0.0.0] (family 0, port 2222)
 +
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
 +
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
 +
^C3884852+0 records in
 +
3884851+0 records out
 +
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s
  
−
mgmt_nodes
+
real    0m45.953s
−
=============
+
user   0m3.719s
−
Blade9 [ID: 1]
+
sys    1m2.406s
−
   Connections: TCP: 1 (101.101.101.9:8008);
+
</syntaxhighlight>
  
−
meta_nodes
+
== Multi Stream Throughput ==
−
=============
+
<div style="border: 1px #888888 solid; background-color: #e57f52;padding: 3px;">'''Eight clients, filesize of ~1GB per storage server''' </div>
−
Blade9 [ID: 11201]
+
=== Two Storage Servers ===
−
  Connections: TCP: 1 (101.101.101.9:8005);
+
==== Tuning ====
 +
Striping across two servers
  
−
storage_nodes
 
−
=============
 
−
Blade4 [ID: 3855]
 
−
  Connections: TCP: 1 (101.101.101.4:8003);
 
−
Blade6 [ID: 17264]
 
−
  Connections: TCP: 1 (101.101.101.6:8003);
 
−
Blade10 [ID: 18046]
 
−
  Connections: TCP: 1 (101.101.101.10:8003);
 
−
Blade5 [ID: 32274]
 
−
  Connections: TCP: 1 (101.101.101.5:8003);
 
−
Blade7 [ID: 41907]
 
−
  Connections: TCP: 1 (101.101.101.7:8003);
 
−
Blade8 [ID: 56479]
 
−
  Connections: TCP: 1 (101.101.101.8:8003);
 
−
Blade3 [ID: 60659]
 
−
  Connections: TCP: 1 (101.101.101.3:8003);
 
−
 
−
</syntaxhighlight>
 
−
== fhgfs-check-servers ==
 
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# fhgfs-check-servers
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
−
Management
+
Mount: '/mnt/fhgfs'; Path: ''
−
==========
+
Chunksize: 524288
−
Blade9 [ID: 1]: reachable
+
Number of storage targets: 2
  
−
Metadata
+
root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
−
==========
+
Path:
−
Blade9 [ID: 11201]: reachable
+
Mount: /mnt/fhgfs
−
 
+
EntryID: root
−
Storage
+
Metadata node: cloud32 [ID: 13708]
−
==========
+
Stripe pattern details:
−
Blade4 [ID: 3855]: reachable
+
+ Type: RAID0
−
Blade6 [ID: 17264]: reachable
+
+ Chunksize: 512K
−
Blade10 [ID: 18046]: reachable
+
+ Number of storage targets: desired: 2
−
Blade5 [ID: 32274]: reachable
 
−
Blade7 [ID: 41907]: reachable
 
−
Blade8 [ID: 56479]: reachable
 
−
Blade3 [ID: 60659]: reachable
 
 
</syntaxhighlight>
 
</syntaxhighlight>
−
== fhgfs-df ==
 
−
<syntaxhighlight>
 
−
[root@blade1 C]# fhgfs-df
 
−
METADATA SERVERS:
 
−
TargetID        Pool      Total        Free      ITotal      IFree
 
−
========        ====      =====        ====      ======      =====
 
−
  11201    [normal]      49.1GB      46.6GB        3.1M        3.0M
 
  
−
STORAGE TARGETS:
+
==== Test ====
−
TargetID        Pool      Total        Free      ITotal      IFree
 
−
========        ====      =====        ====      ======      =====
 
−
    9039      [low]      49.1GB      45.1GB        3.1M        3.1M
 
−
  23710      [low]      49.1GB      44.5GB        3.1M        3.0M
 
−
  26221      [low]      49.1GB      44.5GB        3.1M        3.1M
 
−
  34234      [low]      49.1GB      45.7GB        3.1M        3.1M
 
−
  36548      [low]      49.1GB      45.4GB        3.1M        3.0M
 
−
  57799      [low]      49.1GB      45.1GB        3.1M        3.1M
 
−
  63163      [low]      49.1GB      45.1GB        3.1M        3.1M
 
−
</syntaxhighlight>
 
−
== IOR ==
 
−
=== Test 1 ===
 
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# ./IOR
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
==== Result ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
Run began: Tue Aug  6 07:46:08 2013
 
−
Command line used: ./IOR
 
−
Machine: Linux blade1
 
−
 
 
Summary:
 
Summary:
 
         api                = POSIX
 
         api                = POSIX
−
         test filename      = testFile
+
         test filename      = /mnt/fhgfs/test.ior
−
         access            = single-shared-file
+
         access            = file-per-process
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
−
         ordering inter file= no tasks offsets
+
         ordering inter file=constant task offsets = 1
−
         clients            = 1 (1 per node)
+
         clients            = 8 (1 per node)
−
         repetitions        = 1
+
         repetitions        = 5
−
         xfersize          = 262144 bytes
+
         xfersize          = 524288 bytes
−
         blocksize          = 1 MiB
+
         blocksize          = 256 MiB
−
         aggregate filesize = 1 MiB
+
         aggregate filesize = 2 GiB
  
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
write        531.93     531.93     531.93     0.00    2127.74    2127.74    2127.74     0.00  0.00188   EXCEL
+
write        133.09     129.90     131.16     1.17    266.18    259.79      262.32     2.34  15.61596   EXCEL
−
read         1326.47    1326.47     1326.47     0.00    5305.89    5305.89     5305.89     0.00   0.00075   EXCEL
+
read         340.31     321.88     334.40      6.63    680.62     643.76     668.79    13.25   6.12693   EXCEL
 +
 
 +
Max Write: 133.09 MiB/sec (139.56 MB/sec)
 +
Max Read:  340.31 MiB/sec (356.84 MB/sec)
 +
</syntaxhighlight>
  
−
Max Write: 531.93 MiB/sec (557.77 MB/sec)
+
=== Four Storage Servers ===
−
Max Read:  1326.47 MiB/sec (1390.91 MB/sec)
+
==== Tuning ====
 +
Striping across four storage servers
  
−
Run finished: Tue Aug  6 07:46:08 2013
+
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 4
 +
 
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 4
 
</syntaxhighlight>
 
</syntaxhighlight>
−
=== Test 2 ===
+
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfile
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 +
==== Result ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
Run began: Tue Aug  6 07:35:35 2013
 
−
Command line used: ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfile
 
−
Machine: Linux blade1 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64
 
−
Using synchronized MPI timer
 
−
Start time skew across all tasks: 0.00 sec
 
−
Path: /mnt/fhgfs
 
−
FS: 343.6 GiB  Used FS: 6.6%  Inodes: 0.0 Mi  Used Inodes: -nan%
 
−
Participating tasks: 1
 
−
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
 
−
task 0 on blade1
 
−
 
 
Summary:
 
Summary:
 
         api                = POSIX
 
         api                = POSIX
−
         test filename      = /mnt/fhgfs/iorfile
+
         test filename      = /mnt/fhgfs/test.ior
 
         access            = file-per-process
 
         access            = file-per-process
−
        pattern            = strided (574 segments)
 
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
 
         ordering inter file=constant task offsets = 1
 
         ordering inter file=constant task offsets = 1
−
         clients            = 1 (1 per node)
+
         clients            = 8 (1 per node)
−
         repetitions        = 4
+
         repetitions        = 5
−
         xfersize          = 262144 bytes
+
         xfersize          = 524288 bytes
−
         blocksize          = 10 MiB
+
         blocksize          = 512 MiB
−
         aggregate filesize = 5.61 GiB
+
         aggregate filesize = 4 GiB
−
 
 
−
Using Time Stamp 1375788940 (0x5200df8c) for Data Signature
 
−
Commencing write performance test.
 
−
Tue Aug  6 07:35:40 2013
 
−
 
 
−
access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)  close(s) total(s)  iter
 
−
------    ---------  ---------- ---------  --------  --------  --------  --------  ----
 
−
write    532.48    10240      256.00    0.000449  10.78      0.000363  10.78      0    XXCEL
 
−
Verifying contents of the file(s) just written.
 
−
Tue Aug  6 07:35:56 2013
 
−
 
 
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
 
−
Commencing read performance test.
 
−
Tue Aug  6 07:36:14 2013
 
−
 
 
−
read      475.34    10240      256.00    0.001303  12.07      0.000385  12.08      0    XXCEL
 
−
Using Time Stamp 1375788986 (0x5200dfba) for Data Signature
 
−
Commencing write performance test.
 
−
Tue Aug  6 07:36:26 2013
 
−
 
 
−
write    270.84    10240      256.00    0.238056  20.96      0.000310  21.19      1    XXCEL
 
−
Verifying contents of the file(s) just written.
 
−
Tue Aug  6 07:36:47 2013
 
−
 
 
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
 
−
Commencing read performance test.
 
−
Tue Aug  6 07:37:04 2013
 
−
 
 
−
read      496.80    10240      256.00    0.001265  11.55      0.000352  11.55      1    XXCEL
 
−
Using Time Stamp 1375789036 (0x5200dfec) for Data Signature
 
−
Commencing write performance test.
 
−
Tue Aug  6 07:37:16 2013
 
−
 
 
−
write    217.72    10240      256.00    0.270669  26.09      0.000319  26.36      2    XXCEL
 
−
Verifying contents of the file(s) just written.
 
−
Tue Aug  6 07:37:42 2013
 
−
 
 
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
 
−
Commencing read performance test.
 
−
Tue Aug  6 07:38:00 2013
 
−
 
 
−
read      497.94    10240      256.00    0.001282  11.53      0.000351  11.53      2    XXCEL
 
−
Using Time Stamp 1375789091 (0x5200e023) for Data Signature
 
−
Commencing write performance test.
 
−
Tue Aug  6 07:38:11 2013
 
−
 
 
−
write    219.26    10240      256.00    0.235869  25.94      0.000313  26.18      3    XXCEL
 
−
Verifying contents of the file(s) just written.
 
−
Tue Aug  6 07:38:37 2013
 
−
 
 
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
 
−
Commencing read performance test.
 
−
Tue Aug  6 07:38:55 2013
 
−
 
 
−
read      468.33    10240      256.00    0.001103  12.25      0.000376  12.26      3    XXCEL
 
−
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)  Op grep #Tasks tPN reps  fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize
 
  
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
write        532.48     217.72     310.07    130.17      3.71      1.52        2.16     0.91 21.12902   1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
+
write        264.27     252.20     258.31      4.50    528.53    504.40      516.63     9.00 15.86156   EXCEL
−
read          497.94     468.33     484.60     13.01      3.47      3.26        3.38      0.09  11.85335   1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
+
read          600.25     534.51     564.42     22.51    1200.49    1069.03    1128.85    45.03  7.26843   EXCEL
−
 
 
−
Max Write: 532.48 MiB/sec (558.35 MB/sec)
 
−
Max Read:  497.94 MiB/sec (522.12 MB/sec)
 
  
−
Run finished: Tue Aug 6 07:39:07 2013
+
Max Write: 264.27 MiB/sec (277.10 MB/sec)
 +
Max Read:  600.25 MiB/sec (629.40 MB/sec)
 
</syntaxhighlight>
 
</syntaxhighlight>
−
== MDTEST ==
 
−
=== Test 1 ===
 
−
<syntaxhighlight>
 
−
-- started at 08/06/2013 07:49:58 --
 
  
−
mdtest-1.9.1 was launched with 1 total task(s) on 1 node(s)
+
=== Six Storage Servers ===
−
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -F -C -T -r -N 1
+
==== Tuning ====
−
Path: /mnt/fhgfs
+
Striping across six storage servers.
−
FS: 343.6 GiB  Used FS: 8.3%  Inodes: 0.0 Mi  Used Inodes: -nan%
 
  
−
1 tasks, 1024 files
+
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 6
  
−
SUMMARY: (of 1 iterations)
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
−
  Operation                      Max            Min          Mean        Std Dev
+
Path:
−
  ---------                      ---            ---          ----        -------
+
Mount: /mnt/fhgfs
−
  File creation    :       2071.708      2071.708      2071.708          0.000
+
EntryID: root
−
  File stat        :       6424.898      6424.898      6424.898          0.000
+
Metadata node: cloud32 [ID: 13708]
−
  File read        :         0.000          0.000          0.000          0.000
+
Stripe pattern details:
−
  File removal      :       1879.944      1879.944      1879.944          0.000
+
+ Type: RAID0
−
  Tree creation    :       1157.688      1157.688      1157.688          0.000
+
+ Chunksize: 512K
−
  Tree removal      :       790.483        790.483        790.483          0.000
+
+ Number of storage targets: desired: 6
−
 
 
−
-- finished at 08/06/2013 07:49:59 --
 
 
</syntaxhighlight>
 
</syntaxhighlight>
−
=== Test 2 ===
+
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
-- started at 08/06/2013 07:51:56 --
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
  
−
mdtest-1.9.1 was launched with 1 total task(s) on 1 node(s)
+
==== Result ====
−
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -S -C -T -r -N 1
+
<syntaxhighlight>
−
Path: /mnt/fhgfs
+
Summary:
−
FS: 343.6 GiB   Used FS: 8.3%  Inodes: 0.0 Mi  Used Inodes: -nan%
+
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 768 MiB
 +
        aggregate filesize = 6 GiB
  
−
1 tasks, 1 file
+
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        360.60    335.26      344.56      8.61    721.20    670.53      689.13    17.23  17.84209  EXCEL
 +
read          629.99    596.70      607.71    11.88    1259.99    1193.40    1215.43    23.77  10.11383  EXCEL
  
−
SUMMARY: (of 1 iterations)
+
Max Write: 360.60 MiB/sec (378.11 MB/sec)
−
  Operation                      Max           Min          Mean        Std Dev
+
Max Read: 629.99 MiB/sec (660.60 MB/sec)
−
  ---------                      ---            ---          ----        -------
 
−
  File creation    :       2737.946      2737.946      2737.946          0.000
 
−
  File stat        :        199.851        199.851        199.851          0.000
 
−
  File read        :          0.000          0.000          0.000          0.000
 
−
  File removal      :      2503.440      2503.440      2503.440          0.000
 
−
  Tree creation    :      1277.194      1277.194      1277.194          0.000
 
−
  Tree removal      :      1117.289      1117.289      1117.289          0.000
 
−
 
 
−
-- finished at 08/06/2013 07:52:02 --
 
 
</syntaxhighlight>
 
</syntaxhighlight>
−
=== Test 3 ===
+
=== Eight Storage Servers ===
 +
==== Tuning ====
 +
Striping across eight storage servers.
 
<syntaxhighlight>
 
<syntaxhighlight>
−
-- started at 08/06/2013 07:53:42 --
+
Number of nodes: 8
−
 
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
−
mdtest-1.9.1 was launched with 2 total task(s) on 1 node(s)
+
Mount: '/mnt/fhgfs'; Path: ''
−
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -S -C -T -r -n 1
+
Chunksize: 524288
−
Path: /mnt/fhgfs
+
Number of storage targets: 8
−
FS: 343.6 GiB  Used FS: 8.3%  Inodes: 0.0 Mi  Used Inodes: -nan%
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
−
 
+
Path:
−
2 tasks, 1 file
+
Mount: /mnt/fhgfs
−
 
+
EntryID: root
−
SUMMARY: (of 1 iterations)
+
Metadata node: cloud32 [ID: 13708]
−
  Operation                      Max            Min          Mean        Std Dev
+
Stripe pattern details:
−
  ---------                      ---            ---          ----        -------
+
+ Type: RAID0
−
  File creation    :         0.400          0.400          0.400          0.000
+
+ Chunksize: 512K
−
  File stat        :       1558.931      1558.931      1558.931          0.000
+
+ Number of storage targets: desired: 8
−
  File read        :         0.000          0.000          0.000          0.000
 
−
  File removal      :       3718.355      3718.355      3718.355          0.000
 
−
  Tree creation    :       1237.623      1237.623      1237.623          0.000
 
−
  Tree removal      :       896.027        896.027        896.027          0.000
 
−
 
 
−
-- finished at 08/06/2013 07:53:47 --
 
−
 
 
 
</syntaxhighlight>
 
</syntaxhighlight>
−
=== Test 4 ===
+
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
-- started at 08/06/2013 07:55:05 --
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior
−
 
 
−
mdtest-1.9.1 was launched with 10 total task(s) on 1 node(s)
 
−
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -S -C -T -r -n 1
 
−
Path: /mnt/fhgfs
 
−
FS: 343.6 GiB  Used FS: 8.3%  Inodes: 0.0 Mi  Used Inodes: -nan%
 
−
 
 
−
10 tasks, 1 file
 
−
 
 
−
SUMMARY: (of 1 iterations)
 
−
  Operation                      Max            Min          Mean        Std Dev
 
−
  ---------                      ---            ---          ----        -------
 
−
  File creation    :      1563.231      1563.231      1563.231          0.000
 
−
  File stat        :      10988.483      10988.483      10988.483          0.000
 
−
  File read        :          0.000          0.000          0.000          0.000
 
−
  File removal      :      23750.306      23750.306      23750.306          0.000
 
−
  Tree creation    :      1524.647      1524.647      1524.647          0.000
 
−
  Tree removal      :      1058.366      1058.366      1058.366          0.000
 
−
 
 
−
-- finished at 08/06/2013 07:55:05 --
 
−
 
 
 
</syntaxhighlight>
 
</syntaxhighlight>
−
=== Test 5 ===
 
−
<syntaxhighlight>
 
−
-- started at 08/06/2013 07:57:13 --
 
  
−
mdtest-1.9.1 was launched with 1 total task(s) on 1 node(s)
+
==== Result ====
−
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -F -E -C -T -r -N 1
 
−
Path: /mnt/fhgfs
 
−
FS: 343.6 GiB  Used FS: 8.3%  Inodes: 0.0 Mi  Used Inodes: -nan%
 
−
 
 
−
1 tasks, 1024 files
 
−
 
 
−
SUMMARY: (of 1 iterations)
 
−
  Operation                      Max            Min          Mean        Std Dev
 
−
  ---------                      ---            ---          ----        -------
 
−
  File creation    :      2519.003      2519.003      2519.003          0.000
 
−
  File stat        :      8322.967      8322.967      8322.967          0.000
 
−
  File read        :      3699.997      3699.997      3699.997          0.000
 
−
  File removal      :      2406.156      2406.156      2406.156          0.000
 
−
  Tree creation    :      1179.169      1179.169      1179.169          0.000
 
−
  Tree removal      :        972.705        972.705        972.705          0.000
 
−
 
 
−
-- finished at 08/06/2013 07:57:14 --
 
−
</syntaxhighlight>
 
−
=== Test 6 ===
 
 
<syntaxhighlight>
 
<syntaxhighlight>
−
-- started at 08/06/2013 07:57:43 --
+
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 1 GiB
 +
        aggregate filesize = 8 GiB
  
−
mdtest-1.9.1 was launched with 10 total task(s) on 1 node(s)
+
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
−
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -F -E -C -T -r -N 1
+
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   ------- --------
−
Path: /mnt/fhgfs
+
write        371.61     357.65      366.43      4.82    743.21    715.30      732.85      9.63  22.36045  EXCEL
−
FS: 343.6 GiB   Used FS: 8.3%  Inodes: 0.0 Mi  Used Inodes: -nan%
+
read         648.27    564.96     602.26    27.39   1296.53   1129.92    1204.52    54.78  13.62983  EXCEL
−
 
 
−
10 tasks, 10240 files
 
−
 
 
−
SUMMARY: (of 1 iterations)
 
−
  Operation                      Max            Min          Mean        Std Dev
 
−
  ---------                     ---           ---           ----       -------
 
−
  File creation     :      4349.006      4349.006      4349.006          0.000
 
−
  File stat        :      2006.295      2006.295      2006.295          0.000
 
−
  File read         :      1934.656      1934.656      1934.656          0.000
 
−
  File removal     :      2126.036      2126.036      2126.036          0.000
 
−
   Tree creation    :      1189.199      1189.199      1189.199          0.000
 
−
   Tree removal      :        177.274        177.274        177.274          0.000
 
  
−
-- finished at 08/06/2013 07:58:01 --
+
Max Write: 371.61 MiB/sec (389.66 MB/sec)
 +
Max Read: 648.27 MiB/sec (679.76 MB/sec)
 
</syntaxhighlight>
 
</syntaxhighlight>

Latest revision as of 09:59, 7 February 2014

Network Performance

Iperf

Iperf will validate the efficient bandwidth between every machines, the real network connection speed. 

root@cloud32:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
------------------------------------------------------------
Client connecting to 172.28.12.32, TCP port 5001
TCP window size: 21.6 KByte (default)
------------------------------------------------------------
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  1.0- 2.0 sec   172 MBytes  1.44 Gbits/sec
[  3]  2.0- 3.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  3.0- 4.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  4.0- 5.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  5.0- 6.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  6.0- 7.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  7.0- 8.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  8.0- 9.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  9.0-10.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec

Netcat

Netcat will determine the current bandwidth for the writes. 

root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
Listening on [0.0.0.0] (family 0, port 2222)
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
^C3884852+0 records in
3884851+0 records out
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s

real    0m45.953s
user    0m3.719s
sys     1m2.406s

Multi Stream Throughput

Eight clients, filesize of ~1GB per storage server

Two Storage Servers

Tuning

Striping across two servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 2

root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 2

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 256 MiB
        aggregate filesize = 2 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         133.09     129.90      131.16      1.17     266.18     259.79      262.32      2.34  15.61596   EXCEL
read          340.31     321.88      334.40      6.63     680.62     643.76      668.79     13.25   6.12693   EXCEL

Max Write: 133.09 MiB/sec (139.56 MB/sec)
Max Read:  340.31 MiB/sec (356.84 MB/sec)

Four Storage Servers

Tuning

Striping across four storage servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 4

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 4

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 512 MiB
        aggregate filesize = 4 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         264.27     252.20      258.31      4.50     528.53     504.40      516.63      9.00  15.86156   EXCEL
read          600.25     534.51      564.42     22.51    1200.49    1069.03     1128.85     45.03   7.26843   EXCEL

Max Write: 264.27 MiB/sec (277.10 MB/sec)
Max Read:  600.25 MiB/sec (629.40 MB/sec)

Six Storage Servers

Tuning

Striping across six storage servers. 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 6

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 6

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 768 MiB
        aggregate filesize = 6 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         360.60     335.26      344.56      8.61     721.20     670.53      689.13     17.23  17.84209   EXCEL
read          629.99     596.70      607.71     11.88    1259.99    1193.40     1215.43     23.77  10.11383   EXCEL

Max Write: 360.60 MiB/sec (378.11 MB/sec)
Max Read:  629.99 MiB/sec (660.60 MB/sec)

Eight Storage Servers

Tuning

Striping across eight storage servers. 

Number of nodes: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 8

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 1 GiB
        aggregate filesize = 8 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         371.61     357.65      366.43      4.82     743.21     715.30      732.85      9.63  22.36045   EXCEL
read          648.27     564.96      602.26     27.39    1296.53    1129.92     1204.52     54.78  13.62983   EXCEL

Max Write: 371.61 MiB/sec (389.66 MB/sec)
Max Read:  648.27 MiB/sec (679.76 MB/sec)
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=FhGFS_Benchmarking&oldid=3956"