Difference between revisions of "FhGFS Benchmarking"

From Define Wiki
Jump to navigation Jump to search
(Created page with "== fhgfs-net == <syntaxhighlight> [root@blade1 C]# fhgfs-net mgmt_nodes ============= Blade9 [ID: 1] Connections: TCP: 1 (101.101.101.9:8008); meta_nodes ============= Blade9 [ID: 1...")
 
 
(29 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
== fhgfs-net ==
+
== Network Performance ==
 +
=== Iperf ===
 +
Iperf will validate the efficient bandwidth between every machines, the real network connection speed.
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# fhgfs-net
+
root@cloud32:~# iperf -s
 +
------------------------------------------------------------
 +
Server listening on TCP port 5001
 +
TCP window size: 85.3 KByte (default)
 +
------------------------------------------------------------
 +
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
 +
[ ID] Interval      Transfer    Bandwidth
 +
[ 4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
 +
------------------------------------------------------------
 +
Client connecting to 172.28.12.32, TCP port 5001
 +
TCP window size: 21.6 KByte (default)
 +
------------------------------------------------------------
 +
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
 +
[ ID] Interval      Transfer    Bandwidth
 +
[  3]  0.0- 1.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  1.0- 2.0 sec  172 MBytes  1.44 Gbits/sec
 +
[  3]  2.0- 3.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  3.0- 4.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  4.0- 5.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  5.0- 6.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  6.0- 7.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  7.0- 8.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  8.0- 9.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3] 9.0-10.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
=== Netcat ===
 +
Netcat will determine the current bandwidth for the writes.
 +
<syntaxhighlight>
 +
root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
 +
Listening on [0.0.0.0] (family 0, port 2222)
 +
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
 +
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
 +
^C3884852+0 records in
 +
3884851+0 records out
 +
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s
  
−
mgmt_nodes
+
real    0m45.953s
−
=============
+
user   0m3.719s
−
Blade9 [ID: 1]
+
sys    1m2.406s
−
   Connections: TCP: 1 (101.101.101.9:8008);
+
</syntaxhighlight>
  
−
meta_nodes
+
== Multi Stream Throughput ==
−
=============
+
<div style="border: 1px #888888 solid; background-color: #e57f52;padding: 3px;">'''Eight clients, filesize of ~1GB per storage server''' </div>
−
Blade9 [ID: 11201]
+
=== Two Storage Servers ===
−
  Connections: TCP: 1 (101.101.101.9:8005);
+
==== Tuning ====
 +
Striping across two servers
  
−
storage_nodes
+
<syntaxhighlight>
−
=============
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
−
Blade4 [ID: 3855]
+
Mount: '/mnt/fhgfs'; Path: ''
−
  Connections: TCP: 1 (101.101.101.4:8003);
+
Chunksize: 524288
−
Blade6 [ID: 17264]
+
Number of storage targets: 2
−
  Connections: TCP: 1 (101.101.101.6:8003);
 
−
Blade10 [ID: 18046]
 
−
  Connections: TCP: 1 (101.101.101.10:8003);
 
−
Blade5 [ID: 32274]
 
−
  Connections: TCP: 1 (101.101.101.5:8003);
 
−
Blade7 [ID: 41907]
 
−
  Connections: TCP: 1 (101.101.101.7:8003);
 
−
Blade8 [ID: 56479]
 
−
  Connections: TCP: 1 (101.101.101.8:8003);
 
−
Blade3 [ID: 60659]
 
−
  Connections: TCP: 1 (101.101.101.3:8003);
 
  
 +
root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 2
 
</syntaxhighlight>
 
</syntaxhighlight>
−
== fhgfs-check-servers ==
+
 
 +
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# fhgfs-check-servers
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior
−
Management
+
</syntaxhighlight>
−
==========
 
−
Blade9 [ID: 1]: reachable
 
  
−
Metadata
+
==== Result ====
−
==========
+
<syntaxhighlight>
−
Blade9 [ID: 11201]: reachable
+
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 256 MiB
 +
        aggregate filesize = 2 GiB
 +
 
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        133.09    129.90      131.16      1.17    266.18    259.79      262.32      2.34  15.61596  EXCEL
 +
read          340.31    321.88      334.40      6.63    680.62    643.76      668.79    13.25  6.12693  EXCEL
  
−
Storage
+
Max Write: 133.09 MiB/sec (139.56 MB/sec)
−
==========
+
Max Read: 340.31 MiB/sec (356.84 MB/sec)
−
Blade4 [ID: 3855]: reachable
 
−
Blade6 [ID: 17264]: reachable
 
−
Blade10 [ID: 18046]: reachable
 
−
Blade5 [ID: 32274]: reachable
 
−
Blade7 [ID: 41907]: reachable
 
−
Blade8 [ID: 56479]: reachable
 
−
Blade3 [ID: 60659]: reachable
 
 
</syntaxhighlight>
 
</syntaxhighlight>
−
== fhgfs-df ==
+
 
 +
=== Four Storage Servers ===
 +
==== Tuning ====
 +
Striping across four storage servers
 +
 
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# fhgfs-df
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
−
METADATA SERVERS:
+
Mount: '/mnt/fhgfs'; Path: ''
−
TargetID        Pool      Total        Free      ITotal      IFree
+
Chunksize: 524288
−
========        ====      =====        ====      ======      =====
+
Number of storage targets: 4
−
  11201    [normal]      49.1GB      46.6GB        3.1M        3.0M
 
  
−
STORAGE TARGETS:
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
−
TargetID        Pool      Total        Free      ITotal      IFree
+
Path:
−
========        ====      =====        ====      ======      =====
+
Mount: /mnt/fhgfs
−
    9039      [low]      49.1GB      45.1GB        3.1M        3.1M
+
EntryID: root
−
  23710      [low]     49.1GB      44.5GB        3.1M        3.0M
+
Metadata node: cloud32 [ID: 13708]
−
  26221      [low]      49.1GB      44.5GB        3.1M        3.1M
+
Stripe pattern details:
−
  34234      [low]      49.1GB      45.7GB        3.1M        3.1M
+
+ Type: RAID0
−
  36548      [low]      49.1GB      45.4GB        3.1M        3.0M
+
+ Chunksize: 512K
−
  57799      [low]      49.1GB      45.1GB        3.1M        3.1M
+
+ Number of storage targets: desired: 4
−
  63163      [low]      49.1GB      45.1GB        3.1M        3.1M
 
 
</syntaxhighlight>
 
</syntaxhighlight>
−
== IOR ==
+
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@blade1 C]# ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfile
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 +
==== Result ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
Run began: Tue Aug  6 07:35:35 2013
 
−
Command line used: ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfile
 
−
Machine: Linux blade1 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64
 
−
Using synchronized MPI timer
 
−
Start time skew across all tasks: 0.00 sec
 
−
Path: /mnt/fhgfs
 
−
FS: 343.6 GiB  Used FS: 6.6%  Inodes: 0.0 Mi  Used Inodes: -nan%
 
−
Participating tasks: 1
 
−
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
 
−
task 0 on blade1
 
−
 
 
Summary:
 
Summary:
 
         api                = POSIX
 
         api                = POSIX
−
         test filename      = /mnt/fhgfs/iorfile
+
         test filename      = /mnt/fhgfs/test.ior
 
         access            = file-per-process
 
         access            = file-per-process
−
        pattern            = strided (574 segments)
 
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
 
         ordering inter file=constant task offsets = 1
 
         ordering inter file=constant task offsets = 1
−
         clients            = 1 (1 per node)
+
         clients            = 8 (1 per node)
−
         repetitions        = 4
+
         repetitions        = 5
−
         xfersize          = 262144 bytes
+
         xfersize          = 524288 bytes
−
         blocksize          = 10 MiB
+
         blocksize          = 512 MiB
−
         aggregate filesize = 5.61 GiB
+
         aggregate filesize = 4 GiB
  
−
Using Time Stamp 1375788940 (0x5200df8c) for Data Signature
+
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
−
Commencing write performance test.
+
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
Tue Aug  6 07:35:40 2013
+
write         264.27    252.20      258.31      4.50    528.53    504.40      516.63      9.00  15.86156  EXCEL
 +
read          600.25    534.51      564.42    22.51    1200.49    1069.03    1128.85    45.03  7.26843  EXCEL
  
−
access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)  close(s) total(s)  iter
+
Max Write: 264.27 MiB/sec (277.10 MB/sec)
−
------    ---------  ---------- --------- --------  --------  --------  --------  ----
+
Max Read: 600.25 MiB/sec (629.40 MB/sec)
−
write    532.48    10240      256.00    0.000449  10.78      0.000363  10.78      0    XXCEL
+
</syntaxhighlight>
−
Verifying contents of the file(s) just written.
 
−
Tue Aug  6 07:35:56 2013
 
  
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
+
=== Six Storage Servers ===
−
Commencing read performance test.
+
==== Tuning ====
−
Tue Aug  6 07:36:14 2013
+
Striping across six storage servers.
  
−
read      475.34    10240      256.00    0.001303  12.07      0.000385  12.08      0    XXCEL
+
<syntaxhighlight>
−
Using Time Stamp 1375788986 (0x5200dfba) for Data Signature
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
−
Commencing write performance test.
+
Mount: '/mnt/fhgfs'; Path: ''
−
Tue Aug  6 07:36:26 2013
+
Chunksize: 524288
 +
Number of storage targets: 6
  
−
write    270.84    10240      256.00    0.238056  20.96      0.000310  21.19      1    XXCEL
+
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
−
Verifying contents of the file(s) just written.
+
Path:
−
Tue Aug  6 07:36:47 2013
+
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 6
 +
</syntaxhighlight>
 +
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
  
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
+
==== Result ====
−
Commencing read performance test.
+
<syntaxhighlight>
−
Tue Aug  6 07:37:04 2013
+
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 768 MiB
 +
        aggregate filesize = 6 GiB
  
−
read     496.80     10240     256.00     0.001265   11.55     0.000352  11.55      1   XXCEL
+
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
−
Using Time Stamp 1375789036 (0x5200dfec) for Data Signature
+
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
Commencing write performance test.
+
write        360.60    335.26      344.56     8.61    721.20     670.53     689.13     17.23  17.84209   EXCEL
−
Tue Aug 6 07:37:16 2013
+
read          629.99    596.70     607.71    11.88    1259.99   1193.40    1215.43    23.77 10.11383  EXCEL
  
−
write    217.72    10240      256.00    0.270669  26.09      0.000319  26.36      2    XXCEL
+
Max Write: 360.60 MiB/sec (378.11 MB/sec)
−
Verifying contents of the file(s) just written.
+
Max Read:  629.99 MiB/sec (660.60 MB/sec)
−
Tue Aug  6 07:37:42 2013
+
</syntaxhighlight>
 +
=== Eight Storage Servers ===
 +
==== Tuning ====
 +
Striping across eight storage servers.
 +
<syntaxhighlight>
 +
Number of nodes: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 8
 +
</syntaxhighlight>
 +
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
  
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
+
==== Result ====
−
Commencing read performance test.
+
<syntaxhighlight>
−
Tue Aug  6 07:38:00 2013
+
Summary:
−
 
+
        api                = POSIX
−
read      497.94    10240      256.00    0.001282  11.53      0.000351  11.53      2    XXCEL
+
        test filename     = /mnt/fhgfs/test.ior
−
Using Time Stamp 1375789091 (0x5200e023) for Data Signature
+
        access            = file-per-process
−
Commencing write performance test.
+
        ordering in a file = sequential offsets
−
Tue Aug  6 07:38:11 2013
+
        ordering inter file=constant task offsets = 1
−
 
+
        clients            = 8 (1 per node)
−
write    219.26    10240      256.00    0.235869  25.94      0.000313  26.18     3    XXCEL
+
        repetitions        = 5
−
Verifying contents of the file(s) just written.
+
        xfersize          = 524288 bytes
−
Tue Aug  6 07:38:37 2013
+
        blocksize          = 1 GiB
−
 
+
        aggregate filesize = 8 GiB
−
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
 
−
Commencing read performance test.
 
−
Tue Aug  6 07:38:55 2013
 
−
 
 
−
read      468.33    10240      256.00    0.001103  12.25      0.000376  12.26      3    XXCEL
 
−
Operation  Max (MiB) Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)  Op grep #Tasks tPN reps  fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize
 
  
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
write        532.48     217.72     310.07    130.17      3.71      1.52        2.16     0.91 21.12902   1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
+
write        371.61     357.65     366.43      4.82    743.21    715.30      732.85     9.63 22.36045   EXCEL
−
read          497.94     468.33     484.60     13.01      3.47      3.26        3.38      0.09 11.85335   1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
+
read          648.27     564.96     602.26     27.39    1296.53    1129.92    1204.52    54.78 13.62983   EXCEL
−
 
 
−
Max Write: 532.48 MiB/sec (558.35 MB/sec)
 
−
Max Read:  497.94 MiB/sec (522.12 MB/sec)
 
  
−
Run finished: Tue Aug 6 07:39:07 2013
+
Max Write: 371.61 MiB/sec (389.66 MB/sec)
 +
Max Read:  648.27 MiB/sec (679.76 MB/sec)
 
</syntaxhighlight>
 
</syntaxhighlight>

Latest revision as of 09:59, 7 February 2014

Network Performance

Iperf

Iperf will validate the efficient bandwidth between every machines, the real network connection speed. 

root@cloud32:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
------------------------------------------------------------
Client connecting to 172.28.12.32, TCP port 5001
TCP window size: 21.6 KByte (default)
------------------------------------------------------------
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  1.0- 2.0 sec   172 MBytes  1.44 Gbits/sec
[  3]  2.0- 3.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  3.0- 4.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  4.0- 5.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  5.0- 6.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  6.0- 7.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  7.0- 8.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  8.0- 9.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  9.0-10.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec

Netcat

Netcat will determine the current bandwidth for the writes. 

root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
Listening on [0.0.0.0] (family 0, port 2222)
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
^C3884852+0 records in
3884851+0 records out
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s

real    0m45.953s
user    0m3.719s
sys     1m2.406s

Multi Stream Throughput

Eight clients, filesize of ~1GB per storage server

Two Storage Servers

Tuning

Striping across two servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 2

root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 2

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 256 MiB
        aggregate filesize = 2 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         133.09     129.90      131.16      1.17     266.18     259.79      262.32      2.34  15.61596   EXCEL
read          340.31     321.88      334.40      6.63     680.62     643.76      668.79     13.25   6.12693   EXCEL

Max Write: 133.09 MiB/sec (139.56 MB/sec)
Max Read:  340.31 MiB/sec (356.84 MB/sec)

Four Storage Servers

Tuning

Striping across four storage servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 4

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 4

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 512 MiB
        aggregate filesize = 4 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         264.27     252.20      258.31      4.50     528.53     504.40      516.63      9.00  15.86156   EXCEL
read          600.25     534.51      564.42     22.51    1200.49    1069.03     1128.85     45.03   7.26843   EXCEL

Max Write: 264.27 MiB/sec (277.10 MB/sec)
Max Read:  600.25 MiB/sec (629.40 MB/sec)

Six Storage Servers

Tuning

Striping across six storage servers. 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 6

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 6

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 768 MiB
        aggregate filesize = 6 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         360.60     335.26      344.56      8.61     721.20     670.53      689.13     17.23  17.84209   EXCEL
read          629.99     596.70      607.71     11.88    1259.99    1193.40     1215.43     23.77  10.11383   EXCEL

Max Write: 360.60 MiB/sec (378.11 MB/sec)
Max Read:  629.99 MiB/sec (660.60 MB/sec)

Eight Storage Servers

Tuning

Striping across eight storage servers. 

Number of nodes: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 8

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 1 GiB
        aggregate filesize = 8 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         371.61     357.65      366.43      4.82     743.21     715.30      732.85      9.63  22.36045   EXCEL
read          648.27     564.96      602.26     27.39    1296.53    1129.92     1204.52     54.78  13.62983   EXCEL

Max Write: 371.61 MiB/sec (389.66 MB/sec)
Max Read:  648.27 MiB/sec (679.76 MB/sec)
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=FhGFS_Benchmarking&oldid=3956"