Difference between revisions of "FhGFS Benchmarking"

From Define Wiki
Jump to navigation Jump to search
 
(16 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
== Network Performance ==
 +
=== Iperf ===
 +
Iperf will validate the efficient bandwidth between every machines, the real network connection speed.
 +
<syntaxhighlight>
 +
root@cloud32:~# iperf -s
 +
------------------------------------------------------------
 +
Server listening on TCP port 5001
 +
TCP window size: 85.3 KByte (default)
 +
------------------------------------------------------------
 +
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
 +
[ ID] Interval      Transfer    Bandwidth
 +
[  4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
 +
------------------------------------------------------------
 +
Client connecting to 172.28.12.32, TCP port 5001
 +
TCP window size: 21.6 KByte (default)
 +
------------------------------------------------------------
 +
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
 +
[ ID] Interval      Transfer    Bandwidth
 +
[  3]  0.0- 1.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  1.0- 2.0 sec  172 MBytes  1.44 Gbits/sec
 +
[  3]  2.0- 3.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  3.0- 4.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  4.0- 5.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  5.0- 6.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  6.0- 7.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  7.0- 8.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  8.0- 9.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  9.0-10.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
=== Netcat ===
 +
Netcat will determine the current bandwidth for the writes.
 +
<syntaxhighlight>
 +
root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
 +
Listening on [0.0.0.0] (family 0, port 2222)
 +
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
 +
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
 +
^C3884852+0 records in
 +
3884851+0 records out
 +
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s
 +
 +
real    0m45.953s
 +
user    0m3.719s
 +
sys    1m2.406s
 +
</syntaxhighlight>
 +
 
== Multi Stream Throughput ==
 
== Multi Stream Throughput ==
−
'''5 clients, filesize of ~1GB per storage server'''  
+
<div style="border: 1px #888888 solid; background-color: #e57f52;padding: 3px;">'''Eight clients, filesize of ~1GB per storage server''' </div>
 
=== Two Storage Servers ===
 
=== Two Storage Servers ===
 
==== Tuning ====
 
==== Tuning ====
Line 24: Line 76:
 
==== Test ====
 
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 5 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 410m -F -o /mnt/fhgfs/test.ior  
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 
==== Result ====
 
==== Result ====
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 34: Line 87:
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
 
         ordering inter file=constant task offsets = 1
 
         ordering inter file=constant task offsets = 1
−
         clients            = 5 (1 per node)
+
         clients            = 8 (1 per node)
 
         repetitions        = 5
 
         repetitions        = 5
 
         xfersize          = 524288 bytes
 
         xfersize          = 524288 bytes
−
         blocksize          = 410 MiB
+
         blocksize          = 256 MiB
−
         aggregate filesize = 2.00 GiB
+
         aggregate filesize = 2 GiB
  
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
write        137.05     127.16     131.73     3.33     274.10     254.32     263.46     6.67 15.57218   EXCEL
+
write        133.09     129.90     131.16     1.17     266.18     259.79     262.32     2.34 15.61596   EXCEL
−
read          334.68     305.41     316.45    10.08     669.35     610.82     632.91     20.16   6.48446   EXCEL
+
read          340.31     321.88     334.40      6.63     680.62     643.76     668.79     13.25   6.12693   EXCEL
  
−
Max Write: 137.05 MiB/sec (143.71 MB/sec)
+
Max Write: 133.09 MiB/sec (139.56 MB/sec)
−
Max Read:  334.68 MiB/sec (350.93 MB/sec)
+
Max Read:  340.31 MiB/sec (356.84 MB/sec)
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 71: Line 124:
 
==== Test ====
 
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 5 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 820m -F -o /mnt/fhgfs/test.ior
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 82: Line 135:
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
 
         ordering inter file=constant task offsets = 1
 
         ordering inter file=constant task offsets = 1
−
         clients            = 5 (1 per node)
+
         clients            = 8 (1 per node)
 
         repetitions        = 5
 
         repetitions        = 5
 
         xfersize          = 524288 bytes
 
         xfersize          = 524288 bytes
−
         blocksize          = 820 MiB
+
         blocksize          = 512 MiB
−
         aggregate filesize = 4.00 GiB
+
         aggregate filesize = 4 GiB
  
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
−
write        236.70     227.91     232.03     2.97     473.41     455.83     464.05     5.93 17.67324   EXCEL
+
write        264.27     252.20     258.31     4.50     528.53     504.40     516.63     9.00 15.86156   EXCEL
−
read          417.67     355.39     392.24     22.96    835.34     710.79      784.49     45.91  10.48953   EXCEL
+
read          600.25     534.51     564.42     22.51    1200.49    1069.03     1128.85     45.03  7.26843   EXCEL
  
−
Max Write: 236.70 MiB/sec (248.20 MB/sec)
+
Max Write: 264.27 MiB/sec (277.10 MB/sec)
−
Max Read:  417.67 MiB/sec (437.96 MB/sec)
+
Max Read:  600.25 MiB/sec (629.40 MB/sec)
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 
=== Six Storage Servers ===
 
=== Six Storage Servers ===
 
==== Tuning ====
 
==== Tuning ====
 +
Striping across six storage servers.
 +
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 6
 +
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 6
 +
</syntaxhighlight>
 
==== Test ====
 
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
 +
 
==== Result ====
 
==== Result ====
 +
<syntaxhighlight>
 +
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 768 MiB
 +
        aggregate filesize = 6 GiB
 +
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        360.60    335.26      344.56      8.61    721.20    670.53      689.13    17.23  17.84209  EXCEL
 +
read          629.99    596.70      607.71    11.88    1259.99    1193.40    1215.43    23.77  10.11383  EXCEL
 +
 +
Max Write: 360.60 MiB/sec (378.11 MB/sec)
 +
Max Read:  629.99 MiB/sec (660.60 MB/sec)
 +
</syntaxhighlight>
 
=== Eight Storage Servers ===
 
=== Eight Storage Servers ===
 
==== Tuning ====
 
==== Tuning ====
 +
Striping across eight storage servers.
 +
<syntaxhighlight>
 +
Number of nodes: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 8
 +
</syntaxhighlight>
 
==== Test ====
 
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
 +
 
==== Result ====
 
==== Result ====
 +
<syntaxhighlight>
 +
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 1 GiB
 +
        aggregate filesize = 8 GiB
 +
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        371.61    357.65      366.43      4.82    743.21    715.30      732.85      9.63  22.36045  EXCEL
 +
read          648.27    564.96      602.26    27.39    1296.53    1129.92    1204.52    54.78  13.62983  EXCEL
 +
 +
Max Write: 371.61 MiB/sec (389.66 MB/sec)
 +
Max Read:  648.27 MiB/sec (679.76 MB/sec)
 +
</syntaxhighlight>

Latest revision as of 09:59, 7 February 2014

Network Performance

Iperf

Iperf will validate the efficient bandwidth between every machines, the real network connection speed. 

root@cloud32:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
------------------------------------------------------------
Client connecting to 172.28.12.32, TCP port 5001
TCP window size: 21.6 KByte (default)
------------------------------------------------------------
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  1.0- 2.0 sec   172 MBytes  1.44 Gbits/sec
[  3]  2.0- 3.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  3.0- 4.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  4.0- 5.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  5.0- 6.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  6.0- 7.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  7.0- 8.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  8.0- 9.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  9.0-10.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec

Netcat

Netcat will determine the current bandwidth for the writes. 

root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
Listening on [0.0.0.0] (family 0, port 2222)
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
^C3884852+0 records in
3884851+0 records out
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s

real    0m45.953s
user    0m3.719s
sys     1m2.406s

Multi Stream Throughput

Eight clients, filesize of ~1GB per storage server

Two Storage Servers

Tuning

Striping across two servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 2

root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 2

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 256 MiB
        aggregate filesize = 2 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         133.09     129.90      131.16      1.17     266.18     259.79      262.32      2.34  15.61596   EXCEL
read          340.31     321.88      334.40      6.63     680.62     643.76      668.79     13.25   6.12693   EXCEL

Max Write: 133.09 MiB/sec (139.56 MB/sec)
Max Read:  340.31 MiB/sec (356.84 MB/sec)

Four Storage Servers

Tuning

Striping across four storage servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 4

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 4

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 512 MiB
        aggregate filesize = 4 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         264.27     252.20      258.31      4.50     528.53     504.40      516.63      9.00  15.86156   EXCEL
read          600.25     534.51      564.42     22.51    1200.49    1069.03     1128.85     45.03   7.26843   EXCEL

Max Write: 264.27 MiB/sec (277.10 MB/sec)
Max Read:  600.25 MiB/sec (629.40 MB/sec)

Six Storage Servers

Tuning

Striping across six storage servers. 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 6

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 6

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 768 MiB
        aggregate filesize = 6 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         360.60     335.26      344.56      8.61     721.20     670.53      689.13     17.23  17.84209   EXCEL
read          629.99     596.70      607.71     11.88    1259.99    1193.40     1215.43     23.77  10.11383   EXCEL

Max Write: 360.60 MiB/sec (378.11 MB/sec)
Max Read:  629.99 MiB/sec (660.60 MB/sec)

Eight Storage Servers

Tuning

Striping across eight storage servers. 

Number of nodes: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 8

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 1 GiB
        aggregate filesize = 8 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         371.61     357.65      366.43      4.82     743.21     715.30      732.85      9.63  22.36045   EXCEL
read          648.27     564.96      602.26     27.39    1296.53    1129.92     1204.52     54.78  13.62983   EXCEL

Max Write: 371.61 MiB/sec (389.66 MB/sec)
Max Read:  648.27 MiB/sec (679.76 MB/sec)
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=FhGFS_Benchmarking&oldid=3956"