Difference between revisions of "FhGFS Benchmarking"

From Define Wiki
Jump to navigation Jump to search
 
(22 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
== Network Performance ==
 +
=== Iperf ===
 +
Iperf will validate the efficient bandwidth between every machines, the real network connection speed.
 +
<syntaxhighlight>
 +
root@cloud32:~# iperf -s
 +
------------------------------------------------------------
 +
Server listening on TCP port 5001
 +
TCP window size: 85.3 KByte (default)
 +
------------------------------------------------------------
 +
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
 +
[ ID] Interval      Transfer    Bandwidth
 +
[  4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
 +
------------------------------------------------------------
 +
Client connecting to 172.28.12.32, TCP port 5001
 +
TCP window size: 21.6 KByte (default)
 +
------------------------------------------------------------
 +
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
 +
[ ID] Interval      Transfer    Bandwidth
 +
[  3]  0.0- 1.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  1.0- 2.0 sec  172 MBytes  1.44 Gbits/sec
 +
[  3]  2.0- 3.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  3.0- 4.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  4.0- 5.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  5.0- 6.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  6.0- 7.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  7.0- 8.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  8.0- 9.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  9.0-10.0 sec  171 MBytes  1.44 Gbits/sec
 +
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
 +
</syntaxhighlight>
 +
=== Netcat ===
 +
Netcat will determine the current bandwidth for the writes.
 +
<syntaxhighlight>
 +
root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
 +
Listening on [0.0.0.0] (family 0, port 2222)
 +
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
 +
</syntaxhighlight>
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
 +
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
 +
^C3884852+0 records in
 +
3884851+0 records out
 +
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s
 +
 +
real    0m45.953s
 +
user    0m3.719s
 +
sys    1m2.406s
 +
</syntaxhighlight>
 +
 
== Multi Stream Throughput ==
 
== Multi Stream Throughput ==
−
Tests with 5 clients reading and writing 1GB each
+
<div style="border: 1px #888888 solid; background-color: #e57f52;padding: 3px;">'''Eight clients, filesize of ~1GB per storage server''' </div>
 
=== Two Storage Servers ===
 
=== Two Storage Servers ===
 
==== Tuning ====
 
==== Tuning ====
 +
Striping across two servers
 +
 
<syntaxhighlight>
 
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 2
 +
 
root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
 
root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
 
Path:
 
Path:
Line 14: Line 73:
 
+ Number of storage targets: desired: 2
 
+ Number of storage targets: desired: 2
 
</syntaxhighlight>
 
</syntaxhighlight>
−
==== Test command ====
+
 
 +
==== Test ====
 
<syntaxhighlight>
 
<syntaxhighlight>
−
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 5 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1g -F -o /mnt/fhgfs/test.ior
+
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 
==== Result ====
 
==== Result ====
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 26: Line 87:
 
         ordering in a file = sequential offsets
 
         ordering in a file = sequential offsets
 
         ordering inter file=constant task offsets = 1
 
         ordering inter file=constant task offsets = 1
−
         clients            = 5 (1 per node)
+
         clients            = 8 (1 per node)
 
         repetitions        = 5
 
         repetitions        = 5
 
         xfersize          = 524288 bytes
 
         xfersize          = 524288 bytes
−
         blocksize          = 1 GiB
+
         blocksize          = 256 MiB
−
         aggregate filesize = 5 GiB
+
         aggregate filesize = 2 GiB
 +
 
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        133.09    129.90      131.16      1.17    266.18    259.79      262.32      2.34  15.61596  EXCEL
 +
read          340.31    321.88      334.40      6.63    680.62    643.76      668.79    13.25  6.12693  EXCEL
 +
 
 +
Max Write: 133.09 MiB/sec (139.56 MB/sec)
 +
Max Read:  340.31 MiB/sec (356.84 MB/sec)
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 
=== Four Storage Servers ===
 
=== Four Storage Servers ===
 +
==== Tuning ====
 +
Striping across four storage servers
 +
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 4
 +
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 4
 +
</syntaxhighlight>
 +
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
 +
 +
==== Result ====
 +
<syntaxhighlight>
 +
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 512 MiB
 +
        aggregate filesize = 4 GiB
 +
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        264.27    252.20      258.31      4.50    528.53    504.40      516.63      9.00  15.86156  EXCEL
 +
read          600.25    534.51      564.42    22.51    1200.49    1069.03    1128.85    45.03  7.26843  EXCEL
 +
 +
Max Write: 264.27 MiB/sec (277.10 MB/sec)
 +
Max Read:  600.25 MiB/sec (629.40 MB/sec)
 +
</syntaxhighlight>
 +
 
=== Six Storage Servers ===
 
=== Six Storage Servers ===
 +
==== Tuning ====
 +
Striping across six storage servers.
 +
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 6
 +
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 6
 +
</syntaxhighlight>
 +
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
 +
 +
==== Result ====
 +
<syntaxhighlight>
 +
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 768 MiB
 +
        aggregate filesize = 6 GiB
 +
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        360.60    335.26      344.56      8.61    721.20    670.53      689.13    17.23  17.84209  EXCEL
 +
read          629.99    596.70      607.71    11.88    1259.99    1193.40    1215.43    23.77  10.11383  EXCEL
 +
 +
Max Write: 360.60 MiB/sec (378.11 MB/sec)
 +
Max Read:  629.99 MiB/sec (660.60 MB/sec)
 +
</syntaxhighlight>
 
=== Eight Storage Servers ===
 
=== Eight Storage Servers ===
 +
==== Tuning ====
 +
Striping across eight storage servers.
 +
<syntaxhighlight>
 +
Number of nodes: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
 +
Mount: '/mnt/fhgfs'; Path: ''
 +
Chunksize: 524288
 +
Number of storage targets: 8
 +
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
 +
Path:
 +
Mount: /mnt/fhgfs
 +
EntryID: root
 +
Metadata node: cloud32 [ID: 13708]
 +
Stripe pattern details:
 +
+ Type: RAID0
 +
+ Chunksize: 512K
 +
+ Number of storage targets: desired: 8
 +
</syntaxhighlight>
 +
==== Test ====
 +
<syntaxhighlight>
 +
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior
 +
</syntaxhighlight>
 +
 +
==== Result ====
 +
<syntaxhighlight>
 +
Summary:
 +
        api                = POSIX
 +
        test filename      = /mnt/fhgfs/test.ior
 +
        access            = file-per-process
 +
        ordering in a file = sequential offsets
 +
        ordering inter file=constant task offsets = 1
 +
        clients            = 8 (1 per node)
 +
        repetitions        = 5
 +
        xfersize          = 524288 bytes
 +
        blocksize          = 1 GiB
 +
        aggregate filesize = 8 GiB
 +
 +
Operation  Max (MiB)  Min (MiB)  Mean (MiB)  Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)  Std Dev  Mean (s)
 +
---------  ---------  ---------  ----------  -------  ---------  ---------  ----------  -------  --------
 +
write        371.61    357.65      366.43      4.82    743.21    715.30      732.85      9.63  22.36045  EXCEL
 +
read          648.27    564.96      602.26    27.39    1296.53    1129.92    1204.52    54.78  13.62983  EXCEL
 +
 +
Max Write: 371.61 MiB/sec (389.66 MB/sec)
 +
Max Read:  648.27 MiB/sec (679.76 MB/sec)
 +
</syntaxhighlight>

Latest revision as of 09:59, 7 February 2014

Network Performance

Iperf

Iperf will validate the efficient bandwidth between every machines, the real network connection speed. 

root@cloud32:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec
root@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
------------------------------------------------------------
Client connecting to 172.28.12.32, TCP port 5001
TCP window size: 21.6 KByte (default)
------------------------------------------------------------
[  3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  1.0- 2.0 sec   172 MBytes  1.44 Gbits/sec
[  3]  2.0- 3.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  3.0- 4.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  4.0- 5.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  5.0- 6.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  6.0- 7.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  7.0- 8.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  8.0- 9.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  9.0-10.0 sec   171 MBytes  1.44 Gbits/sec
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec

Netcat

Netcat will determine the current bandwidth for the writes. 

root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
Listening on [0.0.0.0] (family 0, port 2222)
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)
root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
^C3884852+0 records in
3884851+0 records out
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s

real    0m45.953s
user    0m3.719s
sys     1m2.406s

Multi Stream Throughput

Eight clients, filesize of ~1GB per storage server

Two Storage Servers

Tuning

Striping across two servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 2

root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 2

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 256m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 256 MiB
        aggregate filesize = 2 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         133.09     129.90      131.16      1.17     266.18     259.79      262.32      2.34  15.61596   EXCEL
read          340.31     321.88      334.40      6.63     680.62     643.76      668.79     13.25   6.12693   EXCEL

Max Write: 133.09 MiB/sec (139.56 MB/sec)
Max Read:  340.31 MiB/sec (356.84 MB/sec)

Four Storage Servers

Tuning

Striping across four storage servers 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 4

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 4

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 512m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 512 MiB
        aggregate filesize = 4 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         264.27     252.20      258.31      4.50     528.53     504.40      516.63      9.00  15.86156   EXCEL
read          600.25     534.51      564.42     22.51    1200.49    1069.03     1128.85     45.03   7.26843   EXCEL

Max Write: 264.27 MiB/sec (277.10 MB/sec)
Max Read:  600.25 MiB/sec (629.40 MB/sec)

Six Storage Servers

Tuning

Striping across six storage servers. 

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 6

root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 6

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 768m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 768 MiB
        aggregate filesize = 6 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         360.60     335.26      344.56      8.61     721.20     670.53      689.13     17.23  17.84209   EXCEL
read          629.99     596.70      607.71     11.88    1259.99    1193.40     1215.43     23.77  10.11383   EXCEL

Max Write: 360.60 MiB/sec (378.11 MB/sec)
Max Read:  629.99 MiB/sec (660.60 MB/sec)

Eight Storage Servers

Tuning

Striping across eight storage servers. 

Number of nodes: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 8

Test

root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.ior

Result

Summary:
        api                = POSIX
        test filename      = /mnt/fhgfs/test.ior
        access             = file-per-process
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (1 per node)
        repetitions        = 5
        xfersize           = 524288 bytes
        blocksize          = 1 GiB
        aggregate filesize = 8 GiB

Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)
---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write         371.61     357.65      366.43      4.82     743.21     715.30      732.85      9.63  22.36045   EXCEL
read          648.27     564.96      602.26     27.39    1296.53    1129.92     1204.52     54.78  13.62983   EXCEL

Max Write: 371.61 MiB/sec (389.66 MB/sec)
Max Read:  648.27 MiB/sec (679.76 MB/sec)
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=FhGFS_Benchmarking&oldid=3956"