FhGFS Benchmarking
Network Performance
Iperf
Iperf will validate the efficient bandwidth between every machines, the real network connection speed.
root@cloud32:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 172.28.12.32 port 5001 connected with 172.28.12.24 port 40038
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.67 GBytes 1.44 Gbits/secroot@cloud24:~/fhgfs-benchmark# iperf -c 172.28.12.32 -i1 -t 10
------------------------------------------------------------
Client connecting to 172.28.12.32, TCP port 5001
TCP window size: 21.6 KByte (default)
------------------------------------------------------------
[ 3] local 172.28.12.24 port 40038 connected with 172.28.12.32 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 1.0- 2.0 sec 172 MBytes 1.44 Gbits/sec
[ 3] 2.0- 3.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 3.0- 4.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 4.0- 5.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 5.0- 6.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 6.0- 7.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 7.0- 8.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 8.0- 9.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 9.0-10.0 sec 171 MBytes 1.44 Gbits/sec
[ 3] 0.0-10.0 sec 1.67 GBytes 1.44 Gbits/secNetcat
Netcat will determine the current bandwidth for the writes.
root@cloud32:~# nc -v -v -l -n 2222 >/dev/null
Listening on [0.0.0.0] (family 0, port 2222)
Connection from [172.28.12.24] port 2222 [tcp/*] accepted (family 2, sport 39647)root@cloud24:~/fhgfs-benchmark# time dd if=/dev/zero | nc -v -v -n 172.28.12.32 2222
Connection to 172.28.12.32 2222 port [tcp/*] succeeded!
^C3884852+0 records in
3884851+0 records out
1989043712 bytes (2.0 GB) copied, 45.9459 s, 43.3 MB/s
real 0m45.953s
user 0m3.719s
sys 1m2.406sMulti Stream Throughput
5 clients, filesize of ~1GB per storage server
Two Storage Servers
Tuning
Striping across two servers
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=2 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 2
root@cloud28:~# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 2Test
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 5 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 410m -F -o /mnt/fhgfs/test.iorResult
Summary:
api = POSIX
test filename = /mnt/fhgfs/test.ior
access = file-per-process
ordering in a file = sequential offsets
ordering inter file=constant task offsets = 1
clients = 5 (1 per node)
repetitions = 5
xfersize = 524288 bytes
blocksize = 410 MiB
aggregate filesize = 2.00 GiB
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 137.05 127.16 131.73 3.33 274.10 254.32 263.46 6.67 15.57218 EXCEL
read 334.68 305.41 316.45 10.08 669.35 610.82 632.91 20.16 6.48446 EXCEL
Max Write: 137.05 MiB/sec (143.71 MB/sec)
Max Read: 334.68 MiB/sec (350.93 MB/sec)Four Storage Servers
Tuning
Striping across four storage servers
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=4 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 4
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 4Test
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 5 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 820m -F -o /mnt/fhgfs/test.iorResult
Summary:
api = POSIX
test filename = /mnt/fhgfs/test.ior
access = file-per-process
ordering in a file = sequential offsets
ordering inter file=constant task offsets = 1
clients = 5 (1 per node)
repetitions = 5
xfersize = 524288 bytes
blocksize = 820 MiB
aggregate filesize = 4.00 GiB
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 236.70 227.91 232.03 2.97 473.41 455.83 464.05 5.93 17.67324 EXCEL
read 417.67 355.39 392.24 22.96 835.34 710.79 784.49 45.91 10.48953 EXCEL
Max Write: 236.70 MiB/sec (248.20 MB/sec)
Max Read: 417.67 MiB/sec (437.96 MB/sec)Six Storage Servers
Tuning
Striping across six storage servers.
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=6 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 6
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 6Test
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 5 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1229m -F -o /mnt/fhgfs/test.iorResult
Summary:
api = POSIX
test filename = /mnt/fhgfs/test.ior
access = file-per-process
ordering in a file = sequential offsets
ordering inter file=constant task offsets = 1
clients = 5 (1 per node)
repetitions = 5
xfersize = 524288 bytes
blocksize = 1.20 GiB
aggregate filesize = 6.00 GiB
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 253.46 246.93 250.55 2.26 506.93 493.86 501.11 4.52 24.52773 EXCEL
read 403.10 380.50 392.76 7.41 806.20 761.00 785.52 14.83 15.65139 EXCEL
Max Write: 253.46 MiB/sec (265.77 MB/sec)
Max Read: 403.10 MiB/sec (422.68 MB/sec)Eight Storage Servers
Tuning
Striping across eight storage servers.
Number of nodes: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --setpattern --numtargets=8 --chunksize=512k /mnt/fhgfs
Mount: '/mnt/fhgfs'; Path: ''
Chunksize: 524288
Number of storage targets: 8
root@cloud24:~/fhgfs-benchmark# fhgfs-ctl --getentryinfo /mnt/fhgfs
Path:
Mount: /mnt/fhgfs
EntryID: root
Metadata node: cloud32 [ID: 13708]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 8Test
root@cloud24:~/fhgfs-benchmark# mpirun -hostfile hostfile -np 8 /root/ior/IOR/src/C/IOR -wr -C -i5 -t512k -b 1024m -F -o /mnt/fhgfs/test.iorResult
Summary:
api = POSIX
test filename = /mnt/fhgfs/test.ior
access = file-per-process
ordering in a file = sequential offsets
ordering inter file=constant task offsets = 1
clients = 8 (1 per node)
repetitions = 5
xfersize = 524288 bytes
blocksize = 1 GiB
aggregate filesize = 8 GiB
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 371.61 357.65 366.43 4.82 743.21 715.30 732.85 9.63 22.36045 EXCEL
read 648.27 564.96 602.26 27.39 1296.53 1129.92 1204.52 54.78 13.62983 EXCEL
Max Write: 371.61 MiB/sec (389.66 MB/sec)
Max Read: 648.27 MiB/sec (679.76 MB/sec)