FhGFS Benchmarking
Jump to navigation
Jump to search
Test System 1
fhgfs-net
[root@blade1 C]# fhgfs-net
mgmt_nodes
=============
Blade9 [ID: 1]
Connections: TCP: 1 (101.101.101.9:8008);
meta_nodes
=============
Blade9 [ID: 11201]
Connections: TCP: 1 (101.101.101.9:8005);
storage_nodes
=============
Blade4 [ID: 3855]
Connections: TCP: 1 (101.101.101.4:8003);
Blade6 [ID: 17264]
Connections: TCP: 1 (101.101.101.6:8003);
Blade10 [ID: 18046]
Connections: TCP: 1 (101.101.101.10:8003);
Blade5 [ID: 32274]
Connections: TCP: 1 (101.101.101.5:8003);
Blade7 [ID: 41907]
Connections: TCP: 1 (101.101.101.7:8003);
Blade8 [ID: 56479]
Connections: TCP: 1 (101.101.101.8:8003);
Blade3 [ID: 60659]
Connections: TCP: 1 (101.101.101.3:8003);fhgfs-check-servers
[root@blade1 C]# fhgfs-check-servers
Management
==========
Blade9 [ID: 1]: reachable
Metadata
==========
Blade9 [ID: 11201]: reachable
Storage
==========
Blade4 [ID: 3855]: reachable
Blade6 [ID: 17264]: reachable
Blade10 [ID: 18046]: reachable
Blade5 [ID: 32274]: reachable
Blade7 [ID: 41907]: reachable
Blade8 [ID: 56479]: reachable
Blade3 [ID: 60659]: reachablefhgfs-df
[root@blade1 C]# fhgfs-df
METADATA SERVERS:
TargetID Pool Total Free ITotal IFree
======== ==== ===== ==== ====== =====
11201 [normal] 49.1GB 46.6GB 3.1M 3.0M
STORAGE TARGETS:
TargetID Pool Total Free ITotal IFree
======== ==== ===== ==== ====== =====
9039 [low] 49.1GB 45.1GB 3.1M 3.1M
23710 [low] 49.1GB 44.5GB 3.1M 3.0M
26221 [low] 49.1GB 44.5GB 3.1M 3.1M
34234 [low] 49.1GB 45.7GB 3.1M 3.1M
36548 [low] 49.1GB 45.4GB 3.1M 3.0M
57799 [low] 49.1GB 45.1GB 3.1M 3.1M
63163 [low] 49.1GB 45.1GB 3.1M 3.1MIOR
Test 1
[root@blade1 C]# ./IORRun began: Tue Aug 6 07:46:08 2013
Command line used: ./IOR
Machine: Linux blade1
Summary:
api = POSIX
test filename = testFile
access = single-shared-file
ordering in a file = sequential offsets
ordering inter file= no tasks offsets
clients = 1 (1 per node)
repetitions = 1
xfersize = 262144 bytes
blocksize = 1 MiB
aggregate filesize = 1 MiB
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 531.93 531.93 531.93 0.00 2127.74 2127.74 2127.74 0.00 0.00188 EXCEL
read 1326.47 1326.47 1326.47 0.00 5305.89 5305.89 5305.89 0.00 0.00075 EXCEL
Max Write: 531.93 MiB/sec (557.77 MB/sec)
Max Read: 1326.47 MiB/sec (1390.91 MB/sec)
Run finished: Tue Aug 6 07:46:08 2013Test 2
[root@blade1 C]# ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfileRun began: Tue Aug 6 07:35:35 2013
Command line used: ./IOR -vv -k -wWr -C -F -i4 -t 256k -b 10m -s574 -o /mnt/fhgfs/iorfile
Machine: Linux blade1 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64
Using synchronized MPI timer
Start time skew across all tasks: 0.00 sec
Path: /mnt/fhgfs
FS: 343.6 GiB Used FS: 6.6% Inodes: 0.0 Mi Used Inodes: -nan%
Participating tasks: 1
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on blade1
Summary:
api = POSIX
test filename = /mnt/fhgfs/iorfile
access = file-per-process
pattern = strided (574 segments)
ordering in a file = sequential offsets
ordering inter file=constant task offsets = 1
clients = 1 (1 per node)
repetitions = 4
xfersize = 262144 bytes
blocksize = 10 MiB
aggregate filesize = 5.61 GiB
Using Time Stamp 1375788940 (0x5200df8c) for Data Signature
Commencing write performance test.
Tue Aug 6 07:35:40 2013
access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---------- --------- -------- -------- -------- -------- ----
write 532.48 10240 256.00 0.000449 10.78 0.000363 10.78 0 XXCEL
Verifying contents of the file(s) just written.
Tue Aug 6 07:35:56 2013
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
Commencing read performance test.
Tue Aug 6 07:36:14 2013
read 475.34 10240 256.00 0.001303 12.07 0.000385 12.08 0 XXCEL
Using Time Stamp 1375788986 (0x5200dfba) for Data Signature
Commencing write performance test.
Tue Aug 6 07:36:26 2013
write 270.84 10240 256.00 0.238056 20.96 0.000310 21.19 1 XXCEL
Verifying contents of the file(s) just written.
Tue Aug 6 07:36:47 2013
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
Commencing read performance test.
Tue Aug 6 07:37:04 2013
read 496.80 10240 256.00 0.001265 11.55 0.000352 11.55 1 XXCEL
Using Time Stamp 1375789036 (0x5200dfec) for Data Signature
Commencing write performance test.
Tue Aug 6 07:37:16 2013
write 217.72 10240 256.00 0.270669 26.09 0.000319 26.36 2 XXCEL
Verifying contents of the file(s) just written.
Tue Aug 6 07:37:42 2013
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
Commencing read performance test.
Tue Aug 6 07:38:00 2013
read 497.94 10240 256.00 0.001282 11.53 0.000351 11.53 2 XXCEL
Using Time Stamp 1375789091 (0x5200e023) for Data Signature
Commencing write performance test.
Tue Aug 6 07:38:11 2013
write 219.26 10240 256.00 0.235869 25.94 0.000313 26.18 3 XXCEL
Verifying contents of the file(s) just written.
Tue Aug 6 07:38:37 2013
[RANK 000] open for reading file /mnt/fhgfs/iorfile.00000000 XXCEL
Commencing read performance test.
Tue Aug 6 07:38:55 2013
read 468.33 10240 256.00 0.001103 12.25 0.000376 12.26 3 XXCEL
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 532.48 217.72 310.07 130.17 3.71 1.52 2.16 0.91 21.12902 1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
read 497.94 468.33 484.60 13.01 3.47 3.26 3.38 0.09 11.85335 1 1 4 1 1 1 0 0 574 10485760 262144 6018826240 -1 POSIX EXCEL
Max Write: 532.48 MiB/sec (558.35 MB/sec)
Max Read: 497.94 MiB/sec (522.12 MB/sec)
Run finished: Tue Aug 6 07:39:07 2013MDTEST
Test 1
-- started at 08/06/2013 07:49:58 --
mdtest-1.9.1 was launched with 1 total task(s) on 1 node(s)
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -F -C -T -r -N 1
Path: /mnt/fhgfs
FS: 343.6 GiB Used FS: 8.3% Inodes: 0.0 Mi Used Inodes: -nan%
1 tasks, 1024 files
SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 2071.708 2071.708 2071.708 0.000
File stat : 6424.898 6424.898 6424.898 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 1879.944 1879.944 1879.944 0.000
Tree creation : 1157.688 1157.688 1157.688 0.000
Tree removal : 790.483 790.483 790.483 0.000
-- finished at 08/06/2013 07:49:59 --Test 2
-- started at 08/06/2013 07:51:56 --
mdtest-1.9.1 was launched with 1 total task(s) on 1 node(s)
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -S -C -T -r -N 1
Path: /mnt/fhgfs
FS: 343.6 GiB Used FS: 8.3% Inodes: 0.0 Mi Used Inodes: -nan%
1 tasks, 1 file
SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 2737.946 2737.946 2737.946 0.000
File stat : 199.851 199.851 199.851 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 2503.440 2503.440 2503.440 0.000
Tree creation : 1277.194 1277.194 1277.194 0.000
Tree removal : 1117.289 1117.289 1117.289 0.000
-- finished at 08/06/2013 07:52:02 --Test 3
-- started at 08/06/2013 07:53:42 --
mdtest-1.9.1 was launched with 2 total task(s) on 1 node(s)
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -S -C -T -r -n 1
Path: /mnt/fhgfs
FS: 343.6 GiB Used FS: 8.3% Inodes: 0.0 Mi Used Inodes: -nan%
2 tasks, 1 file
SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.400 0.400 0.400 0.000
File stat : 1558.931 1558.931 1558.931 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 3718.355 3718.355 3718.355 0.000
Tree creation : 1237.623 1237.623 1237.623 0.000
Tree removal : 896.027 896.027 896.027 0.000
-- finished at 08/06/2013 07:53:47 --Test 4
-- started at 08/06/2013 07:55:05 --
mdtest-1.9.1 was launched with 10 total task(s) on 1 node(s)
Command line used: ./mdtest -n 1024 -d /mnt/fhgfs/MDTEST -S -C -T -r -n 1
Path: /mnt/fhgfs
FS: 343.6 GiB Used FS: 8.3% Inodes: 0.0 Mi Used Inodes: -nan%
10 tasks, 1 file
SUMMARY: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 1563.231 1563.231 1563.231 0.000
File stat : 10988.483 10988.483 10988.483 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 23750.306 23750.306 23750.306 0.000
Tree creation : 1524.647 1524.647 1524.647 0.000
Tree removal : 1058.366 1058.366 1058.366 0.000
-- finished at 08/06/2013 07:55:05 --