IEEL 2.2 - MDTEST for Unit.tv
Download MDTEST from Sourceforge: [1]
Make a directory called mdtest - copy and Untar MDTEST into that directory.
mkdir /root/mdtest
tar xvf mdtest-1.9.3.tgzEdit Makefile to point CC.Linux to current MPI and then make - My example I modified to:
CC.Linux = /usr/mpi/gcc/openmpi-1.8.4/bin/mpicc
makeInitial Testing & Observations:
Basic run:'
./mdtest -n 1024 -d /mnt/lustrefs-ib/mdtest-n Total amount of files to run per process (Standard run is 1 process)
Optimised Run:
[root@mds1 mdtest]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
-- started at 07/24/2015 03:59:48 --
mdtest-1.9.3 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
16 tasks, 100800 files/directories
SUMMARY: (of 30 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation: 8771.583 8257.319 8498.145 142.601
Directory stat : 51249.434 42772.707 46890.859 1841.967
Directory removal : 8743.787 8206.951 8402.380 133.065
File creation : 7381.270 6614.801 7078.306 143.470
File stat : 31104.592 26010.224 28677.905 1262.069
File read : 11800.048 11575.249 11695.937 65.612
File removal : 9023.598 8360.099 8595.353 144.175
Tree creation : 8087.447 4717.328 6582.154 625.018
Tree removal : 3354.847 1011.802 2532.170 817.395Observations
Looking at the numbers file creation seems to be bound by the number of nodes whereas stat seems to scale per process. Tree creation and removal seem to improve as you add clients but performance drops off the more processes you run per client, so a higher number of clients and lower number of processes per client would be better – like here: https://jira.hpdd.intel.com/browse/LU-5608 – 32 & 64
Other Results
2 Clients - 8 Threads Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 08:42:32 --
mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
16 tasks, 256000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 6226.078 6196.187 6211.441 12.211
File stat : 34523.900 34191.504 34347.555 136.457
File read : 11451.576 11281.008 11389.401 76.921
File removal : 5026.970 4975.436 5002.181 21.084
Tree creation : 864.448 127.405 537.898 306.695
Tree removal : 2.007 1.935 1.977 0.031
-- finished at 07/22/2015 08:49:11 --2 Clients - 4 Threads Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 8 -npernode 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 08:50:12 --
mdtest-1.9.1 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
8 tasks, 128000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 6529.571 6449.035 6486.553 33.108
File stat : 17431.529 17044.547 17276.888 167.265
File read : 11457.978 11408.014 11435.052 20.604
File removal : 4216.588 4173.080 4201.648 20.207
Tree creation : 1848.525 350.753 1280.509 662.808
Tree removal : 198.293 173.577 189.445 11.245
-- finished at 07/22/2015 08:54:10 --1 Client - 8 Threads Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 08:36:41 --
mdtest-1.9.1 was launched with 8 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
8 tasks, 128000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 3536.366 3504.754 3524.054 13.819
File stat : 18162.745 17849.431 18005.063 127.918
File read : 6628.013 6431.326 6557.983 89.727
File removal : 4340.580 4249.596 4307.685 41.195
Tree creation : 577.092 521.096 541.044 25.538
Tree removal : 65.604 50.556 57.142 6.285
-- finished at 07/22/2015 08:41:49 --1 Client - 4 Threds Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 09:00:15 --
mdtest-1.9.1 was launched with 4 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
4 tasks, 64000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 3761.743 3676.986 3722.687 34.920
File stat : 10465.463 9879.783 10222.287 249.205
File read : 6832.391 6672.419 6764.390 67.472
File removal : 4540.434 4496.427 4520.446 18.191
Tree creation : 975.646 104.080 682.572 409.068
Tree removal : 139.647 138.353 139.054 0.534
-- finished at 07/22/2015 09:03:06 --