IEEL 2.2 - MDTEST for Unit.tv
Jump to navigation
Jump to search
Download MDTEST from Sourceforge: [1]
Make a directory called mdtest - copy and Untar MDTEST into that directory.
mkdir /root/mdtest
tar xvf mdtest-1.9.3.tgzEdit Makefile to point CC.Linux to current MPI and then make - My example I modified to:
CC.Linux = /usr/mpi/gcc/openmpi-1.8.4/bin/mpicc
makeInitial Testing & Observations:
Basic run:'
./mdtest -n 1024 -d /mnt/lustrefs-ib/mdtest-n Total amount of files to run per process (Standard run is 1 process)
Optimised Run:
/usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1-i Total Number of Test Itterations -b Branching Factor of hierarchical directory structure -I Number of Items per directory tree -z Depth of hierarchical directory structure -d Test directory
Output
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
-- started at 07/22/2015 09:34:24 --
mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
16 tasks, 80640 files/directories
SUMMARY: (of 20 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation: 8763.188 8232.435 8376.081 117.579
Directory stat : 51421.884 44970.310 47884.412 1761.017
Directory removal : 8541.242 7992.884 8339.238 135.393
File creation : 7226.927 6714.372 7041.462 152.999
File stat : 28105.169 26091.429 27392.123 537.488
File read : 11582.120 11297.195 11443.001 77.403
File removal : 8581.734 8033.429 8385.055 125.556
Tree creation : 7627.989 5464.496 6619.557 367.386
Tree removal : 3626.299 2855.457 3308.709 131.297
-- finished at 07/22/2015 09:51:46 --Observations
Looking at the numbers file creation seems to be bound by the number of nodes whereas stat seems to scale per process. Tree creation and removal seem to improve as you add clients but performance drops off the more processes you run per client, so a higher number of clients and lower number of processes per client would be better – like here: https://jira.hpdd.intel.com/browse/LU-5608 – 32 & 64
Other Results
2 Clients - 8 Threads Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 08:42:32 --
mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
16 tasks, 256000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 6226.078 6196.187 6211.441 12.211
File stat : 34523.900 34191.504 34347.555 136.457
File read : 11451.576 11281.008 11389.401 76.921
File removal : 5026.970 4975.436 5002.181 21.084
Tree creation : 864.448 127.405 537.898 306.695
Tree removal : 2.007 1.935 1.977 0.031
-- finished at 07/22/2015 08:49:11 --2 Clients - 4 Threads Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 8 -npernode 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 08:50:12 --
mdtest-1.9.1 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
8 tasks, 128000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 6529.571 6449.035 6486.553 33.108
File stat : 17431.529 17044.547 17276.888 167.265
File read : 11457.978 11408.014 11435.052 20.604
File removal : 4216.588 4173.080 4201.648 20.207
Tree creation : 1848.525 350.753 1280.509 662.808
Tree removal : 198.293 173.577 189.445 11.245
-- finished at 07/22/2015 08:54:10 --1 Client - 8 Threads Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 08:36:41 --
mdtest-1.9.1 was launched with 8 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
8 tasks, 128000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 3536.366 3504.754 3524.054 13.819
File stat : 18162.745 17849.431 18005.063 127.918
File read : 6628.013 6431.326 6557.983 89.727
File removal : 4340.580 4249.596 4307.685 41.195
Tree creation : 577.092 521.096 541.044 25.538
Tree removal : 65.604 50.556 57.142 6.285
-- finished at 07/22/2015 08:41:49 --1 Client - 4 Threds Per Client
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
-- started at 07/22/2015 09:00:15 --
mdtest-1.9.1 was launched with 4 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB Used FS: 0.4% Inodes: 291.1 Mi Used Inodes: 0.0%
4 tasks, 64000 files
SUMMARY: (of 3 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 3761.743 3676.986 3722.687 34.920
File stat : 10465.463 9879.783 10222.287 249.205
File read : 6832.391 6672.419 6764.390 67.472
File removal : 4540.434 4496.427 4520.446 18.191
Tree creation : 975.646 104.080 682.572 409.068
Tree removal : 139.647 138.353 139.054 0.534
-- finished at 07/22/2015 09:03:06 --