Difference between revisions of "IEEL 2.2 - MDTEST for Unit.tv"

From Define Wiki
Jump to navigation Jump to search
 
(One intermediate revision by the same user not shown)
Line 13: Line 13:
 
make
 
make
 
</syntaxhighlight>
 
</syntaxhighlight>
−
 
  
 
== Initial Testing & Observations: ==
 
== Initial Testing & Observations: ==
Line 26: Line 25:
 
'''Optimised Run:'''
 
'''Optimised Run:'''
 
<syntaxhighlight>
 
<syntaxhighlight>
−
/usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
+
[root@mds1 mdtest]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
−
</syntaxhighlight>
 
−
 
 
−
-i Total Number of Test Itterations
 
−
-b Branching Factor of hierarchical directory structure
 
−
-I Number of Items per directory tree
 
−
-z Depth of hierarchical directory structure
 
−
-d Test directory
 
−
 
 
−
'''Output'''
 
−
 
 
−
<syntaxhighlight>
 
−
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
 
  
−
-- started at 07/22/2015 09:34:24 --
+
-- started at 07/24/2015 03:59:48 --
  
−
mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
+
mdtest-1.9.3 was launched with 16 total task(s) on 2 node(s)
−
Command line used: ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
+
Command line used: ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
 
Path: /mnt/lustrefs-ib/mdtest
 
Path: /mnt/lustrefs-ib/mdtest
 
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
 
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
  
−
16 tasks, 80640 files/directories
+
16 tasks, 100800 files/directories
  
−
SUMMARY: (of 20 iterations)
+
SUMMARY: (of 30 iterations)
 
   Operation                      Max            Min          Mean        Std Dev
 
   Operation                      Max            Min          Mean        Std Dev
 
   ---------                      ---            ---          ----        -------
 
   ---------                      ---            ---          ----        -------
−
   Directory creation:      8763.188       8232.435       8376.081       117.579
+
   Directory creation:      8771.583       8257.319       8498.145       142.601
−
   Directory stat    :      51421.884     44970.310     47884.412       1761.017
+
   Directory stat    :      51249.434     42772.707     46890.859       1841.967
−
   Directory removal :      8541.242       7992.884       8339.238       135.393
+
   Directory removal :      8743.787       8206.951       8402.380       133.065
−
   File creation    :      7226.927       6714.372       7041.462       152.999
+
   File creation    :      7381.270       6614.801       7078.306       143.470
−
   File stat        :      28105.169     26091.429     27392.123        537.488
+
   File stat        :      31104.592     26010.224     28677.905      1262.069
−
   File read        :      11582.120     11297.195     11443.001         77.403
+
   File read        :      11800.048     11575.249     11695.937         65.612
−
   File removal      :      8581.734       8033.429       8385.055       125.556
+
   File removal      :      9023.598       8360.099       8595.353       144.175
−
   Tree creation    :      7627.989       5464.496       6619.557       367.386
+
   Tree creation    :      8087.447       4717.328       6582.154       625.018
−
   Tree removal      :      3626.299       2855.457       3308.709       131.297
+
   Tree removal      :      3354.847       1011.802       2532.170       817.395
  
−
-- finished at 07/22/2015 09:51:46 --
 
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
'''Observations'''
 +
 +
Looking at the numbers file creation seems to be bound by the number of nodes whereas stat seems to scale per process. Tree creation and removal seem to improve as you add clients but performance drops off the more processes you run per client, so a higher number of clients and lower number of processes per client would be better – like here: https://jira.hpdd.intel.com/browse/LU-5608 – 32 & 64
  
 
== Other Results ==
 
== Other Results ==

Latest revision as of 08:35, 24 July 2015

Download MDTEST from Sourceforge: [1]

Make a directory called mdtest - copy and Untar MDTEST into that directory. 

mkdir /root/mdtest
tar xvf mdtest-1.9.3.tgz

Edit Makefile to point CC.Linux to current MPI and then make - My example I modified to: 

CC.Linux = /usr/mpi/gcc/openmpi-1.8.4/bin/mpicc

make

Initial Testing & Observations:

Basic run:' 

./mdtest -n 1024 -d /mnt/lustrefs-ib/mdtest

-n Total amount of files to run per process (Standard run is 1 process)

Optimised Run: 

[root@mds1 mdtest]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2

-- started at 07/24/2015 03:59:48 --

mdtest-1.9.3 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

16 tasks, 100800 files/directories

SUMMARY: (of 30 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:       8771.583       8257.319       8498.145        142.601
   Directory stat    :      51249.434      42772.707      46890.859       1841.967
   Directory removal :       8743.787       8206.951       8402.380        133.065
   File creation     :       7381.270       6614.801       7078.306        143.470
   File stat         :      31104.592      26010.224      28677.905       1262.069
   File read         :      11800.048      11575.249      11695.937         65.612
   File removal      :       9023.598       8360.099       8595.353        144.175
   Tree creation     :       8087.447       4717.328       6582.154        625.018
   Tree removal      :       3354.847       1011.802       2532.170        817.395

Observations

Looking at the numbers file creation seems to be bound by the number of nodes whereas stat seems to scale per process. Tree creation and removal seem to improve as you add clients but performance drops off the more processes you run per client, so a higher number of clients and lower number of processes per client would be better – like here: https://jira.hpdd.intel.com/browse/LU-5608 – 32 & 64

Other Results

2 Clients - 8 Threads Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 08:42:32 --

mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

16 tasks, 256000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       6226.078       6196.187       6211.441         12.211
   File stat         :      34523.900      34191.504      34347.555        136.457
   File read         :      11451.576      11281.008      11389.401         76.921
   File removal      :       5026.970       4975.436       5002.181         21.084
   Tree creation     :        864.448        127.405        537.898        306.695
   Tree removal      :          2.007          1.935          1.977          0.031

-- finished at 07/22/2015 08:49:11 --

2 Clients - 4 Threads Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 8 -npernode 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 08:50:12 --

mdtest-1.9.1 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

8 tasks, 128000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       6529.571       6449.035       6486.553         33.108
   File stat         :      17431.529      17044.547      17276.888        167.265
   File read         :      11457.978      11408.014      11435.052         20.604
   File removal      :       4216.588       4173.080       4201.648         20.207
   Tree creation     :       1848.525        350.753       1280.509        662.808
   Tree removal      :        198.293        173.577        189.445         11.245

-- finished at 07/22/2015 08:54:10 --

1 Client - 8 Threads Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 08:36:41 --

mdtest-1.9.1 was launched with 8 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

8 tasks, 128000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       3536.366       3504.754       3524.054         13.819
   File stat         :      18162.745      17849.431      18005.063        127.918
   File read         :       6628.013       6431.326       6557.983         89.727
   File removal      :       4340.580       4249.596       4307.685         41.195
   Tree creation     :        577.092        521.096        541.044         25.538
   Tree removal      :         65.604         50.556         57.142          6.285

-- finished at 07/22/2015 08:41:49 --

1 Client - 4 Threds Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 09:00:15 --

mdtest-1.9.1 was launched with 4 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

4 tasks, 64000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       3761.743       3676.986       3722.687         34.920
   File stat         :      10465.463       9879.783      10222.287        249.205
   File read         :       6832.391       6672.419       6764.390         67.472
   File removal      :       4540.434       4496.427       4520.446         18.191
   Tree creation     :        975.646        104.080        682.572        409.068
   Tree removal      :        139.647        138.353        139.054          0.534

-- finished at 07/22/2015 09:03:06 --
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=IEEL_2.2_-_MDTEST_for_Unit.tv&oldid=9061"