Difference between revisions of "IEEL 2.2 - MDTEST for Unit.tv"

From Define Wiki
Jump to navigation Jump to search
 
(2 intermediate revisions by the same user not shown)
Line 13: Line 13:
 
make
 
make
 
</syntaxhighlight>
 
</syntaxhighlight>
−
 
  
 
== Initial Testing & Observations: ==
 
== Initial Testing & Observations: ==
Line 26: Line 25:
 
'''Optimised Run:'''
 
'''Optimised Run:'''
 
<syntaxhighlight>
 
<syntaxhighlight>
−
/usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
+
[root@mds1 mdtest]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
 +
 
 +
-- started at 07/24/2015 03:59:48 --
 +
 
 +
mdtest-1.9.3 was launched with 16 total task(s) on 2 node(s)
 +
Command line used: ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
 +
Path: /mnt/lustrefs-ib/mdtest
 +
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
 +
 
 +
16 tasks, 100800 files/directories
 +
 
 +
SUMMARY: (of 30 iterations)
 +
  Operation                      Max            Min          Mean        Std Dev
 +
  ---------                      ---            ---          ----        -------
 +
  Directory creation:      8771.583      8257.319      8498.145        142.601
 +
  Directory stat    :      51249.434      42772.707      46890.859      1841.967
 +
  Directory removal :      8743.787      8206.951      8402.380        133.065
 +
  File creation    :      7381.270      6614.801      7078.306        143.470
 +
  File stat        :      31104.592      26010.224      28677.905      1262.069
 +
  File read        :      11800.048      11575.249      11695.937        65.612
 +
  File removal      :      9023.598      8360.099      8595.353        144.175
 +
  Tree creation    :      8087.447      4717.328      6582.154        625.018
 +
  Tree removal      :      3354.847      1011.802      2532.170        817.395
 +
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
−
-i Total Number of Test Itterations
+
'''Observations'''
−
-b Branching Factor of hierarchical directory structure
+
 
−
-I Number of Items per directory tree
+
Looking at the numbers file creation seems to be bound by the number of nodes whereas stat seems to scale per process. Tree creation and removal seem to improve as you add clients but performance drops off the more processes you run per client, so a higher number of clients and lower number of processes per client would be better – like here: https://jira.hpdd.intel.com/browse/LU-5608 – 32 & 64
−
-z Depth of hierarchical directory structure
 
−
-d Test directory
 
  
−
'''Output'''
+
== Other Results ==
 +
 
 +
2 Clients - 8 Threads Per Client
  
 
<syntaxhighlight>
 
<syntaxhighlight>
−
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
+
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
  
−
-- started at 07/22/2015 09:34:24 --
+
-- started at 07/22/2015 08:42:32 --
  
 
mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
 
mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
−
Command line used: ./mdtest -i 20 -b 2 -I 80 -z 5 -d /mnt/lustrefs-ib/mdtest/1
+
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
 +
Path: /mnt/lustrefs-ib/mdtest
 +
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
 +
 
 +
16 tasks, 256000 files
 +
 
 +
SUMMARY: (of 3 iterations)
 +
  Operation                      Max            Min          Mean        Std Dev
 +
  ---------                      ---            ---          ----        -------
 +
  File creation    :      6226.078      6196.187      6211.441        12.211
 +
  File stat        :      34523.900      34191.504      34347.555        136.457
 +
  File read        :      11451.576      11281.008      11389.401        76.921
 +
  File removal      :      5026.970      4975.436      5002.181        21.084
 +
  Tree creation    :        864.448        127.405        537.898        306.695
 +
  Tree removal      :          2.007          1.935          1.977          0.031
 +
 
 +
-- finished at 07/22/2015 08:49:11 --
 +
</syntaxhighlight>
 +
 
 +
2 Clients - 4 Threads Per Client
 +
 
 +
<syntaxhighlight>
 +
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 8 -npernode 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
 +
 
 +
-- started at 07/22/2015 08:50:12 --
 +
 
 +
mdtest-1.9.1 was launched with 8 total task(s) on 2 node(s)
 +
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
 +
Path: /mnt/lustrefs-ib/mdtest
 +
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
 +
 
 +
8 tasks, 128000 files
 +
 
 +
SUMMARY: (of 3 iterations)
 +
  Operation                      Max            Min          Mean        Std Dev
 +
  ---------                      ---            ---          ----        -------
 +
  File creation    :      6529.571      6449.035      6486.553        33.108
 +
  File stat        :      17431.529      17044.547      17276.888        167.265
 +
  File read        :      11457.978      11408.014      11435.052        20.604
 +
  File removal      :      4216.588      4173.080      4201.648        20.207
 +
  Tree creation    :      1848.525        350.753      1280.509        662.808
 +
  Tree removal      :        198.293        173.577        189.445        11.245
 +
 
 +
-- finished at 07/22/2015 08:54:10 --
 +
</syntaxhighlight>
 +
 
 +
1 Client - 8 Threads Per Client
 +
<syntaxhighlight>
 +
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
 +
 
 +
-- started at 07/22/2015 08:36:41 --
 +
 
 +
mdtest-1.9.1 was launched with 8 total task(s) on 1 node(s)
 +
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
 +
Path: /mnt/lustrefs-ib/mdtest
 +
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
 +
 
 +
8 tasks, 128000 files
 +
 
 +
SUMMARY: (of 3 iterations)
 +
  Operation                      Max            Min          Mean        Std Dev
 +
  ---------                      ---            ---          ----        -------
 +
  File creation    :      3536.366      3504.754      3524.054        13.819
 +
  File stat        :      18162.745      17849.431      18005.063        127.918
 +
  File read        :      6628.013      6431.326      6557.983        89.727
 +
  File removal      :      4340.580      4249.596      4307.685        41.195
 +
  Tree creation    :        577.092        521.096        541.044        25.538
 +
  Tree removal      :        65.604        50.556        57.142          6.285
 +
 
 +
-- finished at 07/22/2015 08:41:49 --
 +
</syntaxhighlight>
 +
 
 +
1 Client - 4 Threds Per Client
 +
 
 +
<syntaxhighlight>
 +
[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
 +
 
 +
-- started at 07/22/2015 09:00:15 --
 +
 
 +
mdtest-1.9.1 was launched with 4 total task(s) on 1 node(s)
 +
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
 
Path: /mnt/lustrefs-ib/mdtest
 
Path: /mnt/lustrefs-ib/mdtest
 
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
 
FS: 290.9 TiB  Used FS: 0.4%  Inodes: 291.1 Mi  Used Inodes: 0.0%
  
−
16 tasks, 80640 files/directories
+
4 tasks, 64000 files
  
−
SUMMARY: (of 20 iterations)
+
SUMMARY: (of 3 iterations)
 
   Operation                      Max            Min          Mean        Std Dev
 
   Operation                      Max            Min          Mean        Std Dev
 
   ---------                      ---            ---          ----        -------
 
   ---------                      ---            ---          ----        -------
−
  Directory creation:      8763.188      8232.435      8376.081        117.579
+
   File creation    :      3761.743       3676.986       3722.687        34.920
−
  Directory stat    :      51421.884      44970.310      47884.412      1761.017
+
   File stat        :      10465.463      9879.783     10222.287       249.205
−
  Directory removal :      8541.242      7992.884      8339.238        135.393
+
   File read        :       6832.391      6672.419      6764.390         67.472
−
   File creation    :      7226.927       6714.372       7041.462        152.999
+
   File removal      :      4540.434       4496.427       4520.446        18.191
−
   File stat        :      28105.169      26091.429     27392.123       537.488
+
   Tree creation    :       975.646        104.080        682.572       409.068
−
   File read        :     11582.120      11297.195      11443.001         77.403
+
   Tree removal      :       139.647        138.353        139.054          0.534
−
   File removal      :      8581.734       8033.429       8385.055        125.556
 
−
   Tree creation    :       7627.989      5464.496      6619.557       367.386
 
−
   Tree removal      :       3626.299      2855.457      3308.709        131.297
 
  
−
-- finished at 07/22/2015 09:51:46 --
+
-- finished at 07/22/2015 09:03:06 --
 
</syntaxhighlight>
 
</syntaxhighlight>

Latest revision as of 08:35, 24 July 2015

Download MDTEST from Sourceforge: [1]

Make a directory called mdtest - copy and Untar MDTEST into that directory. 

mkdir /root/mdtest
tar xvf mdtest-1.9.3.tgz

Edit Makefile to point CC.Linux to current MPI and then make - My example I modified to: 

CC.Linux = /usr/mpi/gcc/openmpi-1.8.4/bin/mpicc

make

Initial Testing & Observations:

Basic run:' 

./mdtest -n 1024 -d /mnt/lustrefs-ib/mdtest

-n Total amount of files to run per process (Standard run is 1 process)

Optimised Run: 

[root@mds1 mdtest]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun --hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2

-- started at 07/24/2015 03:59:48 --

mdtest-1.9.3 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -i 30 -b 2 -I 100 -z 5 -d /mnt/lustrefs-ib/mdtest/2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

16 tasks, 100800 files/directories

SUMMARY: (of 30 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:       8771.583       8257.319       8498.145        142.601
   Directory stat    :      51249.434      42772.707      46890.859       1841.967
   Directory removal :       8743.787       8206.951       8402.380        133.065
   File creation     :       7381.270       6614.801       7078.306        143.470
   File stat         :      31104.592      26010.224      28677.905       1262.069
   File read         :      11800.048      11575.249      11695.937         65.612
   File removal      :       9023.598       8360.099       8595.353        144.175
   Tree creation     :       8087.447       4717.328       6582.154        625.018
   Tree removal      :       3354.847       1011.802       2532.170        817.395

Observations

Looking at the numbers file creation seems to be bound by the number of nodes whereas stat seems to scale per process. Tree creation and removal seem to improve as you add clients but performance drops off the more processes you run per client, so a higher number of clients and lower number of processes per client would be better – like here: https://jira.hpdd.intel.com/browse/LU-5608 – 32 & 64

Other Results

2 Clients - 8 Threads Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 16 -npernode 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 08:42:32 --

mdtest-1.9.1 was launched with 16 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

16 tasks, 256000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       6226.078       6196.187       6211.441         12.211
   File stat         :      34523.900      34191.504      34347.555        136.457
   File read         :      11451.576      11281.008      11389.401         76.921
   File removal      :       5026.970       4975.436       5002.181         21.084
   Tree creation     :        864.448        127.405        537.898        306.695
   Tree removal      :          2.007          1.935          1.977          0.031

-- finished at 07/22/2015 08:49:11 --

2 Clients - 4 Threads Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -hostfile hostfile -np 8 -npernode 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 08:50:12 --

mdtest-1.9.1 was launched with 8 total task(s) on 2 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

8 tasks, 128000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       6529.571       6449.035       6486.553         33.108
   File stat         :      17431.529      17044.547      17276.888        167.265
   File read         :      11457.978      11408.014      11435.052         20.604
   File removal      :       4216.588       4173.080       4201.648         20.207
   Tree creation     :       1848.525        350.753       1280.509        662.808
   Tree removal      :        198.293        173.577        189.445         11.245

-- finished at 07/22/2015 08:54:10 --

1 Client - 8 Threads Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 8 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 08:36:41 --

mdtest-1.9.1 was launched with 8 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

8 tasks, 128000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       3536.366       3504.754       3524.054         13.819
   File stat         :      18162.745      17849.431      18005.063        127.918
   File read         :       6628.013       6431.326       6557.983         89.727
   File removal      :       4340.580       4249.596       4307.685         41.195
   Tree creation     :        577.092        521.096        541.044         25.538
   Tree removal      :         65.604         50.556         57.142          6.285

-- finished at 07/22/2015 08:41:49 --

1 Client - 4 Threds Per Client 

[root@mds1 mdtest-1.9.1]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 4 --allow-run-as-root ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2

-- started at 07/22/2015 09:00:15 --

mdtest-1.9.1 was launched with 4 total task(s) on 1 node(s)
Command line used: ./mdtest -n 16000 -i 3 -p 10 -d /mnt/lustrefs-ib/mdtest/ -F -u -N 2
Path: /mnt/lustrefs-ib/mdtest
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%

4 tasks, 64000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :       3761.743       3676.986       3722.687         34.920
   File stat         :      10465.463       9879.783      10222.287        249.205
   File read         :       6832.391       6672.419       6764.390         67.472
   File removal      :       4540.434       4496.427       4520.446         18.191
   Tree creation     :        975.646        104.080        682.572        409.068
   Tree removal      :        139.647        138.353        139.054          0.534

-- finished at 07/22/2015 09:03:06 --
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=IEEL_2.2_-_MDTEST_for_Unit.tv&oldid=9061"