IEEL 2.2 - IOR for Unit.tv

Download IOR from Sourceforge: <link>

tar xvf IOR-2.10.3.tgz
cd IOR/src/C/
vi Makefile

Amend the Makefile to point to your current mpicc and make:

CC.Linux = /usr/mpi/gcc/openmpi-1.8.4/bin/mpicc
#save and exit
make mpiio

Running and Optimising on Lustre

Tips

Its is best for Block size to always equal transfer size (-t & -b) - Lustre Filesystem Block size is 1M so -t 1m -b 1m recommended.

Ensure (-s) segment size is always big enough to exceed the total amount of memory used in the run - (segment size x block size = total file size written and read) i.e. -b 1m -s 1024 wites a file 1 Gigabyte total.

cd /root/IOR/src/C/
#
/usr/mpi/gcc/openmpi-1.8.4/bin/mpirun ./IOR -vv -k -wwr -C -F -i4 -t 1m -b 1m -s 20480 -o /mnt/lustre-ib/IORTEST1-ib-mpi1

This will write and read a file 160GiB total - enough to exceed memory to avoid any caching. If total file size is less than memory read performance will be ludicrously high.

Results

[root@mds1 C]# /usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 8 --allow-run-as-root ./

IOR-2.10.3: MPI Coordinated Test of Parallel I/O

Run began: Wed Jul 22 12:01:16 2015
Command line used: ./IOR -vv -k -wwr -C -F -i4 -t 1m -b 1m -s 20480 -o /mnt/lustr
Machine: Linux mds1.utv.boston.co.uk 2.6.32-504.12.2.el6_lustre.x86_64 #1 SMP Wed
Using synchronized MPI timer
Start time skew across all tasks: 0.00 sec
Path: /mnt/lustrefs-ib
FS: 290.9 TiB   Used FS: 0.4%   Inodes: 291.1 Mi   Used Inodes: 0.0%
Participating tasks: 8
Using reorderTasks '-C' (expecting block, not cyclic, task assignment)
task 0 on mds1.utv.boston.co.uk
task 1 on mds1.utv.boston.co.uk
task 2 on mds1.utv.boston.co.uk
task 3 on mds1.utv.boston.co.uk
task 4 on mds1.utv.boston.co.uk
task 5 on mds1.utv.boston.co.uk
task 6 on mds1.utv.boston.co.uk
task 7 on mds1.utv.boston.co.uk

Summary:
        api                = POSIX
        test filename      = /mnt/lustrefs-ib/IORTEST1-ib-mpi4
        access             = file-per-process
        pattern            = strided (20480 segments)
        ordering in a file = sequential offsets
        ordering inter file=constant task offsets = 1
        clients            = 8 (8 per node)
        repetitions        = 4
        xfersize           = 1 MiB
        blocksize          = 1 MiB
        aggregate filesize = 160 GiB

Using Time Stamp 1437580876 (0x55afbe4c) for Data Signature
Commencing write performance test.
Wed Jul 22 12:01:16 2015

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s) total(s
------    ---------  ---------- ---------  --------   --------   --------  ------
write     2737.31    1024.00    1024.00    0.018548   59.84      14.41      59.85
[RANK 000] open for reading file /mnt/lustrefs-ib/IORTEST1-ib-mpi4.00000000 XXCEL
Commencing read performance test.
Wed Jul 22 12:02:16 2015

read      3151.92    1024.00    1024.00    0.001005   51.98      5.18       51.98
Using Time Stamp 1437580988 (0x55afbebc) for Data Signature
Commencing write performance test.
Wed Jul 22 12:03:08 2015

write     2728.59    1024.00    1024.00    0.002227   60.04      11.94      60.05
[RANK 000] open for reading file /mnt/lustrefs-ib/IORTEST1-ib-mpi4.00000000 XXCEL
Commencing read performance test.
Wed Jul 22 12:04:08 2015

read      3107.03    1024.00    1024.00    0.001006   52.73      5.03       52.73      1    XXCEL
Using Time Stamp 1437581101 (0x55afbf2d) for Data Signature
Commencing write performance test.
Wed Jul 22 12:05:01 2015

write     2683.20    1024.00    1024.00    0.001974   61.06      12.04      61.06      2    XXCEL
[RANK 000] open for reading file /mnt/lustrefs-ib/IORTEST1-ib-mpi4.00000000 XXCEL
Commencing read performance test.
Wed Jul 22 12:06:02 2015

read      3071.63    1024.00    1024.00    0.001003   53.34      4.50       53.34      2    XXCEL
Using Time Stamp 1437581215 (0x55afbf9f) for Data Signature
Commencing write performance test.
Wed Jul 22 12:06:55 2015

write     2711.78    1024.00    1024.00    0.002081   60.42      13.31      60.42      3    XXCEL
[RANK 000] open for reading file /mnt/lustrefs-ib/IORTEST1-ib-mpi4.00000000 XXCEL
Commencing read performance test.
Wed Jul 22 12:07:56 2015

read      3061.87    1024.00    1024.00    0.000905   53.51      6.92       53.51      3    XXCEL
Operation  Max (MiB)  Min (MiB)  Mean (MiB)   Std Dev  Max (OPs)  Min (OPs)  Mean (OPs)   Std Dev  Mean (s)  Op grep #Tasks tPN reps  fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize

---------  ---------  ---------  ----------   -------  ---------  ---------  ----------   -------  --------
write        2737.31    2683.20     2715.22     20.64       0.13       0.13        0.13      0.00  60.34479   8 8 4 1 1 1 0 0 20480 1048576 1048576 171798691840 -1 POSIX EXCEL
read         3151.92    3061.87     3098.11     35.32       0.15       0.15        0.15      0.00  52.89067   8 8 4 1 1 1 0 0 20480 1048576 1048576 171798691840 -1 POSIX EXCEL

Max Write: 2737.31 MiB/sec (2870.28 MB/sec)
Max Read:  3151.92 MiB/sec (3305.03 MB/sec)

Run finished: Wed Jul 22 12:08:49 2015

Max Write: 2737.31 MiB/sec (2870.28 MB/sec)

Max Read: 3151.92 MiB/sec (3305.03 MB/sec)