Lustre: Using IOR to Benchmark

Setup: 1x MDS, 4x OSS, 14x Clients, Ethernet Connectivity

Note: Really needed to up the number of clients writing to the file system to get near the max speed for each node (~112MB/s per OSS). Running single process per node was only resulting in ~25% of the peak performance.

Best results were when 32 processes per node were used:

[viglen@atass07 bin]$ mpirun -np 56 -hostlist "atass18:4 atass17:4 atass16:4 atass15:4 atass14:4 atass13:4 atass12:4 atass11:4 atass10:4 atass09:4 atass08:4 atass07:4 atass06:4 atass05:4" ./ior-pmpi -a POSIX -F -k -w -t1m -b40m -o /mnt/lustre/iorfile
IOR-2.10.1: MPI Coordinated Test of Parallel I/O

Run began: Tue Aug 30 12:42:02 2011
Command line used: ./ior-pmpi -a POSIX -F -k -w -t1m -b40m -o /mnt/lustre/iorfile
Machine: Linux atass18

Summary:
	api                = POSIX
	test filename      = /mnt/lustre/iorfile
	access             = file-per-process
	ordering           = sequential offsets
	clients            = 56 (4 per node)
	repetitions        = 1
	xfersize           = 1 MiB
	blocksize          = 40 MiB
	aggregate filesize = 2.19 GiB

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   iter
------    ---------  ---------- ---------  --------   --------   --------   ----
write     308.33     40960      1024.00    0.037451   7.25       6.46       0   

Max Write: 308.33 MiB/sec (323.31 MB/sec)

• Scores of 400MB/s+ required np=448 / ppn=32. To be investigated further.