Lustre: Single client perf tests and optimisations Lustre client 2.5.42.1

Benchmarking for customer who required 500MB/s per single thread using iozone; iozone -i 0 -c -e -w -r 1024K -I -s 128g -+n -f iozone1.tmp

Client FDR / Lustre server based on 8x OSTs but initially we are striping to just one OST

Standard results

Fresh off installation, no optimisations allied;

iozone -i 0 -c -e -w -r 1024K -I -s 128g -+n -f iozone2.tmp 
# snip
365MB/s

Disable Checksums

[root@x10-ft-01 ~]# lctl get_param osc.*.checksums
osc.lfs-OST0000-osc-ffff88106e415c00.checksums=1
# snip
[root@x10-ft-01 ~]# lctl set_param osc.*.checksums=0
osc.lfs-OST0000-osc-ffff88106e415c00.checksums=0
# snip
[root@x10-ft-01 ~]# lctl get_param osc.*.checksums
osc.lfs-OST0000-osc-ffff88106e415c00.checksums=0
# snip

Performance: 433650 (iozone) Using dd; (664MB/s so iozone is doing something to slow things down)

Increase the max_dirty_mb size

# 32MB default / increased to 256MB
lctl get_param osc.*.max_dirty_mb
lctl set_param osc.*.max_dirty_mb=256
lctl get_param osc.*.max_dirty_mb

Didnt make much of a difference

Increase the max_rpcs_in_flight size

[root@x10-ft-01 ~]# lctl get_param osc.*.max_rpcs_in_flight
osc.lfs-OST0000-osc-ffff88106e415c00.max_rpcs_in_flight=8

[root@x10-ft-01 ~]# lctl set_param osc.*.max_rpcs_in_flight=32
osc.lfs-OST0000-osc-ffff88106e415c00.max_rpcs_in_flight=32

[root@x10-ft-01 ~]# lctl get_param osc.*.max_rpcs_in_flight
osc.lfs-OST0000-osc-ffff88106e415c00.max_rpcs_in_flight=32

Stripe count

[root@x10-ft-08 C]# mkdir /mnt/lfs/dd-stripe-count-1
[root@x10-ft-08 C]# mkdir /mnt/lfs/dd-stripe-count-0
[root@x10-ft-08 C]# lfs getstripe /mnt/lfs/dd-stripe-count-1
/mnt/lfs/dd-stripe-count-1
stripe_count:   1 stripe_size:    1048576 stripe_offset:  -1 
[root@x10-ft-08 C]# lfs getstripe /mnt/lfs/dd-stripe-count-0
/mnt/lfs/dd-stripe-count-0
stripe_count:   1 stripe_size:    1048576 stripe_offset:  -1 
[root@x10-ft-08 C]# lfs setstripe -c -1 /mnt/lfs/dd-stripe-count-0/

# check the stripe count below; -1 means all OSTs will be wrote to 
[root@x10-ft-08 C]# lfs getstripe /mnt/lfs/dd-stripe-count-1
/mnt/lfs/dd-stripe-count-1
stripe_count:   1 stripe_size:    1048576 stripe_offset:  -1 
[root@x10-ft-08 C]# lfs getstripe /mnt/lfs/dd-stripe-count-0
/mnt/lfs/dd-stripe-count-0
stripe_count:   -1 stripe_size:    1048576 stripe_offset:  -1 

[root@x10-ft-08 C]# dd if=/dev/zero of=/mnt/lfs/dd-stripe-count-0/dd.img bs=1M count=40000
40000+0 records in
40000+0 records out
41943040000 bytes (42 GB) copied, 40.8418 s, 1.0 GB/s
[root@x10-ft-08 C]# dd if=/dev/zero of=/mnt/lfs/dd-stripe-count-1/dd.img bs=1M count=40000
40000+0 records in
40000+0 records out
41943040000 bytes (42 GB) copied, 64.0542 s, 655 MB/s