Benchmarking: IOZone (CERN)

Prerequisites

XFS & lsscsi need installing:

yum install xfsprogs.x86_64 lsscsi

As does IOZONE:

yum install iozone

Preparing the data filesystems

All available block devices in the attached disk enclosure will be used for this test. None of the drives in the front-end server will be used.

Get the list of available block devices

Use lsscsi to identify the drives to be tested (in this case, /dev/sdc to /dev/sdr):

[root@nodea ~]# lsscsi
[0:0:0:0]    enclosu Promise  J630s            060=  -       
[0:0:1:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdc 
[0:0:2:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdd 
[0:0:3:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sde 
[0:0:4:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdf 
[0:0:5:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdg 
[0:0:6:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdh 
[0:0:7:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdi 
[0:0:8:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdj 
[0:0:9:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdk 
[0:0:10:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdl 
[0:0:11:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdm 
[0:0:12:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdn 
[0:0:13:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdo 
[0:0:14:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdp 
[0:0:15:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdq 
[0:0:16:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdr 
[1:0:0:0]    disk    ATA      Hitachi HUA72302 MK7O  /dev/sda 
[2:0:0:0]    disk    ATA      Hitachi HUA72302 MK7O  /dev/sdb

Create an XFS filesystem on each drive

The basic command for this is:

mkfs.xfs -l version=2 -i size=1024 -n size=16384 -d su=64k,sw=1 <block device>

So can be scripted as follows:

for i in {c..r};
do
	mkfs.xfs -l version=2 -i size=1024 -n size=16384 -d su=64k,sw=1 /dev/sd${i} &
done

Create mount points

Create mount points in /srv/castor/XY where XY ranges from 1 to the number of available block devices. For example, it will be 16 on a 16-bay disk enclosure.
This can be scripted as follows:

mkdir /srv/castor
for i in {1..16};
do
	mkdir /srv/castor/${i}
done

Mount filesystems to mount points

Mount these filesystems using the following options: logbufs=8,logbsize=256k,noatime,swallow,inode64
This can be scripted to add mount point to /etc/fstab as follows:

n=1

for i in {c..r}
do
	echo "/dev/sd${i}	/srv/castor/${n}	xfs logbufs=8,logbsize=256k,noatime,swalloc,inode64" >> /etc/fstab
	((n++))
done

Benchmarking a single disk

Cern ask for the first and last disks in the storage array to be tested and for each test to be run 4 times and an average be taken.

This benchmark can be scripted as follows:

for j in `seq 1 4`;
do
	for i in {1,16}; # << Change 16 to the total number of drives being tested
	do
		/opt/iozone/bin/iozone -i 0 -i 1 -i 2 -R -I -l 1 -u 1 -r 1m -s 4g -F /srv/castor/"$i"/iozone.dat | tee -a iozone-1m-4g-single-disk-"$i"-loop-"$j".txt
	done
done

And then averages calculated:

echo " "
echo "*** Results in KB/s ***"
echo " "
echo "1st Disk"
echo "=========="
echo " "
grep '  Read'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$4; avg=sum/NR; printf "Average Read: %7.2f\n", avg}' | tail -n 1 
grep 'Initial write'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Initial write: %7.2f\n", avg}' | tail -n 1 
grep 'Random read'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random read: %7.2f\n", avg}'  | tail -n 1 
grep 'Random write'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random write: %7.2f\n", avg}'  | tail -n 1 
echo " "
echo "16th Disk"
echo "=========="
echo " "
grep '  Read'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$4; avg=sum/NR; printf "Average Read: %7.2f\n", avg}' | tail -n 1
grep 'Initial write'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Initial write: %7.2f\n", avg}' | tail -n 1
grep 'Random read'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random read: %7.2f\n", avg}'  | tail -n 1
grep 'Random write'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random write: %7.2f\n", avg}'  | tail -n 1

Benchmarking all disks

Cern also ask for all disks in the storage array to be tested together and, again, for each test to be run 4 times and an average be taken.

This can be scripted as follows (in this case, with 24 drives being tested):

for i in `seq 1 4`
do
	/opt/iozone/bin/iozone -R -I -l 24 -u 24 -r 1m -s 4g -F /srv/castor/{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24}/iozone.dat | tee -a iozone-1m-4g-24fs-loop-"$i".txt
done

And then averages calculated:

echo " "
echo "*** Results in KB/s ***"
echo " "
grep '  Read'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$4; avg=sum/NR; printf "Average Read: %7.2f\n", avg}' | tail -n 1 
grep 'Initial write'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Initial write: %7.2f\n", avg}' | tail -n 1 
grep 'Random read'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random read: %7.2f\n", avg}'  | tail -n 1 
grep 'Random write'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random write: %7.2f\n", avg}'  | tail -n 1

Single Disk Results

				1st Disk				Last Disk
Server Model	Disk Model	Disk Type	Spindle Speed	Read	Initial Write	Rand. Read	Rand. Write	Read	Initial Write	Rand. Read	Rand. Write	Tender
6027PR-HTR	ST6000NM0024	6Gb/s SATA	7,200	220139	212904	100416	147607	220766	220254	99381	147636	CERN IT-4035