Benchmarking: IOZone (CERN)

Prerequisites

XFS & lsscsi need installing:

yum install xfsprogs.x86_64 lsscsi

As does IOZONE:

yum install iozone

Preparing the data filesystems

All available block devices in the attached disk enclosure will be used for this test. None of the drives in the front-end server will be used.

Get the list of available block devices

Use lsscsi to identify the drives to be tested (in this case, /dev/sdc to /dev/sdr):

[root@nodea ~]# lsscsi
[0:0:0:0]    enclosu Promise  J630s            060=  -       
[0:0:1:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdc 
[0:0:2:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdd 
[0:0:3:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sde 
[0:0:4:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdf 
[0:0:5:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdg 
[0:0:6:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdh 
[0:0:7:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdi 
[0:0:8:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdj 
[0:0:9:0]    disk    ATA      ST6000NM0024-1HT SN02  /dev/sdk 
[0:0:10:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdl 
[0:0:11:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdm 
[0:0:12:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdn 
[0:0:13:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdo 
[0:0:14:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdp 
[0:0:15:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdq 
[0:0:16:0]   disk    ATA      ST6000NM0024-1HT SN02  /dev/sdr 
[1:0:0:0]    disk    ATA      Hitachi HUA72302 MK7O  /dev/sda 
[2:0:0:0]    disk    ATA      Hitachi HUA72302 MK7O  /dev/sdb

Create an XFS filesystem on each drive

The basic command for this is:

mkfs.xfs -l version=2 -i size=1024 -n size=16384 -d su=64k,sw=1 <block device>

So can be scripted as follows:

for i in {c..r};
do
	mkfs.xfs -l version=2 -i size=1024 -n size=16384 -d su=64k,sw=1 /dev/sd${i} &
done

Create mount points

Create mount points in /srv/castor/XY where XY ranges from 1 to the number of available block devices. For example, it will be 16 on a 16-bay disk enclosure.
This can be scripted as follows:

mkdir /srv/castor
for i in {1..16};
do
	mkdir /srv/castor/${i}
done

Mount filesystems to mount points

Mount these filesystems using the following options: logbufs=8,logbsize=256k,noatime,swallow,inode64
This can be scripted to add mount point to /etc/fstab as follows:

n=1

for i in {c..r}
do
	echo "/dev/sd${i}	/srv/castor/${n}	xfs logbufs=8,logbsize=256k,noatime,swalloc,inode64" >> /etc/fstab
	((n++))
done

Benchmarking a single disk

Cern ask for the first and last disks in the storage array to be tested and for each test to be run 4 times and an average be taken.

This benchmark can be scripted as follows:

for j in `seq 1 4`;
do
	for i in {1,16}; # << Change 16 to the total number of drives being tested
	do
		/opt/iozone/bin/iozone -i 0 -i 1 -i 2 -R -I -l 1 -u 1 -r 1m -s 4g -F /srv/castor/"$i"/iozone.dat | tee -a iozone-1m-4g-single-disk-"$i"-loop-"$j".txt
	done
done

And then averages calculated:

echo " "
echo "*** Results in KB/s ***"
echo " "
echo "1st Disk"
echo "=========="
echo " "
grep '  Read'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$4; avg=sum/NR; printf "Average Read: %7.2f\n", avg}' | tail -n 1 
grep 'Initial write'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Initial write: %7.2f\n", avg}' | tail -n 1 
grep 'Random read'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random read: %7.2f\n", avg}'  | tail -n 1 
grep 'Random write'  iozone-1m-4g-single-disk-1-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random write: %7.2f\n", avg}'  | tail -n 1 
echo " "
echo "16th Disk"
echo "=========="
echo " "
grep '  Read'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$4; avg=sum/NR; printf "Average Read: %7.2f\n", avg}' | tail -n 1
grep 'Initial write'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Initial write: %7.2f\n", avg}' | tail -n 1
grep 'Random read'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random read: %7.2f\n", avg}'  | tail -n 1
grep 'Random write'  iozone-1m-4g-single-disk-16-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random write: %7.2f\n", avg}'  | tail -n 1

Benchmarking all disks

Cern also ask for all disks in the storage array to be tested together and, again, for each test to be run 4 times and an average be taken.

This can be scripted as follows (in this case, with 24 drives being tested):

for i in `seq 1 4`
do
	/opt/iozone/bin/iozone -R -I -l 24 -u 24 -r 1m -s 4g -F /srv/castor/{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24}/iozone.dat | tee -a iozone-1m-4g-24fs-loop-"$i".txt
done

For a different total number of drives, adjust the -l 24 and -u 24 values accordingly.

And then averages calculated:

echo " "
echo "*** Results in KB/s ***"
echo " "
grep '  Read'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$4; avg=sum/NR; printf "Average Read: %7.2f\n", avg}' | tail -n 1 
grep 'Initial write'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Initial write: %7.2f\n", avg}' | tail -n 1 
grep 'Random read'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random read: %7.2f\n", avg}'  | tail -n 1 
grep 'Random write'  iozone-1m-4g-24fs-loop* | awk '{sum=sum+$5; avg=sum/NR; printf "Average Random write: %7.2f\n", avg}'  | tail -n 1

Single Disk Results

System Config					1st Disk (KB/s)				Last Disk (KB/s)
Disk Enclosure	Controller	Disk Model	Disk Type	Spindle Speed	Read	Initial Write	Rand. Read	Rand. Write	Read	Initial Write	Rand. Read	Rand. Write	Tender
Promise VTRAK J630S	LSI 9207-8e	ST6000NM0024	6Gb/s SATA	7,200	220139	212904	100416	147607	220766	220254	99381	147636	IT-4032
Promise VTRAK J630S	LSI 9207-8e	HUS726060ALA640	6Gb/s SATA	7,200	184203	184153	87583	90935	183946	183679	87005	89375	IT-4032

Multiple Disk Results

System Config						All Disks Combined (KB/s)
Disk Enclosure	Controller	Disk Model	Disk Type	Spindle Speed	Qty	Read	Initial Write	Rand. Read	Rand. Write	Tender
Promise VTRAK J630S	LSI 9207-8e	ST6000NM0024	6Gb/s SATA	7,200	16	1,872,830	1,699,285	1,321,172	1,743,663	IT-4032
Promise VTRAK J630S	LSI 9207-8e	HUS726060ALA640 (FW 1EN)	6Gb/s SATA	7,200	16	1,894,552	1,598,613	1,301,298	1,370,605	IT-4032
Promise VTRAK J630S	LSI 9207-8e	HUS726060ALA640 (FW 1EQ)	6Gb/s SATA	7,200	16	2,082,375	1,755,122	1,332,840	1,393,900	IT-4032
Promise VTRAK J830S	LSI 9207-8e	WD4000F9YZ (FW AA2)	6Gb/s SATA	7,200	24	1,918,817	1,659,493	1,282,772	1,390,060	IT-4032