Benchmarking: FIO

CERN Fio Benchmarks

• Ref: http://it-div-procurements.web.cern.ch/it-div-procurements/IT-3821/fio/
• Setup fio config file (/root/fio/fio-bench) with content below for the benchmarks run

[global]
direct=1
bsrange=4k-4k
####################
#numjobs should be half of the number of physical cores on the sample system.
numjobs=16
###################
directory=/pool    # directory where '/dev/sda1' disk is mounted on
iodepth=24
timeout=3600
[f1]
size=10000MB
rw=randrw

Run fio benchmark (average over 3 runs)

for i in {1..3}
do
	fio --eta=always --latency-log --bandwidth-log --output=/root/fio/fio-bench.${i}.log /root/fio/fio-bench
	sleep 3
done

Check results

• Check results: grep -i iops /root/fio-bench.[1|2|3].log and sum all the iops values

grep -i iops /root/fio-bench.1.log 
fio-bench.1.log:  read : io=728220KB, bw=207137 B/s, iops=50 , runt=3600019msec
fio-bench.1.log:  write: io=727012KB, bw=206793 B/s, iops=50 , runt=3600019msec
fio-bench.1.log:  read : io=729100KB, bw=207388 B/s, iops=50 , runt=3600001msec
fio-bench.1.log:  write: io=723424KB, bw=205773 B/s, iops=50 , runt=3600001msec

grep -i iops /root/fio-bench.2.log
fio-bench.2.log:  read : io=598968KB, bw=170372 B/s, iops=41 , runt=3600006msec
fio-bench.2.log:  write: io=603348KB, bw=171618 B/s, iops=41 , runt=3600006msec
fio-bench.2.log:  read : io=601040KB, bw=170962 B/s, iops=41 , runt=3600001msec
fio-bench.2.log:  write: io=600416KB, bw=170784 B/s, iops=41 , runt=3600001msec

grep -i iops /root/fio-bench.3.log
fio-bench.3.log:  read : io=675160KB, bw=192044 B/s, iops=46 , runt=3600023msec
fio-bench.3.log:  write: io=675108KB, bw=192029 B/s, iops=46 , runt=3600023msec
fio-bench.3.log:  read : io=675532KB, bw=192149 B/s, iops=46 , runt=3600025msec
fio-bench.3.log:  write: io=676816KB, bw=192515 B/s, iops=47 , runt=3600025msec

IOPS (average):= (200 + 164 + 185)/3 = 183

• Calculate the IOPS from the log file

 grep -i iops fio-bench.1.log | sed -e 's/=/ /g' -e 's/:/ /g'  | awk '{sum=sum+$8} END {print sum}'

CERN Results

Drive Model	Number of drives	Form Factor	Drive RPM	Interface	IOPS
250GB (WD2503ABYX)	3 (raid0)	3.5"	7200 RPM	3Gb SATA	471
Intel 320 40Gb	3 (raid0)	2.5"	N/A	3Gb SATA	1832
Intel 320 40Gb	2 (raid0)	2.5"	N/A	3Gb SATA	870*
Intel 320 40Gb	1	2.5"	N/A	3Gb SATA	606
Intel 320 160Gb	2 (raid0)	2.5"	N/A	3Gb SATA	3279
Intel 320 160Gb	1	2.5"	N/A	3Gb SATA	1591
Seagate Constellation.2 1TB	1	2.5"	7200 RPM	6Gb SATA	332
Seagate ConstellationES 2TB (ST2000NM001)	3 (raid0)	3.5"	7200 RPM	6Gb SAS	766
Hitachi Utrastar 2TB (HUA723020ALA640)	3 (raid0)	3.5"	7200 RPM	6Gb SATA	559
Toshiba 2TB (MK2001TKRB)	3 (raid0)	3.5"	7200 RPM	6Gb SAS	818
OCZ Z-Drive R4 800GB	1	PCI-E	N/A	PCI-E x8 (5Gb/s)	24,440

*Below expected, please run again when drives are available.
Could be due to wear as this test was run directly after the 3 drive raid 0.

General IOPS Tests

I've found fio (http://freshmeat.net/projects/fio/) to be an excellent testing tool for disk systems. To use it, compile it (requires libaio-devel), and then run it as

fio input.fio

For a nice simple IOP test, try this:

[random]
rw=randread
size=4g
directory=/data
iodepth=32
blocksize=4k
numjobs=16
nrfiles=1
group_reporting
ioengine=sync
loops=1

This file will perform 4GB of IO into a directory named /data, using an IO depth of 32, a block size of 4k (the IOP measurement standard) with random reads as the major operation, using standard unix IO. We have 16 simultaneous jobs doing IO, each job using 1 file. It will aggregate all the information from each job and report it, and it will run once.

then to run the test:

fio file.fio

CLI FIO

# Another IOPS test:
fio -name iops -rw=read -bs=512 -runtime=180 -iodepth 4 -filename /dev/sdb -ioengine libaio -direct=1 -numjobs=16

# Write IOPS Test
fio --filename=/dev/md0 --direct=1 --rw=randwrite --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1
  
#Read IOPS Test
fio --filename=/dev/md0 --direct=1 --rw=randread --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1

FIO to generate network traffic

• Ripped from the web somewhere

Generating Constant Bandwidth on Linux using fio

It took a lot of searching for me to find a way to generate network traffic at a specific rate between two hosts, so I thought I would share the answer. It's pretty easy to test the available bandwidth between two hosts using netcat to transfer a bunch of random data as fast as the network allows. However I wanted to test resource monitoring and graphing system, which means I needed to generate network traffic at a known rate so that I could judge the resulting graphs against my expectations.

I found you can use fio, which is a generic I/O testing tool, to achieve this. fio allows specifying the transfer rate and also has a network engine. So using fio I can configure one host as the receiver and one as the sender and transfer data at a known rate. Here's what the config files look like:

Sender jobfile.ini:

[test1]
filename=receiver.dns.name/7777
rw=write
rate=750k
size=100M
ioengine=net

Receiver jobfile.ini:

[test1]
filename=localhost/7777
rw=read
size=100M
ioengine=net

Obviously you would replace "receiver.dns.name" with the DNS name of the receiving host, and adjust the size and rate parameters as you like. (It's worth noting that the fio documentation is either wrong or misleading on what the filename should be for the receiver. It claims the receiver should only specify the port, but when I tried that it failed to run. Setting the host to localhost seemed to work and the receiver started listening.) To run the test, simply run:

fio jobfile.ini

first on the receiving host, then on the sending host. fio will then transfer 100 Megabytes of data at a rate of 750KB/sec between the two hosts. And we can see from the chart that indeed a constant rate was generated:

The observed rate is a bit above the 750KB/sec specified, but what's being measured is the number of bytes being transferred through the eth0 interface. Since the data is transferred over TCP there is some overhead to the packet structure, which I believe accounts for the extra few KB/sec observed.