Difference between revisions of "Benchmarking: FIO"

From Define Wiki
Jump to navigation Jump to search
 
(22 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
==== CERN Fio Benchmarks ====
+
== Drive preparation ==
 +
Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:
 +
=== Secure Erase ===
 +
* '''NVMe''' drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
 +
<syntaxhighlight>
 +
[root@localhost nvme-cli]# nvme format /dev/nvme0n1 --ses=1
 +
</syntaxhighlight>
 +
* '''SSDs''' can be secure erased using '''<code>hdparm</code>''' (further information can be found at [[https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase]]):
 +
<syntaxhighlight>
 +
root@ubuntu-10-04:~# hdparm --user-master u --security-set-pass password /dev/sdb
 +
security_password="password"
 +
 
 +
/dev/sdb:
 +
Issuing SECURITY_SET_PASS command, password="password", user=user, mode=high
 +
 
 +
-----
 +
 
 +
root@ubuntu-10-04:~# time hdparm --user-master u --security-erase password /dev/sdb
 +
security_password="password"
 +
 
 +
/dev/sdb:
 +
Issuing SECURITY_ERASE command, password="password", user=user
 +
 
 +
real 0m55.408s
 +
user 0m0.000s
 +
sys 0m0.000s
 +
root@ubuntu-10-04:~#
 +
</syntaxhighlight>
 +
 
 +
== CERN Fio Benchmarks ==
 
* Ref: http://it-div-procurements.web.cern.ch/it-div-procurements/IT-3821/fio/
 
* Ref: http://it-div-procurements.web.cern.ch/it-div-procurements/IT-3821/fio/
 
* Setup fio config file (/root/fio/fio-bench) with content below for the benchmarks run  
 
* Setup fio config file (/root/fio/fio-bench) with content below for the benchmarks run  
  
−
<pre>
+
<syntaxhighlight>
 
[global]
 
[global]
 
direct=1
 
direct=1
Line 9: Line 38:
 
####################
 
####################
 
#numjobs should be half of the number of physical cores on the sample system.
 
#numjobs should be half of the number of physical cores on the sample system.
−
numjobs=2
+
numjobs=16
 
###################
 
###################
−
directory=/srv/castor/04   # directory where '/dev/sda1' disk is mounted on
+
directory=/pool   # directory where '/dev/sda1' disk is mounted on
 
iodepth=24
 
iodepth=24
 
timeout=3600
 
timeout=3600
Line 17: Line 46:
 
size=10000MB
 
size=10000MB
 
rw=randrw  
 
rw=randrw  
−
</pre>
+
</syntaxhighlight>
  
−
* Run fio benchmark (average over 3 runs)
+
=== Run fio benchmark (average over 3 runs) ===
−
<pre>
+
<syntaxhighlight>
 
for i in {1..3}
 
for i in {1..3}
 
do
 
do
Line 27: Line 56:
 
done
 
done
  
−
</pre>
+
</syntaxhighlight>
  
−
* Results:- grep -i iops /root/fio-bench.[1|2|3].log and sum all the iops values
+
== Check results ==
−
<pre>
+
*Check results: <code>grep -i iops /root/fio-bench.[1|2|3].log</code> and sum all the iops values
 +
<syntaxhighlight>
 
grep -i iops /root/fio-bench.1.log  
 
grep -i iops /root/fio-bench.1.log  
 
fio-bench.1.log:  read : io=728220KB, bw=207137 B/s, iops=50 , runt=3600019msec
 
fio-bench.1.log:  read : io=728220KB, bw=207137 B/s, iops=50 , runt=3600019msec
Line 49: Line 79:
 
fio-bench.3.log:  write: io=676816KB, bw=192515 B/s, iops=47 , runt=3600025msec
 
fio-bench.3.log:  write: io=676816KB, bw=192515 B/s, iops=47 , runt=3600025msec
  
−
IOPS (single drive):= (200 + 164 + 185)/3 = 183
+
IOPS (average):= (200 + 164 + 185)/3 = 183
−
</pre>
+
</syntaxhighlight>
  
 
* Calculate the IOPS from the log file
 
* Calculate the IOPS from the log file
−
<pre>
+
<syntaxhighlight>
−
  grep -i iops fio-bench.1.log | sed -e 's/=/ /g' -e 's/:/ /g'  | awk '{sum=sum+$8} END {print sum}'
+
grep -i iops fio-bench.1.log | sed -e 's/=/ /g' -e 's/:/ /g'  | awk '{sum=sum+$8} END {print sum}'
−
</pre>
+
</syntaxhighlight>
  
−
==== Results ====
+
== CERN Results ==
  
−
{| class="wikitable" cellpadding=5 style="border:1px solid #BBB"
+
{| class="wikitable" cellpadding=5 style="border:1px; width:100%;"
 +
|-
 +
! Drive Model                                    !! Number of drives !! Form Factor !! Drive RPM || Interface ||  IOPS
 +
|-
 +
| 250GB (WD2503ABYX)                                    || 3 (raid0) || 3.5" || 7200 RPM || 3Gb SATA || 471
 +
|-
 +
| Intel 320 40Gb                                              || 3 (raid0) || 2.5" || N/A          || 3Gb SATA || 1832
 +
|-
 +
| Intel 320 40Gb                                              || 2 (raid0) || 2.5" || N/A          || 3Gb SATA || 870*
 
|-
 
|-
−
! Drive Model  !! Number of drives !! Drive RPM || IOPS
+
| Intel 320 40Gb                                              || 1            || 2.5" || N/A          || 3Gb SATA || 606
 
|-
 
|-
−
| 250GB (WD2503ABYX)  || 3 (raid0)   || 7200 RPM  || 471
+
| Intel 320 160Gb                                            || 2 (raid0) || 2.5" || N/A          || 3Gb SATA || 3279
 
|-
 
|-
−
| 2TB                || 2 (raid0)  || 5400 RPM  || XXX
+
| Intel 320 160Gb                                            || 1            || 2.5" || N/A          || 3Gb SATA || 1591
 
|-
 
|-
−
| Intel 320 40Gb      || 3 (raid0)  || N/A  || 1832
+
| Seagate Constellation.2 1TB                          || 1            || 2.5" || 7200 RPM || 6Gb SATA || 332
 
|-
 
|-
−
| Intel 320 40Gb      || 2 (raid0)   || N/A  || 870*
+
| Seagate ConstellationES 2TB (ST2000NM001) || 3 (raid0) || 3.5" || 7200 RPM || 6Gb SAS    || 766
 
|-
 
|-
−
| Intel 320 40Gb      || 1          || N/A  || 606
+
| Hitachi Utrastar 2TB (HUA723020ALA640)    || 3 (raid0) || 3.5" || 7200 RPM || 6Gb SATA || 559
 
|-
 
|-
−
| Intel 320 160Gb    || 2 (raid0)   || N/A   || 3279
+
| Toshiba 2TB (MK2001TKRB)                          || 3 (raid0) || 3.5" || 7200 RPM || 6Gb SAS    || 818
 +
|-
 +
| OCZ Z-Drive R4 800GB                                  || 1            || PCI-E || N/A       || PCI-E x8 (5Gb/s) || 24,440
 
|-
 
|-
−
| Intel 320 160Gb    || 1          || N/A  || 1591
 
 
|}
 
|}
  
 
  *Below expected, please run again when drives are available.
 
  *Below expected, please run again when drives are available.
 
  Could be due to wear as this test was run directly after the 3 drive raid 0.
 
  Could be due to wear as this test was run directly after the 3 drive raid 0.
 +
 +
== General IOPS Tests ==
 +
I've found fio (http://freshmeat.net/projects/fio/) to be an excellent testing tool for disk systems.  To use it, compile it (requires libaio-devel), and then run it as
 +
 +
fio input.fio
 +
 +
For a nice simple IOP test, try this:
 +
 +
<syntaxhighlight>
 +
[random]
 +
rw=randread
 +
size=4g
 +
directory=/data
 +
iodepth=32
 +
blocksize=4k
 +
numjobs=16
 +
nrfiles=1
 +
group_reporting
 +
ioengine=sync
 +
loops=1
 +
</syntaxhighlight>
 +
 +
This file will perform 4GB of IO into a directory named /data, using an IO depth of 32, a block size of 4k (the IOP measurement standard) with random reads as the major operation, using standard unix IO.  We have 16 simultaneous jobs doing IO, each job using 1 file.  It will aggregate all the information from each job and report it, and it will run once.
 +
 +
then to run the test:
 +
<syntaxhighlight>
 +
fio file.fio
 +
</syntaxhighlight>
 +
 +
== CLI FIO ==
 +
<syntaxhighlight>
 +
# Another IOPS test:
 +
fio -name iops -rw=read -bs=512 -runtime=180 -iodepth 4 -filename /dev/sdb -ioengine libaio -direct=1 -numjobs=16
 +
 +
# Write IOPS Test
 +
fio --filename=/dev/md0 --direct=1 --rw=randwrite --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1
 +
 
 +
#Read IOPS Test
 +
fio --filename=/dev/md0 --direct=1 --rw=randread --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1 
 +
</syntaxhighlight>
 +
 +
Yet more FIO tests (bandwidth/iops)
 +
<syntaxhighlight>
 +
# Write Bandwidth test
 +
 +
$ fio --name=writebw --filename=/dev/fioa --direct=1 --rw=randwrite --bs=1m --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
 +
 +
# Read IOPS test
 +
$ fio --name=readiops --filename=/dev/fioa --direct=1 --rw=randread --bs=512 --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
 +
 +
# Read Bandwidth test
 +
$ fio --name=readbw --filename=/dev/fioa --direct=1 --rw=randread --bs=1m --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
 +
 +
# Write IOPS test
 +
$ fio --name=writeiops --filename=/dev/fioa --direct=1 --rw=randwrite --bs=512 --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
 +
</syntaxhighlight>
 +
 +
== FIO to generate network traffic ==
 +
* Ripped from the web somewhere
 +
 +
Generating Constant Bandwidth on Linux using fio
 +
 +
 +
It took a lot of searching for me to find a way to generate network traffic at a specific rate between two hosts, so I thought I would share the answer. It's pretty easy to test the available bandwidth between two hosts using netcat to transfer a bunch of random data as fast as the network allows. However I wanted to test resource monitoring and graphing system, which means I needed to generate network traffic at a known rate so that I could judge the resulting graphs against my expectations.
 +
 +
 +
 +
I found you can use fio, which is a generic I/O testing tool, to achieve this. fio allows specifying the transfer rate and also has a network engine. So using fio I can configure one host as the receiver and one as the sender and transfer data at a known rate. Here's what the config files look like:
 +
 +
Sender jobfile.ini:
 +
<syntaxhighlight>
 +
[test1]
 +
filename=receiver.dns.name/7777
 +
rw=write
 +
rate=750k
 +
size=100M
 +
ioengine=net
 +
</syntaxhighlight>
 +
 +
Receiver jobfile.ini:
 +
<syntaxhighlight>
 +
[test1]
 +
filename=localhost/7777
 +
rw=read
 +
size=100M
 +
ioengine=net
 +
</syntaxhighlight>
 +
 +
Obviously you would replace "receiver.dns.name" with the DNS name of the receiving host, and adjust the size and rate parameters as you like. (It's worth noting that the fio documentation is either wrong or misleading on what the filename should be for the receiver. It claims the receiver should only specify the port, but when I tried that it failed to run. Setting the host to localhost seemed to work and the receiver started listening.) To run the test, simply run:
 +
 +
<syntaxhighlight>
 +
fio jobfile.ini
 +
</syntaxhighlight>
 +
 +
first on the receiving host, then on the sending host. fio will then transfer 100 Megabytes of data at a rate of 750KB/sec between the two hosts. And we can see from the chart that indeed a constant rate was generated:
 +
 +
The observed rate is a bit above the 750KB/sec specified, but what's being measured is the number of bytes being transferred through the eth0 interface. Since the data is transferred over TCP there is some overhead to the packet structure, which I believe accounts for the extra few KB/sec observed.

Latest revision as of 09:13, 4 November 2016

Drive preparation

Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:

Secure Erase

[root@localhost nvme-cli]# nvme format /dev/nvme0n1 --ses=1
root@ubuntu-10-04:~# hdparm --user-master u --security-set-pass password /dev/sdb
security_password="password"

/dev/sdb:
 Issuing SECURITY_SET_PASS command, password="password", user=user, mode=high

-----

root@ubuntu-10-04:~# time hdparm --user-master u --security-erase password /dev/sdb
security_password="password"

/dev/sdb:
 Issuing SECURITY_ERASE command, password="password", user=user

real	0m55.408s
user	0m0.000s
sys	0m0.000s
root@ubuntu-10-04:~#

CERN Fio Benchmarks

[global]
direct=1
bsrange=4k-4k
####################
#numjobs should be half of the number of physical cores on the sample system.
numjobs=16
###################
directory=/pool    # directory where '/dev/sda1' disk is mounted on
iodepth=24
timeout=3600
[f1]
size=10000MB
rw=randrw

Run fio benchmark (average over 3 runs)

for i in {1..3}
do
	fio --eta=always --latency-log --bandwidth-log --output=/root/fio/fio-bench.${i}.log /root/fio/fio-bench
	sleep 3
done

Check results

  • Check results: grep -i iops /root/fio-bench.[1|2|3].log and sum all the iops values
grep -i iops /root/fio-bench.1.log 
fio-bench.1.log:  read : io=728220KB, bw=207137 B/s, iops=50 , runt=3600019msec
fio-bench.1.log:  write: io=727012KB, bw=206793 B/s, iops=50 , runt=3600019msec
fio-bench.1.log:  read : io=729100KB, bw=207388 B/s, iops=50 , runt=3600001msec
fio-bench.1.log:  write: io=723424KB, bw=205773 B/s, iops=50 , runt=3600001msec

grep -i iops /root/fio-bench.2.log
fio-bench.2.log:  read : io=598968KB, bw=170372 B/s, iops=41 , runt=3600006msec
fio-bench.2.log:  write: io=603348KB, bw=171618 B/s, iops=41 , runt=3600006msec
fio-bench.2.log:  read : io=601040KB, bw=170962 B/s, iops=41 , runt=3600001msec
fio-bench.2.log:  write: io=600416KB, bw=170784 B/s, iops=41 , runt=3600001msec

grep -i iops /root/fio-bench.3.log
fio-bench.3.log:  read : io=675160KB, bw=192044 B/s, iops=46 , runt=3600023msec
fio-bench.3.log:  write: io=675108KB, bw=192029 B/s, iops=46 , runt=3600023msec
fio-bench.3.log:  read : io=675532KB, bw=192149 B/s, iops=46 , runt=3600025msec
fio-bench.3.log:  write: io=676816KB, bw=192515 B/s, iops=47 , runt=3600025msec

IOPS (average):= (200 + 164 + 185)/3 = 183
  • Calculate the IOPS from the log file
 grep -i iops fio-bench.1.log | sed -e 's/=/ /g' -e 's/:/ /g'  | awk '{sum=sum+$8} END {print sum}'

CERN Results

Drive Model Number of drives Form Factor Drive RPM Interface IOPS
250GB (WD2503ABYX) 3 (raid0) 3.5" 7200 RPM 3Gb SATA 471
Intel 320 40Gb 3 (raid0) 2.5" N/A 3Gb SATA 1832
Intel 320 40Gb 2 (raid0) 2.5" N/A 3Gb SATA 870*
Intel 320 40Gb 1 2.5" N/A 3Gb SATA 606
Intel 320 160Gb 2 (raid0) 2.5" N/A 3Gb SATA 3279
Intel 320 160Gb 1 2.5" N/A 3Gb SATA 1591
Seagate Constellation.2 1TB 1 2.5" 7200 RPM 6Gb SATA 332
Seagate ConstellationES 2TB (ST2000NM001) 3 (raid0) 3.5" 7200 RPM 6Gb SAS 766
Hitachi Utrastar 2TB (HUA723020ALA640) 3 (raid0) 3.5" 7200 RPM 6Gb SATA 559
Toshiba 2TB (MK2001TKRB) 3 (raid0) 3.5" 7200 RPM 6Gb SAS 818
OCZ Z-Drive R4 800GB 1 PCI-E N/A PCI-E x8 (5Gb/s) 24,440 
*Below expected, please run again when drives are available.
Could be due to wear as this test was run directly after the 3 drive raid 0.

General IOPS Tests

I've found fio (http://freshmeat.net/projects/fio/) to be an excellent testing tool for disk systems. To use it, compile it (requires libaio-devel), and then run it as

fio input.fio

For a nice simple IOP test, try this: 

[random]
rw=randread
size=4g
directory=/data
iodepth=32
blocksize=4k
numjobs=16
nrfiles=1
group_reporting
ioengine=sync
loops=1

This file will perform 4GB of IO into a directory named /data, using an IO depth of 32, a block size of 4k (the IOP measurement standard) with random reads as the major operation, using standard unix IO. We have 16 simultaneous jobs doing IO, each job using 1 file. It will aggregate all the information from each job and report it, and it will run once.

then to run the test: 

fio file.fio

CLI FIO

# Another IOPS test:
fio -name iops -rw=read -bs=512 -runtime=180 -iodepth 4 -filename /dev/sdb -ioengine libaio -direct=1 -numjobs=16

# Write IOPS Test
fio --filename=/dev/md0 --direct=1 --rw=randwrite --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1
  
#Read IOPS Test
fio --filename=/dev/md0 --direct=1 --rw=randread --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1

Yet more FIO tests (bandwidth/iops) 

# Write Bandwidth test

$ fio --name=writebw --filename=/dev/fioa --direct=1 --rw=randwrite --bs=1m --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
 
# Read IOPS test
$ fio --name=readiops --filename=/dev/fioa --direct=1 --rw=randread --bs=512 --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting

# Read Bandwidth test
$ fio --name=readbw --filename=/dev/fioa --direct=1 --rw=randread --bs=1m --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting

# Write IOPS test
$ fio --name=writeiops --filename=/dev/fioa --direct=1 --rw=randwrite --bs=512 --numjobs=4 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting

FIO to generate network traffic

  • Ripped from the web somewhere

Generating Constant Bandwidth on Linux using fio


It took a lot of searching for me to find a way to generate network traffic at a specific rate between two hosts, so I thought I would share the answer. It's pretty easy to test the available bandwidth between two hosts using netcat to transfer a bunch of random data as fast as the network allows. However I wanted to test resource monitoring and graphing system, which means I needed to generate network traffic at a known rate so that I could judge the resulting graphs against my expectations.


I found you can use fio, which is a generic I/O testing tool, to achieve this. fio allows specifying the transfer rate and also has a network engine. So using fio I can configure one host as the receiver and one as the sender and transfer data at a known rate. Here's what the config files look like:

Sender jobfile.ini: 

[test1]
filename=receiver.dns.name/7777
rw=write
rate=750k
size=100M
ioengine=net

Receiver jobfile.ini: 

[test1]
filename=localhost/7777
rw=read
size=100M
ioengine=net

Obviously you would replace "receiver.dns.name" with the DNS name of the receiving host, and adjust the size and rate parameters as you like. (It's worth noting that the fio documentation is either wrong or misleading on what the filename should be for the receiver. It claims the receiver should only specify the port, but when I tried that it failed to run. Setting the host to localhost seemed to work and the receiver started listening.) To run the test, simply run: 

fio jobfile.ini

first on the receiving host, then on the sending host. fio will then transfer 100 Megabytes of data at a rate of 750KB/sec between the two hosts. And we can see from the chart that indeed a constant rate was generated:

The observed rate is a bit above the 750KB/sec specified, but what's being measured is the number of bytes being transferred through the eth0 interface. Since the data is transferred over TCP there is some overhead to the packet structure, which I believe accounts for the extra few KB/sec observed.

Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Benchmarking:_FIO&oldid=17751"