Benchmarking: FIO - Revision history

Jon: /* Secure Erase */

2016-11-04T09:13:57Z

Secure Erase

Jon: /* Drive preparation */

2016-11-04T09:12:51Z

Drive preparation

Jon: /* Secure Erase */

2016-11-04T09:12:25Z

Secure Erase

Jon: /* Secure Erase */

2016-11-04T09:11:39Z

Secure Erase

Jon: /* Drive preparation */

2016-11-04T09:10:37Z

Drive preparation

Jon at 09:09, 4 November 2016

2016-11-04T09:09:41Z

David: /* CLI FIO */

2013-09-03T07:30:02Z

CLI FIO

David at 07:20, 3 September 2013

2013-09-03T07:20:59Z

David: /* CLI FIO */

2012-08-23T12:18:33Z

CLI FIO

Jon: /* CERN Results */

2012-08-22T17:56:44Z

CERN Results

@@ Line 6: / Line 6: @@
 [root@localhost nvme-cli]# nvme format /dev/nvme0n1 --ses=1
 </syntaxhighlight>
-* '''SSDs''' can be secure erased using '''<code>hdparm</code>''':
+* '''SSDs''' can be secure erased using '''<code>hdparm</code>''' (further information can be found at [[https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase]]):
 <syntaxhighlight>
 root@ubuntu-10-04:~# hdparm --user-master u --security-set-pass password /dev/sdb
@@ Line 27: / Line 27: @@
 root@ubuntu-10-04:~#
 </syntaxhighlight>
 == CERN Fio Benchmarks ==

@@ Line 1: / Line 1: @@
 == Drive preparation ==
-* Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:
+Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:
 === Secure Erase ===
 * '''NVMe''' drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]

@@ Line 2: / Line 2: @@
 * Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:
 === Secure Erase ===
-* NVMe drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
+* '''NVMe''' drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
 <syntaxhighlight>
 [root@localhost nvme-cli]# nvme format /dev/nvme0n1 --ses=1
 </syntaxhighlight>
-* SSDs can be secure erased using '''<code>hdparm</code>''':
+* '''SSDs''' can be secure erased using '''<code>hdparm</code>''':
 <syntaxhighlight>
 root@ubuntu-10-04:~# hdparm --user-master u --security-set-pass password /dev/sdb
@@ Line 12: / Line 12: @@
 /dev/sdb:
-  Issuing SECURITY_SET_PASS command, password="Eins", user=user, mode=high
+  Issuing SECURITY_SET_PASS command, password="password", user=user, mode=high
 -----

@@ Line 2: / Line 2: @@
 * Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:
 === Secure Erase ===
-* NVMe drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation|https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
+* NVMe drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
 <syntaxhighlight>
 [root@localhost nvme-cli]# nvme format /dev/nvme0n1 --ses=1
@@ Line 27: / Line 27: @@
 root@ubuntu-10-04:~#
 </syntaxhighlight>
-N.B. - Further information can be found at [[https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase|https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase]]
+N.B. - Further information can be found at [[https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase]]
 == CERN Fio Benchmarks ==

@@ Line 2: / Line 2: @@
 * Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:
 === Secure Erase ===
-* NVMe drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
+* NVMe drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation|https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
 <syntaxhighlight>
 [root@localhost nvme-cli]# nvme format /dev/nvme0n1 --ses=1
@@ Line 27: / Line 27: @@
 root@ubuntu-10-04:~#
 </syntaxhighlight>
-N.B. - Further information can be found at [[https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase]]
+N.B. - Further information can be found at [[https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase|https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase]]
 == CERN Fio Benchmarks ==

← Older revision		Revision as of 09:09, 4 November 2016
Line 1:		Line 1:
		+	== Drive preparation ==
		+	* Before benchmarking SSDs or NVMe drives, the devices in question should be erased and pre-conditioned with random data:
		+	=== Secure Erase ===
		+	* NVMe drives can be secure erased using this handy name tool: [[https://github.com/linux-nvme/nvme-cli/tree/master/Documentation]]
		+	<syntaxhighlight>
		+	[root@localhost nvme-cli]# nvme format /dev/nvme0n1 --ses=1
		+	</syntaxhighlight>
		+	* SSDs can be secure erased using '''<code>hdparm</code>''':
		+	<syntaxhighlight>
		+	root@ubuntu-10-04:~# hdparm --user-master u --security-set-pass password /dev/sdb
		+	security_password="password"
		+
		+	/dev/sdb:
		+	Issuing SECURITY_SET_PASS command, password="Eins", user=user, mode=high
		+
		+	-----
		+
		+	root@ubuntu-10-04:~# time hdparm --user-master u --security-erase password /dev/sdb
		+	security_password="password"
		+
		+	/dev/sdb:
		+	Issuing SECURITY_ERASE command, password="password", user=user
		+
		+	real 0m55.408s
		+	user 0m0.000s
		+	sys 0m0.000s
		+	root@ubuntu-10-04:~#
		+	</syntaxhighlight>
		+	N.B. - Further information can be found at [[https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase]]
		+
		+
	== CERN Fio Benchmarks ==		== CERN Fio Benchmarks ==
	* Ref: http://it-div-procurements.web.cern.ch/it-div-procurements/IT-3821/fio/		* Ref: http://it-div-procurements.web.cern.ch/it-div-procurements/IT-3821/fio/

@@ Line 129: / Line 129: @@
 #Read IOPS Test
 fio --filename=/dev/md0 --direct=1 --rw=randread --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1
 </syntaxhighlight>

← Older revision		Revision as of 07:20, 3 September 2013
Line 130:		Line 130:
	fio --filename=/dev/md0 --direct=1 --rw=randread --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1		fio --filename=/dev/md0 --direct=1 --rw=randread --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1
	</syntaxhighlight>		</syntaxhighlight>
		+
		+	== FIO to generate network traffic ==
		+	* Ripped from the web somewhere
		+
		+	Generating Constant Bandwidth on Linux using fio
		+
		+
		+	It took a lot of searching for me to find a way to generate network traffic at a specific rate between two hosts, so I thought I would share the answer. It's pretty easy to test the available bandwidth between two hosts using netcat to transfer a bunch of random data as fast as the network allows. However I wanted to test resource monitoring and graphing system, which means I needed to generate network traffic at a known rate so that I could judge the resulting graphs against my expectations.
		+
		+
		+
		+	I found you can use fio, which is a generic I/O testing tool, to achieve this. fio allows specifying the transfer rate and also has a network engine. So using fio I can configure one host as the receiver and one as the sender and transfer data at a known rate. Here's what the config files look like:
		+
		+	Sender jobfile.ini:
		+	<syntaxhighlight>
		+	[test1]
		+	filename=receiver.dns.name/7777
		+	rw=write
		+	rate=750k
		+	size=100M
		+	ioengine=net
		+	</syntaxhighlight>
		+
		+	Receiver jobfile.ini:
		+	<syntaxhighlight>
		+	[test1]
		+	filename=localhost/7777
		+	rw=read
		+	size=100M
		+	ioengine=net
		+	</syntaxhighlight>
		+
		+	Obviously you would replace "receiver.dns.name" with the DNS name of the receiving host, and adjust the size and rate parameters as you like. (It's worth noting that the fio documentation is either wrong or misleading on what the filename should be for the receiver. It claims the receiver should only specify the port, but when I tried that it failed to run. Setting the host to localhost seemed to work and the receiver started listening.) To run the test, simply run:
		+
		+	<syntaxhighlight>
		+	fio jobfile.ini
		+	</syntaxhighlight>
		+
		+	first on the receiving host, then on the sending host. fio will then transfer 100 Megabytes of data at a rate of 750KB/sec between the two hosts. And we can see from the chart that indeed a constant rate was generated:
		+
		+	The observed rate is a bit above the 750KB/sec specified, but what's being measured is the number of bytes being transferred through the eth0 interface. Since the data is transferred over TCP there is some overhead to the packet structure, which I believe accounts for the extra few KB/sec observed.

@@ Line 121: / Line 121: @@
 == CLI FIO ==
 <syntaxhighlight>
 # Write IOPS Test
 fio --filename=/dev/md0 --direct=1 --rw=randwrite --bs=4k --size=16G --numjobs=64 --runtime=10 --group_reporting --name=file1

@@ Line 83: / Line 83: @@
 |-
 | Toshiba 2TB (MK2001TKRB)                           || 3 (raid0) || 3.5" || 7200 RPM || 6Gb SAS    || 818
 |}