Difference between revisions of "Benchmarking: STFC Storage (GridPP) with Iozone"

From Define Wiki
Jump to navigation Jump to search
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
* Notes below are based on the 2014 test suits (these tests have been fairly consistent over the years)
 
* Notes below are based on the 2014 test suits (these tests have been fairly consistent over the years)
 +
* Using Centos 6.5
 +
* Installed Iozone and setup in the $PATH
 +
  
 
== Setup the RAID Array ==
 
== Setup the RAID Array ==
 +
Useful background to LSI Tuning
 +
* http://www.lsi.com/downloads/Public/Direct%20Assets/LSI/Benchmark_Tips.pdf
 +
* http://www.lsi.com/downloads/Public/Direct%20Assets/LSI/6Gbs_LSI_LRb.pdf
 +
 +
System Setup:
 +
 
* 22 Drives RAID6 array with WD SE WD4000F9YS Drives
 
* 22 Drives RAID6 array with WD SE WD4000F9YS Drives
 
* E5-2620v2 CPUs
 
* E5-2620v2 CPUs
 
* 64GB RAM  
 
* 64GB RAM  
 
* LSI 9261 Controller with the following settings:
 
* LSI 9261 Controller with the following settings:
−
** RAID Level: 6
+
** RAID Level: <b>6</b>
−
** Stripe Size: 256k
+
** Stripe Size: <b>256k</b>
−
** Disk Cache Policy: Enabled
+
** Disk Cache Policy: <b>Enabled</b>
−
** Read Policy: Always Read Ahead
+
** Read Policy: <b>Always Read Ahead</b>
−
** Write Policy: Always Write Back  
+
** Write Policy: <b>Always Write Back </b>
−
** IO Policy: Direct IO
+
** IO Policy: <b>Direct IO</b>
  
 
== Partition and Create FS ==
 
== Partition and Create FS ==
Line 72: Line 81:
 
[scheduler]:  
 
[scheduler]:  
 
noop anticipatory [deadline] cfq  
 
noop anticipatory [deadline] cfq  
 +
</syntaxhighlight>
 +
 +
== Run the tests ==
 +
<syntaxhighlight>
 +
[root@localhost dp]# cat run_tests.sh
 +
#!/bin/bash
 +
 +
for i in {1..3}; do
 +
  for j in {1..4}; do
 +
      iozone -Mce -s24g -r256k -i0 -i1 -f /exportstage/castor${i}/iozone.dat >> single-result1
 +
  done
 +
done
 +
for i in {1..3}; do
 +
  for j in {1..4}; do
 +
      iozone -Mce -s24g -r256k -i0 -i1 -f /exportstage/castor${i}/iozone.dat >> single-result2
 +
  done
 +
done
 +
 +
 +
for i in {1..3}; do
 +
  iozone -MCce -t12 -s24g -r256k -i0 -i1 -F /exportstage/castor{1,2,3}/iozone{1,2,3,4}.dat \
 +
  >> multi-result1
 +
done
 +
 +
for i in {1..3}; do
 +
  iozone -MCce -t12 -s24g -r256k -i0 -i1 -F /exportstage/castor{1,2,3}/iozone{1,2,3,4}.dat \
 +
  >> multi-result2
 +
done
 +
 +
iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 33 -F \
 +
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-write-heavy-result1
 +
 +
iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 33 -F \
 +
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-write-heavy-result2
 +
 +
iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 66 -F \
 +
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-read-heavy-result1
 +
 +
iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 66 -F \
 +
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-read-heavy-result2
 +
 +
 +
</syntaxhighlight>
 +
 +
== Verify the Results ==
 +
* Run the script below with an argument of '1' or '2' to check the outputs from the tests above
 +
<syntaxhighlight>
 +
[david@head-boston results_final_FULLYBUILTARRAY]$ cat ../check_results.sh
 +
#!/bin/bash
 +
 +
 +
 +
# Single Tests
 +
echo "Single Tests"
 +
echo "grep '    25165824' single-result${1}"
 +
grep '    25165824' single-result${1}
 +
 +
# Multi Tests
 +
echo "Aggregate rate testing"
 +
echo "grep 'initial writers' multi-result "
 +
grep 'initial writers' multi-result${1}
 +
echo "grep '12 readers' multi-result "
 +
grep '12 readers' multi-result${1}
 +
 +
# Real world performance tests
 +
echo "Real world performance tests; Some reads, many writes"
 +
echo "grep 'Parent sees throughput for 150 mixed workload' mixed-io-write-heavy-result "
 +
grep 'Parent sees throughput for 150 mixed workload' mixed-io-write-heavy-result${1}
 +
 +
# Real world performance tests; Some writes, many reads
 +
echo "Real world performance tests; Some writes, many reads"
 +
echo "grep 'Parent sees throughput for 150 mixed workload' mixed-io-read-heavy-result${1} "
 +
grep 'Parent sees throughput for 150 mixed workload' mixed-io-read-heavy-result${1}
 +
[david@head-boston results_final_FULLYBUILTARRAY]$ ../check_results.sh 1
 +
Single Tests
 +
grep '    25165824' single-result1
 +
        25165824    256  1116764  1236586  5711246  5838919                                                                         
 +
        25165824    256  1060506  1238687  4380646  4491684                                                                         
 +
        25165824    256  1091064  1338011  6440526  6795254                                                                         
 +
        25165824    256  1044567  1236360  5227321  5267242                                                                         
 +
        25165824    256  1076830  1251129  5674987  5815534                                                                         
 +
        25165824    256  1017221  1281913  6160165  6461448                                                                         
 +
        25165824    256  1088152  1237634  5419563  5553995                                                                         
 +
        25165824    256  1173280  1299911  6320133  6623178                                                                         
 +
        25165824    256  1078512  1310252  6186337  6484043                                                                         
 +
        25165824    256  1128769  1354401  6427776  6796484                                                                         
 +
        25165824    256  1051144  1223478  4271137  4314917                                                                         
 +
        25165824    256  1063606  1316473  6128245  6442826                                                                         
 +
Aggregate rate testing
 +
grep 'initial writers' multi-result
 +
Children see throughput for 12 initial writers = 1245435.05 kB/sec
 +
Parent sees throughput for 12 initial writers = 1237482.76 kB/sec
 +
Children see throughput for 12 initial writers = 1243377.82 kB/sec
 +
Parent sees throughput for 12 initial writers = 1237695.98 kB/sec
 +
Children see throughput for 12 initial writers = 1244007.71 kB/sec
 +
Parent sees throughput for 12 initial writers = 1236185.42 kB/sec
 +
grep '12 readers' multi-result
 +
Children see throughput for 12 readers = 1243598.75 kB/sec
 +
Parent sees throughput for 12 readers = 1243328.93 kB/sec
 +
Children see throughput for 12 readers = 1269178.80 kB/sec
 +
Parent sees throughput for 12 readers = 1268976.48 kB/sec
 +
Children see throughput for 12 readers = 1249611.54 kB/sec
 +
Parent sees throughput for 12 readers = 1249376.63 kB/sec
 +
Real world performance tests; Some reads, many writes
 +
grep 'Parent sees throughput for 150 mixed workload' mixed-io-write-heavy-result
 +
Parent sees throughput for 150 mixed workload =  265133.85 kB/sec
 +
Real world performance tests; Some writes, many reads
 +
grep 'Parent sees throughput for 150 mixed workload' mixed-io-read-heavy-result1
 +
Parent sees throughput for 150 mixed workload =  356142.63 kB/sec
 
</syntaxhighlight>
 
</syntaxhighlight>

Latest revision as of 08:49, 20 October 2014

  • Notes below are based on the 2014 test suits (these tests have been fairly consistent over the years)
  • Using Centos 6.5
  • Installed Iozone and setup in the $PATH


Setup the RAID Array

Useful background to LSI Tuning

System Setup:

  • 22 Drives RAID6 array with WD SE WD4000F9YS Drives
  • E5-2620v2 CPUs
  • 64GB RAM
  • LSI 9261 Controller with the following settings:
    • RAID Level: 6
    • Stripe Size: 256k
    • Disk Cache Policy: Enabled
    • Read Policy: Always Read Ahead
    • Write Policy: Always Write Back
    • IO Policy: Direct IO

Partition and Create FS

[root@localhost scripts]# cat intelligent-partitioner-dp.sh 
#!/bin/bash

# Parted stuff
parted -a optimal /dev/sda unit % mkpart primary 0% 33%    ; sleep 1
parted -a optimal /dev/sda unit % mkpart primary 33% 66%   ; sleep 1
parted -a optimal /dev/sda unit % mkpart primary 66% 100%  ; sleep 1

# sleep for 5
sleep 5

partprobe /dev/sda

sleep 5
# Create the XFS Part
mkfs.xfs -f -l version=2 -i size=1024 -n size=65536 -d su=256k,sw=22 -L castor1 /dev/sda1
sleep 2

mkfs.xfs -f -l version=2 -i size=1024 -n size=65536 -d su=256k,sw=22 -L castor2 /dev/sda2
sleep 2

mkfs.xfs -f -l version=2 -i size=1024 -n size=65536 -d su=256k,sw=22 -L castor3 /dev/sda3
sleep 2

Mount the Filesystems - add the follwing to fstab 

# fstab
/dev/disk/by-label/castor1     /exportstage/castor1  xfs  logbufs=8,logbsize=256k,noatime,swalloc,inode64  0 2
/dev/disk/by-label/castor2     /exportstage/castor2  xfs  logbufs=8,logbsize=256k,noatime,swalloc,inode64  0 2
/dev/disk/by-label/castor3     /exportstage/castor3  xfs  logbufs=8,logbsize=256k,noatime,swalloc,inode64  0 2

Tuning on the System

  • Set the following
    • Readahead
    • Request Queue
    • Scheduler
# To Set
# blockdev --setra 16384 /dev/sda
# echo 512 > /sys/block/sda/queue/nr_requests
# echo deadline > /sys/block/sda/queue/scheduler
[root@localhost dp]# cat check_settings.sh 
#!/bin/bash

echo -en "[readahead]: "
blockdev --getra /dev/sda
echo -en "[nr_requests]: "
cat /sys/block/sda/queue/nr_requests
echo "[scheduler]: "
cat /sys/block/sda/queue/scheduler
[root@localhost dp]# ./check_settings.sh 
[readahead]: 16384
[nr_requests]: 512
[scheduler]: 
noop anticipatory [deadline] cfq

Run the tests

[root@localhost dp]# cat run_tests.sh 
#!/bin/bash

for i in {1..3}; do
   for j in {1..4}; do
      iozone -Mce -s24g -r256k -i0 -i1 -f /exportstage/castor${i}/iozone.dat >> single-result1
   done
done
for i in {1..3}; do
   for j in {1..4}; do
      iozone -Mce -s24g -r256k -i0 -i1 -f /exportstage/castor${i}/iozone.dat >> single-result2
   done
done


for i in {1..3}; do
   iozone -MCce -t12 -s24g -r256k -i0 -i1 -F /exportstage/castor{1,2,3}/iozone{1,2,3,4}.dat \
   >> multi-result1
done

for i in {1..3}; do
   iozone -MCce -t12 -s24g -r256k -i0 -i1 -F /exportstage/castor{1,2,3}/iozone{1,2,3,4}.dat \
   >> multi-result2
done

iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 33 -F \
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-write-heavy-result1

iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 33 -F \
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-write-heavy-result2

iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 66 -F \
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-read-heavy-result1

iozone -MCce -t 150 -s2g -r256k -i0 -i8 -+p 66 -F \
/exportstage/castor{1..3}/iozone{1..50}.dat >> mixed-io-read-heavy-result2

Verify the Results

  • Run the script below with an argument of '1' or '2' to check the outputs from the tests above
[david@head-boston results_final_FULLYBUILTARRAY]$ cat ../check_results.sh 
#!/bin/bash



# Single Tests
echo "Single Tests"
echo "grep '     25165824' single-result${1}"
grep '     25165824' single-result${1}

# Multi Tests
echo "Aggregate rate testing"
echo "grep 'initial writers' multi-result "
grep 'initial writers' multi-result${1}
echo "grep '12 readers' multi-result "
grep '12 readers' multi-result${1}

# Real world performance tests
echo "Real world performance tests; Some reads, many writes"
echo "grep 'Parent sees throughput for 150 mixed workload' mixed-io-write-heavy-result "
grep 'Parent sees throughput for 150 mixed workload' mixed-io-write-heavy-result${1}

# Real world performance tests; Some writes, many reads
echo "Real world performance tests; Some writes, many reads"
echo "grep 'Parent sees throughput for 150 mixed workload' mixed-io-read-heavy-result${1} "
grep 'Parent sees throughput for 150 mixed workload' mixed-io-read-heavy-result${1}
[david@head-boston results_final_FULLYBUILTARRAY]$ ../check_results.sh 1
Single Tests
grep '     25165824' single-result1
        25165824     256  1116764  1236586  5711246  5838919                                                                          
        25165824     256  1060506  1238687  4380646  4491684                                                                          
        25165824     256  1091064  1338011  6440526  6795254                                                                          
        25165824     256  1044567  1236360  5227321  5267242                                                                          
        25165824     256  1076830  1251129  5674987  5815534                                                                          
        25165824     256  1017221  1281913  6160165  6461448                                                                          
        25165824     256  1088152  1237634  5419563  5553995                                                                          
        25165824     256  1173280  1299911  6320133  6623178                                                                          
        25165824     256  1078512  1310252  6186337  6484043                                                                          
        25165824     256  1128769  1354401  6427776  6796484                                                                          
        25165824     256  1051144  1223478  4271137  4314917                                                                          
        25165824     256  1063606  1316473  6128245  6442826                                                                          
Aggregate rate testing
grep 'initial writers' multi-result 
	Children see throughput for 12 initial writers 	= 1245435.05 kB/sec
	Parent sees throughput for 12 initial writers 	= 1237482.76 kB/sec
	Children see throughput for 12 initial writers 	= 1243377.82 kB/sec
	Parent sees throughput for 12 initial writers 	= 1237695.98 kB/sec
	Children see throughput for 12 initial writers 	= 1244007.71 kB/sec
	Parent sees throughput for 12 initial writers 	= 1236185.42 kB/sec
grep '12 readers' multi-result 
	Children see throughput for 12 readers 		= 1243598.75 kB/sec
	Parent sees throughput for 12 readers 		= 1243328.93 kB/sec
	Children see throughput for 12 readers 		= 1269178.80 kB/sec
	Parent sees throughput for 12 readers 		= 1268976.48 kB/sec
	Children see throughput for 12 readers 		= 1249611.54 kB/sec
	Parent sees throughput for 12 readers 		= 1249376.63 kB/sec
Real world performance tests; Some reads, many writes
grep 'Parent sees throughput for 150 mixed workload' mixed-io-write-heavy-result 
	Parent sees throughput for 150 mixed workload 	=  265133.85 kB/sec
Real world performance tests; Some writes, many reads
grep 'Parent sees throughput for 150 mixed workload' mixed-io-read-heavy-result1 
	Parent sees throughput for 150 mixed workload 	=  356142.63 kB/sec
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Benchmarking:_STFC_Storage_(GridPP)_with_Iozone&oldid=5009"