Difference between revisions of "Memblaze Cards"
Jump to navigation
Jump to search
| (One intermediate revision by the same user not shown) | |
(No difference)
| |
Latest revision as of 15:10, 30 August 2017
Driver Installation
All drivers were provided with teh cards and a copy can be found on storage1 in the hpc folder.
Cd into the driver folder and simply run these two commands:
make
make install
Check cards are detected
in the tools folder run the memmonitor to see the status of the cards:
Memblaze Monitor - Memblaze PCIE Accelerator Monitor
(c) 2014 Memblaze Technology Co., Ltd (All Rights Reserved)
Basic Information
------------------------------------------------------------------------------------------------------------------------------------------------------
Manufacture Name: Memblaze Technology Co.,Ltd
Product Name: PBlaze3
Flash Cell: MLC
Model: PB3LM1.20T01
Form Factor: Half length, half height
Serial Number: MH13240245
Raw Capacity: 1.68TB
Available Capacity: 1.20TB
Maximum Capacity: 1.20TB
Max Write Capacity: 8.38PB
Already Read Capacity: 336.11TB
Already Write Capacity: 262.27TB
Write Amplification: 1.74
Device Information
------------------------------------------------------------------------------------------------------------------------------------------------------
Name: /dev/memdiska
Driver: 01.05.4083RTM (Compiled on Feb 10 2015 11:30:45), 355.01kB
Firmware: 01.05.4073RTM (Compiled on Jul 24 2014 16:01:15), 12.58MB
Initialization Counts: 3
Starup Counts: 34
PCIe Link: x8(Gen2)
Status: Online
Read only: No
Alert Code: 0
Device Remain Life: |>>------------------| (94.57%)
Performance Information
------------------------------------------------------------------------------------------------------------------------------------------------------
<6.96us 6.96us~27.8us 27.8us~111.3us 111.3us~445.2us 445.2us~1.78ms 1.78ms~7.12ms 7.12ms~28.5ms >28.5mVerify the Performance
- Use FIO for IOPS and Bandwidth tests
- Random Read 4k IOPS: ~750k IOPS
- Random Write 4k IOPS: ~250k IOPS
- Sequential Read 1m BW: ~2500MB/sec
- Sequential Write 1m BW: ~1800MB/sec
RandRead IOPS Test
[root@haswell9 ~]# fio --name=readiops --filename=/dev/memdiska --direct=1 --rw=randread --bs=4096 --numjobs=8 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
readiops: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
readiops: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.0.13
Starting 8 processes
Jobs: 8 (f=8): [rrrrrrrr] [100.0% done] [2755M/0K/0K /s] [705K/0 /0 iops] [eta 00m:00s]
readiops: (groupid=0, jobs=8): err= 0: pid=13212: Tue Feb 17 12:56:24 2015
read : io=870175MB, bw=2900.6MB/s, iops=742545 , runt=300001msec
slat (usec): min=18 , max=281509 , avg=99.04, stdev=227.66
clat (usec): min=1 , max=281668 , avg=242.03, stdev=365.13
lat (usec): min=38 , max=365377 , avg=341.34, stdev=451.37
clat percentiles (usec):
| 1.00th=[ 1], 5.00th=[ 45], 10.00th=[ 79], 20.00th=[ 126],
| 30.00th=[ 163], 40.00th=[ 191], 50.00th=[ 213], 60.00th=[ 235],
| 70.00th=[ 262], 80.00th=[ 298], 90.00th=[ 390], 95.00th=[ 524],
| 99.00th=[ 996], 99.50th=[ 1368], 99.90th=[ 2832], 99.95th=[ 3888],
| 99.99th=[ 8512]
bw (KB/s) : min= 12, max=486656, per=12.48%, avg=370822.29, stdev=33550.30
lat (usec) : 2=1.13%, 4=1.74%, 10=0.15%, 20=0.32%, 50=2.13%
lat (usec) : 100=8.67%, 250=51.90%, 500=28.43%, 750=3.66%, 1000=0.88%
lat (msec) : 2=0.78%, 4=0.17%, 10=0.04%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=3.54%, sys=64.44%, ctx=66465371, majf=0, minf=470
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=101.6%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=222764496/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=870175MB, aggrb=2900.6MB/s, minb=2900.6MB/s, maxb=2900.6MB/s, mint=300001msec, maxt=300001msec
Disk stats (read/write):
memdiska: ios=226196097/0, merge=0/0, ticks=41615182/0, in_queue=45701588, util=100.00%RandWrite IOPS Test
[root@haswell9 ~]# fio --name=readiops --filename=/dev/memdiska --direct=1 --rw=randwrite --bs=4096 --numjobs=8 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
readiops: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
readiops: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.0.13
Starting 8 processes
Jobs: 8 (f=8): [wwwwwwww] [100.0% done] [0K/957.4M/0K /s] [0 /245K/0 iops] [eta 00m:00s]
readiops: (groupid=0, jobs=8): err= 0: pid=14917: Tue Feb 17 13:23:04 2015
write: io=293126MB, bw=977.85MB/s, iops=250132 , runt=300001msec
slat (usec): min=16 , max=15637 , avg=68.77, stdev=108.63
clat (usec): min=1 , max=13286 , avg=943.26, stdev=831.27
lat (usec): min=36 , max=13366 , avg=1012.01, stdev=836.34
clat percentiles (usec):
| 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 48], 20.00th=[ 92],
| 30.00th=[ 171], 40.00th=[ 422], 50.00th=[ 844], 60.00th=[ 1192],
| 70.00th=[ 1432], 80.00th=[ 1672], 90.00th=[ 2128], 95.00th=[ 2448],
| 99.00th=[ 2928], 99.50th=[ 3120], 99.90th=[ 3632], 99.95th=[ 4016],
| 99.99th=[ 7136]
bw (KB/s) : min= 12, max=148480, per=12.49%, avg=124954.81, stdev=7908.66
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=10.75%
lat (usec) : 100=10.91%, 250=12.59%, 500=7.83%, 750=5.90%, 1000=5.93%
lat (msec) : 2=34.02%, 4=12.02%, 10=0.05%, 20=0.01%
cpu : usr=3.05%, sys=16.11%, ctx=10592839, majf=0, minf=210
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=101.7%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=75039990/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=293126MB, aggrb=977.85MB/s, minb=977.85MB/s, maxb=977.85MB/s, mint=300001msec, maxt=300001msec
Disk stats (read/write):
memdiska: ios=138/76294126, merge=0/0, ticks=7/49464252, in_queue=49796768, util=99.69%Sequential Read Perf
[root@haswell9 ~]# fio --name=read --filename=/dev/memdiska --direct=1 --rw=read --bs=1m --numjobs=8 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
read: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
...
read: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
fio-2.0.13
Starting 8 processes
Jobs: 8 (f=8): [RRRRRRRR] [100.0% done] [2784M/0K/0K /s] [2784 /0 /0 iops] [eta 00m:00s]
read: (groupid=0, jobs=8): err= 0: pid=16467: Tue Feb 17 13:48:25 2015
read : io=725272MB, bw=2416.1MB/s, iops=2416 , runt=300082msec
slat (usec): min=885 , max=14012 , avg=1223.93, stdev=318.84
clat (msec): min=77 , max=119 , avg=104.71, stdev= 8.27
lat (msec): min=78 , max=120 , avg=105.93, stdev= 8.29
clat percentiles (msec):
| 1.00th=[ 86], 5.00th=[ 89], 10.00th=[ 91], 20.00th=[ 94],
| 30.00th=[ 108], 40.00th=[ 109], 50.00th=[ 110], 60.00th=[ 110],
| 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 112], 95.00th=[ 112],
| 99.00th=[ 113], 99.50th=[ 114], 99.90th=[ 115], 99.95th=[ 115],
| 99.99th=[ 116]
bw (KB/s) : min= 3208, max=364088, per=12.50%, avg=309472.98, stdev=28099.73
lat (msec) : 100=25.44%, 250=74.57%
cpu : usr=0.03%, sys=3.56%, ctx=716058, majf=0, minf=4438
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=101.6%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=725024/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=725272MB, aggrb=2416.1MB/s, minb=2416.1MB/s, maxb=2416.1MB/s, mint=300082msec, maxt=300082msec
Disk stats (read/write):
memdiska: ios=2208989/0, merge=0/0, ticks=218864223/0, in_queue=218903348, util=100.00%Sequential Write Perf
- Write Perf
[root@haswell9 ~]# fio --name=write --filename=/dev/memdiska --direct=1 --rw=write --bs=1m --numjobs=8 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
write: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
...
write: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
fio-2.0.13
Starting 8 processes
Jobs: 8 (f=8): [WWWWWWWW] [100.0% done] [0K/1744M/0K /s] [0 /1744 /0 iops] [eta 00m:00s]
write: (groupid=0, jobs=8): err= 0: pid=15771: Tue Feb 17 13:36:19 2015
write: io=538440MB, bw=1794.9MB/s, iops=1793 , runt=300120msec
slat (msec): min=1 , max=15 , avg= 3.65, stdev= 1.50
clat (msec): min=79 , max=211 , avg=136.91, stdev=21.11
lat (msec): min=81 , max=215 , avg=140.56, stdev=21.05
clat percentiles (msec):
| 1.00th=[ 102], 5.00th=[ 106], 10.00th=[ 108], 20.00th=[ 116],
| 30.00th=[ 126], 40.00th=[ 130], 50.00th=[ 139], 60.00th=[ 147],
| 70.00th=[ 149], 80.00th=[ 151], 90.00th=[ 165], 95.00th=[ 172],
| 99.00th=[ 198], 99.50th=[ 202], 99.90th=[ 206], 99.95th=[ 210],
| 99.99th=[ 212]
bw (KB/s) : min= 3206, max=340045, per=12.48%, avg=229281.64, stdev=33806.76
lat (msec) : 100=0.75%, 250=99.28%
cpu : usr=5.85%, sys=2.77%, ctx=598469, majf=0, minf=214
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=101.9%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=538192/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=538440MB, aggrb=1794.9MB/s, minb=1794.9MB/s, maxb=1794.9MB/s, mint=300120msec, maxt=300120msec
Disk stats (read/write):
memdiska: ios=59/1646200, merge=0/0, ticks=91/218543271, in_queue=218569756, util=100.00%RAID0 on two cards
- Lets check the perf of RAID0 on two cards:
- Setup the RAID array:
mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/memdiska /dev/memdiskb- Perf Summary
- RAID0 Seqential Read 1m Perf: ~4650MB/sec
RAID0 Sequential Read 1m Perf
[root@haswell9 ~]# fio --name=read --filename=/dev/md0 --direct=1 --rw=read --bs=1m --numjobs=8 --iodepth=32 --direct=1 --iodepth_batch=16 --iodepth_batch_complete=16 --runtime=300 --ramp_time=5 --norandommap --time_based --ioengine=libaio --group_reporting
read: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
...
read: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
fio-2.0.13
Starting 8 processes
Jobs: 8 (f=8): [RRRRRRRR] [100.0% done] [5419M/0K/0K /s] [5418 /0 /0 iops] [eta 00m:00s]
read: (groupid=0, jobs=8): err= 0: pid=18447: Tue Feb 17 14:22:08 2015
read : io=1360.4GB, bw=4641.6MB/s, iops=4640 , runt=300047msec
slat (usec): min=930 , max=15193 , avg=1685.61, stdev=219.85
clat (msec): min=28 , max=130 , avg=53.47, stdev= 5.78
lat (msec): min=30 , max=131 , avg=55.15, stdev= 5.76
clat percentiles (usec):
| 1.00th=[42240], 5.00th=[45312], 10.00th=[45824], 20.00th=[45824],
| 30.00th=[52480], 40.00th=[52992], 50.00th=[53504], 60.00th=[54016],
| 70.00th=[55040], 80.00th=[57088], 90.00th=[60160], 95.00th=[63232],
| 99.00th=[69120], 99.50th=[72192], 99.90th=[77312], 99.95th=[82432],
| 99.99th=[97792]
bw (KB/s) : min= 3246, max=759104, per=12.49%, avg=593734.49, stdev=62875.98
lat (msec) : 50=22.86%, 100=77.14%, 250=0.01%
cpu : usr=0.04%, sys=7.39%, ctx=1359590, majf=0, minf=4430
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=101.7%, >=64=0.0%
submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=1392432/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=1360.4GB, aggrb=4641.6MB/s, minb=4641.6MB/s, maxb=4641.6MB/s, mint=300047msec, maxt=300047msec
Disk stats (read/write):
md0: ios=5662110/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2831136/0, aggrmerge=0/0, aggrticks=74362166/0, aggrin_queue=74366196, aggrutil=97.91%
memdiska: ios=2831136/0, merge=0/0, ticks=71890172/0, in_queue=71892101, util=97.91%
memdiskb: ios=2831136/0, merge=0/0, ticks=76834160/0, in_queue=76840291, util=93.69%