Well it really helps speed things up if you enable pcie2 and not run tests at pcie1 speeds.
Also if you are a plonker and forget to edit /boot/hw_intfc.conf from #intfc:dtoverlay=pcie-gen2 to intfc:dtoverlay=pcie-gen2 you will be running on pcie-gen1
RAID 10
Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
102400 4 11719 15447 55220 53720 25421 12773
102400 16 39410 54840 139482 145128 81258 43792
102400 512 228002 220126 334104 339660 265930 225507
102400 1024 244376 243730 451377 462467 397566 258481
102400 16384 270088 304411 597462 610057 615669 297855
RAID 5
Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
102400 4 6133 6251 47505 46013 25046 8190
102400 16 17103 17134 113272 133606 79753 20420
102400 512 61418 50852 241860 246467 244030 58031
102400 1024 79325 73325 363343 359830 361882 83655
102400 16384 127548 124702 625256 642094 650407 136680
RAID 1
Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
102400 4 23713 29698 45608 45983 23657 30381
102400 16 79205 82546 138060 144557 82126 93921
102400 512 212859 221943 307613 304036 259783 179355
102400 1024 235985 243783 366101 369935 317354 198861
102400 16384 289036 290279 410520 398875 399868 295329
RAID 0
Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
102400 4 33519 47927 52701 51023 26700 46382
102400 16 105763 132604 138080 155514 87026 135111
102400 512 276220 320320 311343 294629 267624 335363
102400 1024 493565 522038 463105 470833 398584 522560
102400 16384 687516 701200 625733 623531 555318 681535
4 individual disk concurrent
Command line used: iozone -l 4 -u 4 -r 16k -s 512M -F /srv/dev-disk-by-label-sda/tmp1 /srv/dev-disk-by-label-sdb/tmp2 /srv/dev-disk-by-label-sdc/tmp3 /srv/dev-disk-by-label-sdd/tmp4
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Min process = 4
Max process = 4
Throughput test with 4 processes
Each process writes a 524288 kByte file in 16 kByte records
Children see throughput for 4 initial writers = 884590.91 kB/sec
Parent sees throughput for 4 initial writers = 701620.17 kB/sec
Min throughput per process = 195561.27 kB/sec
Max throughput per process = 234457.59 kB/sec
Avg throughput per process = 221147.73 kB/sec
Min xfer = 437344.00 kB
Children see throughput for 4 rewriters = 822771.77 kB/sec
Parent sees throughput for 4 rewriters = 701488.29 kB/sec
Min throughput per process = 180381.25 kB/sec
Max throughput per process = 232223.50 kB/sec
Avg throughput per process = 205692.94 kB/sec
Min xfer = 408720.00 kB
Children see throughput for 4 readers = 755252.30 kB/sec
Parent sees throughput for 4 readers = 753357.02 kB/sec
Min throughput per process = 169105.11 kB/sec
Max throughput per process = 198976.81 kB/sec
Avg throughput per process = 188813.07 kB/sec
Min xfer = 445664.00 kB
Children see throughput for 4 re-readers = 753492.39 kB/sec
Parent sees throughput for 4 re-readers = 750353.64 kB/sec
Min throughput per process = 160626.64 kB/sec
Max throughput per process = 201223.11 kB/sec
Avg throughput per process = 188373.10 kB/sec
Min xfer = 418528.00 kB
Children see throughput for 4 reverse readers = 780261.86 kB/sec
Parent sees throughput for 4 reverse readers = 778761.55 kB/sec
Min throughput per process = 58371.02 kB/sec
Max throughput per process = 254657.08 kB/sec
Avg throughput per process = 195065.47 kB/sec
Min xfer = 120192.00 kB
Children see throughput for 4 stride readers = 317923.62 kB/sec
Parent sees throughput for 4 stride readers = 316905.36 kB/sec
Min throughput per process = 63171.63 kB/sec
Max throughput per process = 98114.27 kB/sec
Avg throughput per process = 79480.91 kB/sec
Min xfer = 337600.00 kB
Children see throughput for 4 random readers = 798898.78 kB/sec
Parent sees throughput for 4 random readers = 794905.95 kB/sec
Min throughput per process = 57059.89 kB/sec
Max throughput per process = 391248.59 kB/sec
Avg throughput per process = 199724.70 kB/sec
Min xfer = 76480.00 kB
Children see throughput for 4 mixed workload = 647158.06 kB/sec
Parent sees throughput for 4 mixed workload = 491223.65 kB/sec
Min throughput per process = 28319.04 kB/sec
Max throughput per process = 305288.75 kB/sec
Avg throughput per process = 161789.51 kB/sec
Min xfer = 48720.00 kB
Children see throughput for 4 random writers = 734947.98 kB/sec
Parent sees throughput for 4 random writers = 544531.66 kB/sec
Min throughput per process = 167241.00 kB/sec
Max throughput per process = 207134.38 kB/sec
Avg throughput per process = 183737.00 kB/sec
Min xfer = 424704.00 kB
Children see throughput for 4 pwrite writers = 879712.72 kB/sec
Parent sees throughput for 4 pwrite writers = 686621.58 kB/sec
Min throughput per process = 186624.69 kB/sec
Max throughput per process = 236047.30 kB/sec
Avg throughput per process = 219928.18 kB/sec
Min xfer = 415856.00 kB
Children see throughput for 4 pread readers = 777243.34 kB/sec
Parent sees throughput for 4 pread readers = 773302.81 kB/sec
Min throughput per process = 184983.08 kB/sec
Max throughput per process = 203392.77 kB/sec
Avg throughput per process = 194310.84 kB/sec
Min xfer = 476896.00 kB
Children see throughput for 4 fwriters = 820877.50 kB/sec
Parent sees throughput for 4 fwriters = 693823.17 kB/sec
Min throughput per process = 194228.28 kB/sec
Max throughput per process = 217311.28 kB/sec
Avg throughput per process = 205219.38 kB/sec
Min xfer = 524288.00 kB
Children see throughput for 4 freaders = 1924029.62 kB/sec
Parent sees throughput for 4 freaders = 1071393.99 kB/sec
Min throughput per process = 268087.50 kB/sec
Max throughput per process = 970331.94 kB/sec
Avg throughput per process = 481007.41 kB/sec
Min xfer = 524288.00 kB
Single disk sda reference
Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
102400 4 35191 45728 56689 53307 27889 48508
102400 16 104379 122405 154385 157484 88670 113964
102400 512 315788 347042 351932 348604 271399 288430
102400 1024 358399 366194 388893 379453 338470 369888
102400 16384 353154 443256 425396 422384 410580 444530