1. B ENCHMARK ON D ELL 2950+MD1000 ATLAS Tier2/Tier3 workshop Wenjing wu AGLT2 / University of Michigan 2008/05/27

2. D ELL 2950+4 MD1000 2

3. CURRENT SETUP

4. 2950 H ARDWARE EQUIPMENT Chassis Model: PowerEdge 2950 2 CPUS: Quad core, Intel Xeon CPU E5335@2.00GHz Model 15 Stepping 11 Memory : 16GB DDR II SDRAM, Memory Speed: 667 MHz NIC : Broadcom NetXtreme II BCM5708 Gigabit Ethernet Myricom 10G-PCIE-8A-C Raid controllers PERC 5/E Adapter Version 5.1.1-0040 (Slot 1 PCI-e 8x) PERC 5/E Adapter Version 5.1.1-0040 (Slot 2 PCI-e 4x) PERC 6/E Adapter Firmware version 6.0.2-0002 (Slot 1 PCI-e 8x) (extra 700$) PERC 6/E Adapter Firmware version 6.0.2-0002(Slot 2 PCI-e 4x) (extra 700$) Storage Enclosures 4 MD1000 (each has15 SATA-II 750GB disks)

5. 2950 S OFTWARE EQUIPMENT OS Scientific Linux CERN SLC release 4.5 (Beryllium) Kernel version: 2.6.20-20UL3smp (current 2.6.20-20UL5smp ) Version Report BIOS Version : 1.5.1 (current 2.2.6) BMC Version : 1.33 (current 2.0.5) DRAC 5 Version : 1.14 (current 1.33)

6. B ENCHMARK T OOL Benchmark tool: iozone ( iozone-3.279-1.el4.rf.x86_64 ) Raid configuration tool: omconfig ( srvadmin-omacore- 5.2.0-460.i386 ) Soft Raid: mdadm ( mdadm-2.6.1-4.x86_64 )

7. M ETRICS OF B ENCHMARK Controller Level (both perc5/perc6) raid setup (R0, R5,R50,R6,R60) Read and write policy (ra, ara,nra, wb, wt, fwb) Threshold of both Controllers Stripe size (8KB,16KB,32KB,64KB, 128KB, 256KB,512Kb,1024KB) Perc5 support maximum 128KB stripe size, perc6 support maximum 1024KB stripe size Kernel tuning (2.6.20-20UL3smp) read Ahead size Request queue length IO scheduler File System tuning (xfs) inode size su/sw size internal/external log device

8. G ENERAL PRINCIPLE FOR B ENCHMARK There are various factors which would impact the benchmark result, to measure one, we are trying to fix the other affecting factors on a best value we have got or we anticipate.. We need to benchmark different IO patterns (sequence read/write random read/write/mix workload) In all, we need a benchmark for all best options for our Dell2950.

9. C ONTROLLER L EVEL raid setup (R5,R50,R6,R60) Read and write policy (ra, ara,nra, wb, wt, fwb) Threshold of Controller(perc5/perc6) Stripe size (8KB,16KB,32KB,64KB, 128KB, 256KB,512Kb,1024KB) Perc5 support maximum 128KB stripe size, perc6 support maximum 1024KB stripe size

10. P ERC 5 VS PERC 6 System setup: Controller=perc6/perc5 PCI slots= both pci express x4 and x8 raid=r60/r6/r50 stripe size =128KB read=ra, write=wb OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB, multiple threads Measure: perc 5/6

11. READ

12. WRITE

13. R AID SETUP System setup: Controller=perc5 /perc6 PCI slots= both pci express x4 and x8 stripe size =128KB OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB, multiple threads Measure: different raid (r5, r50,r6,r60)

14. W RITE

15. S OFT RAID ON PERC 5 Soft raid 0 over 2 r5: Soft raid stripe size should be the same as the hard raid5 stripe size(128KB) Soft raid 0 over 2 r50: Soft raid stripe size should be the same as the hard raid5 stripe size(128KB)

16. WRITE

17. R EAD

19. R EAD AND W RITE POLICY System setup: Controller=perc5 PCI slots= both pci express x4 and x8 raid=r50 stripe size =128KB OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB different record size Measure: different policies (ra, nra,ara, wb,wt,fwb)

20. W RITE

21. R EAD

22. P ERC 5 THRESHOLD System setup: Controller=perc5 Pci slots= pci express x8 raid=r0 stripe size =128KB read=ra, write=wb OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB Measure single controller with different number of disks.(4-30disks)

23. P ERC 5 THRESHOLD

24. P ERC 6 THRESHOLD System setup: Controller=perc6 Pci slots= pci express x8 raid=r60 stripe size =512KB read=ra, write=wb OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=512 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB Measure single controller with different number of disks.(8, 12,24,30,45)

25. P ERC 6 THRESHOLD

26. S TRIPE SIZE System setup: Controller=perc6 PCI slots= both pci express x4 and x8 raid=r60 stripe size =(64,128,256,512,1024)KB read=ra, write=wb OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=512 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB, multiple threads Measure: different stripe size (64,128,256,512,1024)KB

27. R60 – STRIPE SIZE

28. R60- STRIPE SIZE

29. K ERNEL TUNING read Ahead size Request queue length IO scheduler

30. R EAD A HEAD SIZE System setup: Controller=perc5 PCI slots= both pci express x4 and x8 raid=r50 stripe size =128KB read=ra, write=wb OS kernel= 2.6.20-20UL3smp nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB, Measure: different readAhead size

31. R EAD

32. R EQUEST Q UEUE LENGTH System setup: Controller=perc6 PCI slots= both pci express x4 and x8 raid=r60 stripe size =128KB read=ra, write=wb OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB, multiple threads Measure: different request queue length

33. R EAD

34. W RITE

35. IO SCHEDULER System setup: Controller=perc6 PCI slots= both pci express x4 and x8 raid=r50 stripe size =128KB read=ra, write=wb OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=512 queue_depth=128 file system options: su=0, sw=0 isize=256, bsize=4096, log=internal bsize=4096 iozone options: filesize=32GB, ram size=16GB record size=512KB, multiple threads Measure: different scheduler

36. READ

37. WRITE

38. R ANDOM READ

39. F ILESYSTEM TUNING inode size su/sw size internal/external log device

40. F ILE S YSTME System setup: Controller=perc5 Raid=r50 PCI slots= both pci express x4 and x8 stripe size =128KB OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 isize=256, bsize=4096, dd options: filesize=10GB, ram size=320MB record size=1MB Measure: internal or external log device for xfs

41. W RITE

42. R EAD

43. X FS INODE SIZE System setup: Controller=perc5 Raid=r50 PCI slots= both pci express x4 and x8 stripe size =128KB OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: su=0, sw=0 bsize=4096, Internal Log, isize=256, bsize=4096 dd options: filesize=10GB, ram size=320MB record size=1MB Measure: xfs inode size

44. X FS INODE SIZE

45. X FS SU / SW SIZE System setup: Controller=perc5 Raid=r50 PCI slots= both pci express x4 and x8 stripe size =128KB OS kernel= 2.6.20-20UL3smp readAhead size=10240Blocks=5MB nr_queue=128 queue_depth=128 IO_scheduler=deadline file system options: isize=256KB, bsize=4096KB, Internal Log, isize=256KB, bsize=4096KB iozone options: filesize=10GB, ram size=320MB record size=1MB Measure: xfs sw/su size

46. S U / SW SIZE

47. O UR SETUP N OW System setup: Controller=perc56 Raid=r60 PCI slots= both pci express x4 and x8 stripe size =512KB OS kernel= 2.6.20-20UL5smp Kernel options: readAhead size=10240Blocks=5MB nr_queue=512 queue_depth=128 IO_scheduler=deadline file system options: isize=256KB, bsize=4096KB, Internal Log, isize=256KB, bsize=4096KB

48. O UR PERFORMANCE NOW Single read=670MB/s Aggregate read=1500MB/s (threads>=2) Even with 40 concurrent readers, it can still achieve 1200MB/s.. Single write=320MB/s Aggregate write=680MB/s (threads>=2) This is not the best IO, r60 with stripe size 128KB can achieve 760MB/s of single read and single write performs almost the same. For a production system, we focus more on the aggregate performance…

49. O NGOING PROJECT CITI people of UM are doing: Disk-to-disk transfer over 10 GbE Deliverables Monthly report on performance tests, server configurations, kernel tuning, and kernel bottlenecks Final report on performance tests, server configurations, kernel tuning, and kernel bottlenecks UltraLight kernel Deliverables Tuned and tested UltraLight kernel with full feature set Current 10GbE NIC drivers Current storage drivers Tuned for WAN data movement Web100 patches Other patches for performance, security, and stability Release document and web page updates for UltraLight kernel http://www.ultralight.org/web-site/ultralight/workgroups/network/Kernel/kernel.html Recommend sustainable options for the Ultralight kernel in the near and intermediate term

50. O NGOING PROJECT ( CONT ) QoS experiments Deliverable Document throughput performance with and without QoS in the face of competing traffic

51. M ORE INFORMATION AGLT2 IO benchmark page: https://hep.pa.msu.edu/twiki/bin/view/AGLT2/IOTe stOnRaidSystems References: http://www.makarevitch.com/rant/3ware/ http://insights.oetiker.ch/linux/raidoptimization.ht ml