1. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Experience and Lessons learnt from running High Availability Databases on Network Attached Storage Ruben Gaspar Manuel Guijarro Nilo Segura IT/DES

2. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 2 Outline Background Concepts: RAC, NAS, Bonding Why NAS system for DB Infrastructure Topology NAS & Server Node Configuration Performance Maintenance & Monitoring Conclusions

3. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 3 Background Databases running on Sun SPARC servers & storage Mix of JBOD and “Intelligent” RAID hw & sw –ex Sun StorEdge (3310, 3510) –Veritas Volume Manager Project to migrate to Linux and Oracle Real Application Cluster –Reliable/Certified/Supported Shared storage required Storage Technology ? –NAS & FC

4. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 4 ORACLE RAC Databases

5. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 5 What is a NAS ? Network-attached storage (NAS) is a Data Storage technology that provides centralized data access and storage to heterogeneous network clients. Several file transfer protocols supported (NFS, iSCSI, etc) Operating System on the NAS unit is specialized on all aspects of data management (i.e. scrubbing,…) By contrast to a SAN (Storage Area Network, block access protocols i.e. FCP), remote devices do not appear as locally attached.

6. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 6 Why NAS for DB infrastructure ? To ease file sharing needed for Oracle RAC –Database sees the NAS box as just NFS mounted filesystem –It is much simpler than using raw devices –The complexity is managed by the appliance instead of the server To use NAS specific features: Snapshots, RAID, Failover based on NAS partnership, Dynamic re-Sizing of file systems, Remote Sync to offsite NAS, etc Ethernet technology much simpler to manage than Fiber Channel –CERN IT/CS favors the use of Ethernet over FC To ease File Server Management: –automatic failure reporting to vendor –integrated packages to monitor/tune the box To simplify administration of Data Base Running since beginning 2007 in production databases

7. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 7 BONDING Bonding (AKA Trunking): Aggregate multiple network interfaces into a single logical bonded interface. Two flavors are used: –load balancing (between switches and NAS) –active/backup (host and switches) It provides redundancy and reliability to the network link NAS system provides bonding under the concept of VIF (Virtual Interface)

8. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 8 Topology

9. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 9 CASTOR cluster cabling dbnasc101 dbnasc102 Disk shelf 1 Disk shelf 2

10. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 10

11. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 11 DES cluster cabling dbnasd101 Disk shelf: 1 dbnasd102 Disk shelf: 2 Disk shelf: 3 Disk shelf: 4

12. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 12 NAS Configuration Netapp 3040: NVRAM 512MB, RAM 4GB, 2 AMD 2.4Ghz opteron CPUs OS on the filer Data Ontap 7.2.4 64 bits Disk Shelves 13+1 disks in each (all in a single disk aggregate) RAID_DP (Double Parity): Improved RAID 6 implementation Volumes implemented by FlexVol technology Automatic resize of volumes based on thresholds SnapShots –Used to create cold backups of the databases before an upgrade.

13. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 13 NAS Configuration Aggregate (13 disks + 1 spare disk) dbnasc101> sysconfig -r Aggregate dbdskc101 (online, raid_dp) (block checksums) Plex /dbdskc101/plex0 (online, normal, active) RAID group /dbdskc101/plex0/rg0 (normal) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------- ------ dparity 0a.16 0a 1 0 FC:A - FCAL 10000 136000/278528000 137104/280790184 parity 0a.17 0a 1 1 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0a.18 0a 1 2 FC:A - FCAL 10000 136000/278528000 137104/280790184 –Raid_dp: two failing disks protection

14. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 14 NAS Configuration Flexible volumes vol create volname aggregate 1g vol autosize volname -m 20g -i 1g on vol options volname nosnap on vol options volname no_atime_update on Redundancy accessing disks in the storage system, dual loop: dbnasc101> storage show disk -p PRIMARY PORT SECONDARY PORT SHELF BAY ------- ---- --------- ---- --------- 0d.32 B 0b.32 A 2 0 0b.33 A 0d.33 B 2 1

15. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 15 Server Node Configuration Quattor based installation and configuration covering: –Basic RHE4 x86_64 installation (Oracle Support condition) –NCM Components used to configure network bonding, tsm, tdpo, Sendmail, Squid (Autosupport),Oracle software installation CRS and RDBMS via rpms, RAC ncm… –All hardware components definition in CDB (including NAS boxes and disk shelves) Lemon based monitoring for all Server node resources Always using CERN’s CC infrastructure (Installations performed by SysAdmin support team, TSM, computer center operators, CDB…)

16. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 16 Server Node Configuration: standard DB set- up Four volumes created, critical files split among volumes (RAC databases) –/ORA/dbs00: logging and archive redo log files and a copy of the control file –/ORA/dbs02: a copy of the control file and the voting disk –/ORA/dbs03: redo logs, spfile and datafiles, and a copy of the control file, the voting disk and registry file –/ORA/dbs04: redo logs, a copy of the control file, the voting disk and the registry - different aggregate (i.e. different disk shelf and different NAS filer). Accessed via NFS from the sever machines Mount options for Linux rw,bg,hard,nointr,tcp,vers=3,actimeo=0,timeo=600,rsize=32768, wsize=32768 DATABASES in production on NAS. RAC instances in production: –Castor Name Server and Stagers, Lemon, Remedy, AIS Services, LHC (Logging, Measurements and Control)

17. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 17 Oracle Software Installation CRS, RDBMS are packaged in RPMs. (Oracle Universal Installer + Cloning) A Quattor component configures RAC accordingly and starts daemons successfully

18. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 18 Performance Oracle uses direct I/O over NFS –avoids external caching in the OS page cache –more performing (for typical Oracle I/O database workloads) than buffered I/O (double as much) Use of Asynchronous I/O over NFS requires Red Hat 4.0 upgrade 3 –Oracle support notes incorrectly states that Upgrade 4 is required To enable both direct/asynch we need to do –alter system set FILESYSTEMIO_OPTIONS = setall scope=spfile sid='*'; Measurements –Stress test: 8KB read operations by multiple threads in a set of aggregates (with a total of 12 data disks): we reach the physical limit of the disk max of ~ 150 I/O operations per second and per disk.

19. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 19 Performance

20. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 20 Performance measured by Netapp & Oracle

21. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 21 Performance

22. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 22 Maintenance and Monitoring All interventions done with NO downtime: –Some failed Disks replacement –NAS Filer OS update Data ONTAP upgrade to 7.2.4 (rolling forward) –RAC member full automatic reinstallation –PSU replacement (both in NAS filer and Server nodes) –Ethernet Network cable replacement –Disk firmware updates Several Hardware failures successfully tested –Powering off: Network switche(s), NAS filer, RAC member,… –Unplugging: FC cables connecting NAS filer to Disk Shelf, Active bond members cable(s) of Private interconnect, VIF’s,.. Using Oracle Enterprise Manager monitoring (out of the box + for free) + autosupport via email

23. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 23 NetApp Plug-in

24. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 24 Conclusions NAS based infrastructure is easy to maintain All interventions so far done with 0 downtime Good Performance for Oracle RAC databases Configuration of Server Nodes automated via Quattor - CDB Ongoing migration of legacy db’s to NAS based infrastructure Strategic deployment platform –Reduces the number of Storage Technologies to be maintained –Avoids complexity of SAN based infrastructure (less dba work to maintain db, multipathing, Oracle ASM, etc). Oracle goes strongly into NFS –Starting with 11gR1, the Oracle kernel integrates an NFS driver that should boost NFS performance

25. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 25 QUESTIONS

26. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 26 Ontap Simulator: for free http://forums.netapp.com/conversation.asp?tid=1&vid=47289&thd=1 &cid=208 http://forums.netapp.com/conversation.asp?tid=1&vid=47289&thd=1 &cid=208 Minimizing media error disruption of RAID reconstructions: –Scrub process –Disk media error failure threshold – Predictive disk failure –Wafliron

27. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 27 NETApp in the market NetApp is the fastest growing company in the SAN market and continues to hold the leading share in both NAS and IP SAN/iSCSI markets (Source: IDC Worldwide Quarterly Disk Storage Systems Tracker Q3 2006 ) Company Grew Faster Than the FC SAN and iSCSI Markets; Demonstrates Continued Leadership in Capacity Share in NAS and iSCSI Markets (Source: IDC Worldwide Quarterly Disk Storage Systems Tracker Q4 2006 ) http://www.netapp.com

28. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t 28