1. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1 Exadata Software Maintenance IOUG 2013 Exadata Virtual Conference Doug Utzig Consulting Member of Technical Staff Oracle MAA Development

2. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 2 Agenda  Software Architecture Overview  Planning  Updating Exadata Software  Updating Grid Infrastructure and Database Software  Security Updates  Best Practices

3. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 3 Planned Maintenance Improvements 2009-2010 2011 2012-2013 Auto Cell firmware update Exadata bundle patch (BP) Out-of-place patching step-by-step Minimal Pack firmware update Cell rolling patch apply BP EM install DB+GI BP merge BP OPatch Auto install BP DG Standby-First install Minimal Pack Linux kernel update BP RAC rolling install BP OPlan IB Switch update w/ ILOM Linux upgrade step-by-step 11.2.0.2 upgrade step-by-step Oracle Advisor webcasts IOUG Exadata SIG webcast 11.2.0.3 upgrade step-by-step 11.2.0.3 Exachk upgrade readiness 11.2.0.3 QDPE 11.2.0.3 QFSDP EM 12c Out-of-place patching Linux dbserver ULN updates Exadata channel ISO Flash ESM removal Reduce Downtime Simplify Install Reduce Risk Transfer Knowledge Continual Improvements

4. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 4 Exadata Database Machine Database Grid – Exadata Database Server operating system (Linux or Solaris) – Grid Infrastructure and Database Storage Grid – Exadata Storage Server Software Networking – InfiniBand switch software Other: Ethernet switch, PDU, KVM Software Review

5. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 5 Planning

6. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 6 Planning  MOS 888828.1 – “The Patching Note” – Critical Issues and additional required patches – Guidelines for related products (OEM, DBFS, RM, GG, SAP) Latest Releases Software ComponentLatest ReleaseRecommended Minimum Exadata11.2.3.2.111.2.2.4.2 Grid Infrastructure and Database 11.2.0.3: 11.2.0.3.14 11.2.0.2: BP19 11.2.0.3: 11.2.0.3.8 (Jul 2012 QDPE) 11.2.0.2: BP17 InfiniBand Switch1.3.3-2

7. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 7 Software Delivery Quarterly Full Stack Download Patch (QFSDP)  Follow quarterly updates for planned maintenance  Contains latest recommended – Infrastructure  Exadata Storage Server  InfiniBand Switch – Database – SystemsManagement (EM)  Release aligned on PSU / CPU schedule |- Infrastructure/ | --ExadataStorageServer/ | -11.2.3.2.1/ | --InfiniBandSwitchSoftware/ | -1.3.3-2/ | --SunRackIIPDUMeteringUnitFirmware/ | -1.04/ |- Database/ | --11.2.0.3_QDPE_Jan2013/ | --OPatch/ | --OPlan/ |- SystemsManagement/ | --CloudControl12/ | --Agent/ | --WLS/ | --OMS/

8. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 8 Planned Maintenance Suggested Schedule FrequencyPlanned Maintenance – Software Updates 3-6 months Grid Infrastructure / Database quarterly patch (QDPE) Exadata software 1-2 years Grid Infrastructure / Database patch set InfiniBand switch software 2-4 years Grid Infrastructure / Database release (e.g. 11.1 to 11.2, or 11 to 12)

9. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 9 Software Maintenance Patching Tools System readiness - Exachk (MOS 1070954.1) ComponentSoftwarePatching Tool Storage ServerExadatapatchmgr Database Server ExadataLinux – YUM Solaris – PKG GI/DB QuarterlyPatchOPatch GI/DB PatchSetOUI and DBUA InfiniBand SwitchAllILOM

10. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 10 Updating Exadata Software Exadata Storage Server Exadata Database Server

11. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 11 Exadata Storage Server  Release contains all storage software / firmware  patchmgr controls install and rollback – One invocation patches all cells – Rollback to most recent successfully installed version MethodDowntimePatch install time RollingNoneUp to 2 hours per cell Non-RollingUp to 2 hours total Rules Install nothing but this patch on cells No unsupported configuration changes Will be lost next update May cause next update to FAIL

12. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 12 Exadata Database Server Operating System Two types of packages Oracle Linux 1. Exadata Core Packages (update required)  Coincides with Exadata Storage Server release  YUM update from Exadata ULN channel only (or the ISO)  Includes firmware Exadata Core Packages kernel* kexec-tools 2. Other Packages (--exclude Exadata Core Packages) (update optional)  Update existing Other Packages from ULN (ol5_x86_64_latest channel)  Install new Other Packages  e.g. backup agents, monitoring agents # yum -–exclude=kernel-* --exclude=… update

13. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 13 Updating Grid Infrastructure and Database Software

14. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 14 Grid Infrastructure and Database  OPatch installed and rolled back – Patches are cumulative – One invocation per dbnode  Methods to reduce downtime and risk Quarterly Patch Update README

15. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 15 Grid Infrastructure and Database Benefit for Quarterly Patch Update 1. Clone home (GI or DB) 2. OPatch cloned home 3. Switch current home to cloned home (rolling or non-rolling)  Enterprise Manager 12c and OPlan supported Out-of-Place (OOP) Patching Performed during normal business hours, ahead of “patching window” Patch inactive home to reduce risk

16. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 16 Grid Infrastructure and Database  Generates step-by-step patching instructions – Specific to your environment – 3 options – Like OPatch, always get the latest OPlan ApplyInstructions.html $ oplan generateApplySteps /tmp/13556724

17. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 17 Grid Infrastructure and Database Database Grid Grid Infrastructure Database  Apply Exadata Quarterly Database patches – GI and DB homes – Orchestrates across all database servers  Multiple modes supported – Out-of-Place or In-Place – Rolling or Non-Rolling Patching Exadata with EM EM Out-of-Place Patching - Recommended Analyze (Before patching window) 1.Stage patch and perform pre-apply check Prepare (Before patching window) 2.Clone Oracle homes (GI and Database) 3.Apply patch to cloned Oracle homes Deploy (Switch, Rolling or Non-Rolling) 4.Switch GI to cloned Oracle home 5.Switch databases to cloned Oracle home 6.Apply SQL portion of the patch

18. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 18 Grid Infrastructure and Database Patch Sets Follow Exadata validated steps 11.2.0.3 - MOS 1373255.1 11.2.0.2 - MOS 1315926.1 Storage Server compatibility DatabaseRequired Storage 11.2.0.311.2.2.4 or later 11.2.0.211.2.2 or later Grid Infrastructure upgrade Rolling Database upgrade Requires downtime Reduce downtime and risk (Transient) Logical standby Golden Gate

19. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 19 Security Updates

20. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 20 Security Updates Exadata Storage ServerExadata Database Server - Oracle Linux  All software (“Closed” system)  Update to later Exadata release  Exadata Core Packages - Review MOS 1405320.1  Update to later Exadata release  Open Service Request for CVEs not covered  Other packages  Update typically from ol5_x86_64_latest  -- exclude Exadata Core Packages Database Server Grid Infrastructure and Database  Quarterly Database Patch for Exadata  Contains concurrent PSU/CPU content

21. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 21 Best Practices

22. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 22 Methods Rolling versus Non-Rolling  Patch Non-Rolling if maintenance window allows  Patch Rolling during low workload  Mix-and-match Exadata Storage Server Rolling Patching  Reduce disk failure risk w/ high redundancy (or DG) Database Server (Exadata & GI/DB) Rolling Patching  Use Automatic Workload Management RollingNon-Rolling Lowest Database Downtime Lowest Installation Time

23. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 23 Reduce Risk and Downtime with Data Guard  Reduce Planned Maintenance Risk and Downtime with Data Guard – Upgrade standby system and switchover Data Guard Independent Components (Upgrade Standby with no impact to Primary)  Exadata software (Storage and Database Servers)  Database Server Grid Infrastructure home  InfiniBand Switch software Dependent Components  Database Server Database home (MOS 1265700.1)  Data Guard Standby-First Installable qualified patches only

24. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 24 Consolidation and Large Environments Consolidation GI version >= DB version  Required to 4th digit, recommended to 5th digit Advance diskgroup compatible.rdbms with care  Once advanced cannot be reset Share database homes whenever possible  Extend /u01 size  Patch Out-of-Place  Move databases individually to new home Large Environments Mixed Exadata version supported  Storage servers and database servers may run different versions (recommended only during rolling upgrade) Split updates to multiple components across more than one maintenance window  In general – update up the stack 1.InfiniBand switches 2.Exadata Storage Servers 3.Database Servers OS / Firmware 4.Grid Infrastructure / Database

25. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 25 Avoid Common Pitfalls Exadata Database Server OS and Firmware  Test previous configuration changes to avoid latent patching failures  Backup OS using validated method (e.g. dbserver_backup.sh) Database Server Grid Infrastructure and Database  Follow Exadata step-by-step MOS note for patch set upgrade  CV_DESTLOC avoids cluvfy conflicts during concurrent opatch auto  Ensure unzipped bundle patch location has proper permission  Shutdown software as documented Exadata Storage Server  Do NOT make unsupported configuration changes General  Stay up-to-date  Familiarize with doc and known issues  Test and rehearse  Run latest Exachk regularly  Fix Exachk warnings and alerts before patching

26. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 26 Oracle Support for Exadata  24/7 support  Specialized Engineered Systems Support Team  2-hour onsite response to hardware issues 1  New Updates and Upgrades for Database, Server, Storage, and OS software  My Oracle Support proactive support portal  "Phone home" automated service requests (ASR) Complete. Integrated. Proactive. High Availability. No Additional Cost. ORACLE PLATINUM SERVICES  Higher support levels for the complete Oracle stack – Includes higher support levels for Database software  Proactive remote monitoring for faults  Industry leading service level response times: – 5 Minute Fault Notification – 15 Minute Restoration or Escalation to Development – 30 Minute Joint Debugging with Development  Oracle Engineers perform quarterly patching and updates Available for certified configurations on Exadatacertified configurations 1 Covered system must be within an Oracle two-hour service area to receive two-hour response as a standard service.

27. Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 27