Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1 Exadata Software Maintenance IOUG 2013 Exadata Virtual Conference Doug Utzig Consulting Member of Technical Staff Oracle MAA Development

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 2 Agenda  Software Architecture Overview  Planning  Updating Exadata Software  Updating Grid Infrastructure and Database Software  Security Updates  Best Practices

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 3 Planned Maintenance Improvements Auto Cell firmware update Exadata bundle patch (BP) Out-of-place patching step-by-step Minimal Pack firmware update Cell rolling patch apply BP EM install DB+GI BP merge BP OPatch Auto install BP DG Standby-First install Minimal Pack Linux kernel update BP RAC rolling install BP OPlan IB Switch update w/ ILOM Linux upgrade step-by-step upgrade step-by-step Oracle Advisor webcasts IOUG Exadata SIG webcast upgrade step-by-step Exachk upgrade readiness QDPE QFSDP EM 12c Out-of-place patching Linux dbserver ULN updates Exadata channel ISO Flash ESM removal Reduce Downtime Simplify Install Reduce Risk Transfer Knowledge Continual Improvements

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 4 Exadata Database Machine Database Grid – Exadata Database Server operating system (Linux or Solaris) – Grid Infrastructure and Database Storage Grid – Exadata Storage Server Software Networking – InfiniBand switch software Other: Ethernet switch, PDU, KVM Software Review

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 5 Planning

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 6 Planning  MOS – “The Patching Note” – Critical Issues and additional required patches – Guidelines for related products (OEM, DBFS, RM, GG, SAP) Latest Releases Software ComponentLatest ReleaseRecommended Minimum Exadata Grid Infrastructure and Database : : BP : (Jul 2012 QDPE) : BP17 InfiniBand Switch

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 7 Software Delivery Quarterly Full Stack Download Patch (QFSDP)  Follow quarterly updates for planned maintenance  Contains latest recommended – Infrastructure  Exadata Storage Server  InfiniBand Switch – Database – SystemsManagement (EM)  Release aligned on PSU / CPU schedule |- Infrastructure/ | --ExadataStorageServer/ | / | --InfiniBandSwitchSoftware/ | / | --SunRackIIPDUMeteringUnitFirmware/ | -1.04/ |- Database/ | _QDPE_Jan2013/ | --OPatch/ | --OPlan/ |- SystemsManagement/ | --CloudControl12/ | --Agent/ | --WLS/ | --OMS/

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 8 Planned Maintenance Suggested Schedule FrequencyPlanned Maintenance – Software Updates 3-6 months Grid Infrastructure / Database quarterly patch (QDPE) Exadata software 1-2 years Grid Infrastructure / Database patch set InfiniBand switch software 2-4 years Grid Infrastructure / Database release (e.g to 11.2, or 11 to 12)

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 9 Software Maintenance Patching Tools System readiness - Exachk (MOS ) ComponentSoftwarePatching Tool Storage ServerExadatapatchmgr Database Server ExadataLinux – YUM Solaris – PKG GI/DB QuarterlyPatchOPatch GI/DB PatchSetOUI and DBUA InfiniBand SwitchAllILOM

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 10 Updating Exadata Software Exadata Storage Server Exadata Database Server

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 11 Exadata Storage Server  Release contains all storage software / firmware  patchmgr controls install and rollback – One invocation patches all cells – Rollback to most recent successfully installed version MethodDowntimePatch install time RollingNoneUp to 2 hours per cell Non-RollingUp to 2 hours total Rules Install nothing but this patch on cells No unsupported configuration changes Will be lost next update May cause next update to FAIL

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 12 Exadata Database Server Operating System Two types of packages Oracle Linux 1. Exadata Core Packages (update required)  Coincides with Exadata Storage Server release  YUM update from Exadata ULN channel only (or the ISO)  Includes firmware Exadata Core Packages kernel* kexec-tools 2. Other Packages (--exclude Exadata Core Packages) (update optional)  Update existing Other Packages from ULN (ol5_x86_64_latest channel)  Install new Other Packages  e.g. backup agents, monitoring agents # yum -–exclude=kernel-* --exclude=… update

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 13 Updating Grid Infrastructure and Database Software

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 14 Grid Infrastructure and Database  OPatch installed and rolled back – Patches are cumulative – One invocation per dbnode  Methods to reduce downtime and risk Quarterly Patch Update README

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 15 Grid Infrastructure and Database Benefit for Quarterly Patch Update 1. Clone home (GI or DB) 2. OPatch cloned home 3. Switch current home to cloned home (rolling or non-rolling)  Enterprise Manager 12c and OPlan supported Out-of-Place (OOP) Patching Performed during normal business hours, ahead of “patching window” Patch inactive home to reduce risk

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 16 Grid Infrastructure and Database  Generates step-by-step patching instructions – Specific to your environment – 3 options – Like OPatch, always get the latest OPlan ApplyInstructions.html $ oplan generateApplySteps /tmp/

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 17 Grid Infrastructure and Database Database Grid Grid Infrastructure Database  Apply Exadata Quarterly Database patches – GI and DB homes – Orchestrates across all database servers  Multiple modes supported – Out-of-Place or In-Place – Rolling or Non-Rolling Patching Exadata with EM EM Out-of-Place Patching - Recommended Analyze (Before patching window) 1.Stage patch and perform pre-apply check Prepare (Before patching window) 2.Clone Oracle homes (GI and Database) 3.Apply patch to cloned Oracle homes Deploy (Switch, Rolling or Non-Rolling) 4.Switch GI to cloned Oracle home 5.Switch databases to cloned Oracle home 6.Apply SQL portion of the patch

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 18 Grid Infrastructure and Database Patch Sets Follow Exadata validated steps MOS MOS Storage Server compatibility DatabaseRequired Storage or later or later Grid Infrastructure upgrade Rolling Database upgrade Requires downtime Reduce downtime and risk (Transient) Logical standby Golden Gate

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 19 Security Updates

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 20 Security Updates Exadata Storage ServerExadata Database Server - Oracle Linux  All software (“Closed” system)  Update to later Exadata release  Exadata Core Packages - Review MOS  Update to later Exadata release  Open Service Request for CVEs not covered  Other packages  Update typically from ol5_x86_64_latest  -- exclude Exadata Core Packages Database Server Grid Infrastructure and Database  Quarterly Database Patch for Exadata  Contains concurrent PSU/CPU content

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 21 Best Practices

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 22 Methods Rolling versus Non-Rolling  Patch Non-Rolling if maintenance window allows  Patch Rolling during low workload  Mix-and-match Exadata Storage Server Rolling Patching  Reduce disk failure risk w/ high redundancy (or DG) Database Server (Exadata & GI/DB) Rolling Patching  Use Automatic Workload Management RollingNon-Rolling Lowest Database Downtime Lowest Installation Time

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 23 Reduce Risk and Downtime with Data Guard  Reduce Planned Maintenance Risk and Downtime with Data Guard – Upgrade standby system and switchover Data Guard Independent Components (Upgrade Standby with no impact to Primary)  Exadata software (Storage and Database Servers)  Database Server Grid Infrastructure home  InfiniBand Switch software Dependent Components  Database Server Database home (MOS )  Data Guard Standby-First Installable qualified patches only

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 24 Consolidation and Large Environments Consolidation GI version >= DB version  Required to 4th digit, recommended to 5th digit Advance diskgroup compatible.rdbms with care  Once advanced cannot be reset Share database homes whenever possible  Extend /u01 size  Patch Out-of-Place  Move databases individually to new home Large Environments Mixed Exadata version supported  Storage servers and database servers may run different versions (recommended only during rolling upgrade) Split updates to multiple components across more than one maintenance window  In general – update up the stack 1.InfiniBand switches 2.Exadata Storage Servers 3.Database Servers OS / Firmware 4.Grid Infrastructure / Database

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 25 Avoid Common Pitfalls Exadata Database Server OS and Firmware  Test previous configuration changes to avoid latent patching failures  Backup OS using validated method (e.g. dbserver_backup.sh) Database Server Grid Infrastructure and Database  Follow Exadata step-by-step MOS note for patch set upgrade  CV_DESTLOC avoids cluvfy conflicts during concurrent opatch auto  Ensure unzipped bundle patch location has proper permission  Shutdown software as documented Exadata Storage Server  Do NOT make unsupported configuration changes General  Stay up-to-date  Familiarize with doc and known issues  Test and rehearse  Run latest Exachk regularly  Fix Exachk warnings and alerts before patching

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 26 Oracle Support for Exadata  24/7 support  Specialized Engineered Systems Support Team  2-hour onsite response to hardware issues 1  New Updates and Upgrades for Database, Server, Storage, and OS software  My Oracle Support proactive support portal  "Phone home" automated service requests (ASR) Complete. Integrated. Proactive. High Availability. No Additional Cost. ORACLE PLATINUM SERVICES  Higher support levels for the complete Oracle stack – Includes higher support levels for Database software  Proactive remote monitoring for faults  Industry leading service level response times: – 5 Minute Fault Notification – 15 Minute Restoration or Escalation to Development – 30 Minute Joint Debugging with Development  Oracle Engineers perform quarterly patching and updates Available for certified configurations on Exadatacertified configurations 1 Covered system must be within an Oracle two-hour service area to receive two-hour response as a standard service.

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 27