Presentation is loading. Please wait.

Presentation is loading. Please wait.

Oracle Database Patching Best Practices II Eleanor Meritt, David Price Vice Presidents Oracle Product Development, Sustaining Engineering September 30,

Similar presentations


Presentation on theme: "Oracle Database Patching Best Practices II Eleanor Meritt, David Price Vice Presidents Oracle Product Development, Sustaining Engineering September 30,"— Presentation transcript:

1

2 Oracle Database Patching Best Practices II Eleanor Meritt, David Price Vice Presidents Oracle Product Development, Sustaining Engineering September 30, 2014 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

4 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Program Agenda Preparing for Patching Best Practices for Testing Patches Applying Patches Patching in the Cloud Other Patching Related News

5 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Preparing for Patching

6 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Assessing Risk

7 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | One Off / Interim Patch Fixes a single bug: easy to verify if problem has been fixed, quickly available Released with full component level regression testing at Oracle Accessible by everyone with a support license Low risk of introducing breakages Vast majority can be installed with zero or minimal downtime Drawback: – Easy availability leads to tendency to customize environments by combining interim patches with other patches, causing supportability & maintainability problems Risk Assessment

8 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Security Patch Update (SPU or CPU) Quarterly Patch to fix security vulnerabilities Extensively tested at Oracle Some fixes can be quite involved and may require post installation steps or configuration changes Fix verification is by nature is very difficult Historically has been very low risk Drawbacks: – Business wide coordination effort may need to be in place to manage fast roll-out. – Because the SPU does not contain fixes for high impact non-security bugs encountered by customers, a mission critical system will almost certainly need to combine the SPU with one-offs, increasing overall risk. Risk Assessment

9 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patch Set Update (PSU) Quarterly Patch to fix most recent high impact bugs Contains Security Content Extensively tested at Oracle Strict content inclusion criteria: – No optimizer changes, must be RAC rolling installable, DG Standby First Installable, fixes already tested by customers – Low level volume of content Low risk Drawback: – Strict content inclusion restrictions can sometimes lead to need to customize environments with one-off patches as well. Risk Assessment

10 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Bundle Patch (BP) Quarterly Patch to fix high impact bugs for a given configuration (e.g. Exadata) – Contains PSU Content Extensively tested at Oracle Content inclusion criteria to address stabilization needs of majority of customers running this configuration: – Also optimizer changes, must be RAC rolling installable, DG Standby First Installable. A little higher risk than other patches Drawbacks: – Less restriction on content leads to higher volumes of fixes. Some fixes are getting released for the first time within the BP. Risk Assessment

11 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Risk Assessment Try to avoid combining patches. These increase risk due to untested combinations. Apply BP’s if you are running with engineered systems, Database In- Memory, or on Windows – PSU’s for everyone else Apply the latest PSU or BP on Upgrading to a new release Ideally Patch Proactively every six months. We know about patch problems usually within 4 weeks of release. Check PAD on MOS for details on regressions & remedies Summary Recommendations

12 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | And Finally on Risk All patches can be rolled back – Scripts are provided to undo SQL patch changes within patches. datapatch accomplishes this with 12c – Binary patches can be undone by executing opatch(n)rollback or restoring the ORACLE_HOME from backup

13 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Accessing Patches

14 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Downloading patches with My Oracle Support (MOS) Find the patch(es) by performing one of the following – a Simple search for a patch or a group of patches – a Saved search – a Recent search – an Advanced search – a search using the Recommended Patch Advisor Oracle Confidential – Internal/Restricted/Highly Restricted14

15 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Patch Search Screen Oracle Confidential – Internal/Restricted/Highly Restricted15

16 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Patch Search Screen Oracle Confidential – Internal/Restricted/Highly Restricted16

17 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Patch Search Screen Oracle Confidential – Internal/Restricted/Highly Restricted17

18 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Patch Search Screen Oracle Confidential – Internal/Restricted/Highly Restricted18

19 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Patch Search Screen Oracle Confidential – Internal/Restricted/Highly Restricted19

20 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Patch Search Screen Oracle Confidential – Internal/Restricted/Highly Restricted20

21 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Conflict Checker A new conflict resolution tool is available from the patch Search screen Self Service tool that doesn’t require an SR to be logged Upload your OPatch inventory to resolve conflicts Resolution patches that are available are provided immediately Resolution requests are automatically filed if they do not exist Document Document "How to use the My Oracle Support Conflict Checker Tool" Oracle Confidential – Internal/Restricted/Highly Restricted21

22 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Conflict Checker Oracle Confidential – Internal/Restricted/Highly Restricted22

23 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Conflict Checker Oracle Confidential – Internal/Restricted/Highly Restricted23

24 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MOS Conflict Checker Oracle Confidential – Internal/Restricted/Highly Restricted24

25 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patch Download Oracle Confidential – Internal/Restricted/Highly Restricted25

26 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patch Download Oracle Confidential – Internal/Restricted/Highly Restricted26

27 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Testing Patches

28 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | A “How To” Guide for Testing Patches

29 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Categorize Your Tests Functional Regression Tests – Purpose is to check if application flows and administrator activities behave as expected. – These tests should be repeatable – Catalog all activities to test for and create tests to represent these. – Save expected results. – Run tests against the newly patched version. Compare new results against expected results. – Divide tests into groups based on functional area so you can target runs depending on patch. The Major Categories

30 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Categorize Your Tests Load Tests – Check how the system or application behaves under production load. – You may also want to simulate load beyond limits of normal operation for mission critical systems. – Key outcome is that availability should not be affected. Performance Tests – Define key performance indicators to measure against for application flows. Examples are responsiveness and throughput. – Measuring against these, no material negative differences should be seen. The Major Categories

31 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Testing Patches Real Application Testing (RAT) Capture / Replay functionality offers the ability to easily create load tests by capturing production system workloads and replaying them. RAT SQL Performance Analyzer (SPA) automates the process of assessing the effect of a patch on every SQL statement in your workload. It produces a report which you can analyze in order to remedy any potentially negative effect. A plug for Real Applications Testing

32 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Test Patches According to Risk Risk Level Functional Testing Load TestingPerformance Testing Single One-off Patch LowestTargetedNot Required SPUVery LowFullNot Required PSULowFullOptionalNot Required Bundle PatchSlightly higherFullRecommendedOptional Any Combination of the Above Low to MediumFullRecommendedOptional

33 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Applying Patches 33

34 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Best Practices for Applying Patches Aim for end to end automation of the patch application process. Write scripts to do this if necessary. Ensure you have all prerequisites in place for patch application to complete without error If you have multiple patches to apply at a time, apply them in one downtime The vast majority of patches (>98%) can be installed in a highly available manner. Check the patch README for what is possible – RAC Rolling – Dataguard Standby First – Online patches: patch a running Database – All patches can be applied Out of Place : patch a Cloned ORACLE_HOME. Afterwards check that patch has installed correctly 34

35 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Prerequisite Checking opatch prereq CheckConflictAgainstOHWithDetail Check if the patch will conflict with already installed patches opatch prereq CheckConflictAmongPatches Check if multiple patches will conflict with each other opatch prereq CheckSystemSpace Check if there is enough system space to install the patch opatch prereq CheckMinimumOPatchVersion Check the OPatch version against the required version for the patch. OPatch/ocm/bin/emocmrsp Create a response file for OCM – You may also want to run cvu and exachk where applicable 35

36 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Optimizing patch application Apply several patches during the same downtime Unzip the patches in the same patch location Execute opatch napply -skip_subset -skip_duplicate Don’t mix bundle patches and overlays (ordering issue). Should apply BP first, then overlays. Will be fixed in a later OPatch release. napply applies multiple patches in the same session skip_duplicates won’t apply patch if the patch is already on the system skip_subset won’t apply patch if the patch in the system already contains all the fixes In 12.1 datapatch takes care of applying the post SQL 36

37 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patch Installation based on Patch Packaging System Patches ‘opatchauto’ automatically installs “System Patches” – Install with Command : opatchauto apply Used for Exadata and GI bundles in RAC rolling Install mode – Other options : opatchauto apply -nonrolling Singleton Patches Used for singleton / interim patches – Install with Command : opatch apply – Other options : -silent : parameters are passed via the response file ‘-ocmrf’ -local : Apply the patch on the local node of a RAC cluster 37

38 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Running ‘datapatch’ with Database 12c Ensures data/sql related changes are patched Takes care of installing/rolling back data changes Steps After ‘OPatch apply’, Connect to the DB SQL> Connect / as sysdba SQL> startup SQL> alter pluggable database all open; [only for Multitenant DB] SQL> quit > $ORACLE_HOME/OPatch/datapatch [-verbose] Other options – -apply / -rollback – -force : runs apply/rollback as provided even if it was already done. Log location /cfgtoollogs/sqlpatch/ /. And the file name is _[apply/rollback]_ [_ ].log. 38

39 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | What’s new in datapatch in No need to call catbundle.sql any more with PSU, BP installs – Datapatch is now ‘bundle-aware’ and takes care of installing specific bundles as needed, using the dbms_sqlpatch package – Both bundle and non bundle patches are now only queryable via dba_registry_sqlpatch – dba_registry_history is no longer used for patch information datapatch -rollbackall option is available to rollback all SQL patches currently installed Oracle Confidential – Internal/Restricted/Highly Restricted39

40 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Checking Installed Patches via OPatch OPatch Commands – opatch lsinventory or opatch lsinv List the patches installed in the oracle home Other Options – - all_nodes: Report the patches installed on given Oracle Home in all nodes of RAC system – - detail(s) : Display the components and the list of patches with their associated files – - xml : Generate xml formatted output – opatch lspatches List the installed patches and their description Other options: – -bugs : Lists bug fixed by each patch – -verify : Verifies if specified patch is installed in the oracle home 40

41 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Checking Installed Patches via PL/SQL Database 12c Queryable Patch Inventory dbms_qopatch provides access to the OPatch inventory information from within the database – PLSQL/SQL interface to view list of patches applied check if a particular patch is applied patch inventory across RAC nodes SQL patch status Check datapatch entry in registry table : ‘ select * from dba_registry_sqlpatch ;’ GET_OPATCH_LISTGET_SQLPATCH_STATUSIS_PATCH_INSTALLEDGET_OPATCH_LSINVENTORY DBMS_QOPATCH subprograms

42 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patching in the Cloud 42

43 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patching in the Cloud Cloud patching is all about scaling. Every part of the patching process must be considered in terms of scaling. Automation is crucial. Zero manual steps can be permitted when patching Cloud environments. Customization is the enemy of automation. All systems should be at uniform patch levels. Administrators must have complete confidence that a patching exercise has been successful. Logging & diagnostics are very important. Scale adds to overall risk, so testing strategy has to aim for comprehensive testing coverage, automating as much of the testing as possible. Basic Principles

44 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patching in the Cloud Start with ensuring you can automate patching a single system end to end. Push out the patching related changes from the base patched system to the entire Cloud. There are various alternatives including IT automation software like Puppet or Chef. Oracle Enterprise Manager has some capabilities. Oracle Rapid Home Provisioning is newly available. Best practices for Cloud patching can be easily applied to single systems. Not necessarily so the other way round! The goals of Cloud patching can be met with all supported versions of the Oracle Database. However, 12c does have some extra features to make Cloud patching easy. Best Practices

45 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Database 12c Features to Ease Cloud Patching

46 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Multitenant for Simplified Patching Patching the Container Database Results in Patching all of its Pluggable Databases Patch Container Database Shutdown Startup

47 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Oracle Confidential Patching Using the Multi-Tenancy Feature - Best Practices When plugging into a new container a PDB may have violations due to Database version ( vs ) SQL patch mismatches Database parameter mismatches such as character sets or block size The dbms_pdb.describe and dbms_pdb.check_plug_compatibility APIs can be used to determine if a given PDB can be plugged in successfully to a target container

48 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Oracle Confidential Patching Using the Multi-Tenancy Feature - Best Practices Preparation: 1) Create PDB description XML file for PDB(PDB1) in question: exec dbms_pdb.describe (‘PDB1_Unplug.xml’, ‘PDB1’); 2) In the target container environment, check plug compatibility begin if dbms_pdb.check_plug_compatibility('PDB1_Unplug.xml', ‘PDB1') then dbms_output.put_line(‘no violations found'); else dbms_output.put_line(‘violations found'); end if; end; Plugin compatibility issues, if any, will be reported in pdb_plug_in_violations view

49 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patching Using the Multi-Tenancy Feature - Best Practices SQL> BEGIN 2 IF dbms_pdb.check_plug_compatibility('/tmp/PDBORCL.xml') THEN 3 dbms_output.put_line('no violations found'); 4 ELSE 5 dbms_output.put_line('violations found'); 6 END IF; 7 END; 8 / no violations found PL/SQL procedure successfully completed. SQL> SELECT type, message, action 2 FROM pdb_plug_in_violations 3 WHERE name = 'PDBORCL'; no rows selected Oracle Confidential Scenario 1 – No plug in violations

50 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patching Using the Multi-Tenancy Feature - Best Practices SQL> BEGIN 2 IF dbms_pdb.check_plug_compatibility('/tmp/PDBORCL.xml') THEN 3 dbms_output.put_line('no violations found'); 4 ELSE 5 dbms_output.put_line('violations found'); 6 END IF; 7 END; 8 / violations found SQL> SELECT type, message, action 2 FROM pdb_plug_in_violations 3 WHERE name = 'PDBORCL'; TYPE MESSAGE ACTION ERROR PSU bundle patch 1 (PSU Patch 12345): Installed in the CDB but not in the PDB. Call datapatch to install in the PDB or the CDB Oracle Confidential Scenario 2 – SQL patch present in target container but not in source container

51 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Patching Using the Multi-Tenancy Feature - Best Practices SQL> BEGIN 2 IF dbms_pdb.check_plug_compatibility('/tmp/PDBORCL.xml') THEN 3 dbms_output.put_line('no violations found'); 4 ELSE 5 dbms_output.put_line('violations found'); 6 END IF; 7 END; 8 / violations found SQL> SELECT type, message, action 2 FROM pdb_plug_in_violations 3 WHERE name = 'PDBORCL'; TYPE MESSAGE ACTION ERROR PSU bundle patch 1 (PSU Patch 12345): Installed in the PDB but not in the CDB. Call datapatch to install in the PDB or the CDB Oracle Confidential Scenario 3 – SQL patch present in source container but not in target container

52 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Multitenant Plug/Unplug ScenerioRecommended Action 1: SQL Patches in both source and target containerNone needed – safe to plug in 2: SQL Patches in target container onlyRun datapatch in target after plug in 3: SQL Patches in source container onlyRun datapatch -rollback – force [–bundle_series] in source before unplug Oracle Confidential Scenario Summary

53 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | DB Cloning using EM12c Provisioning - Deploy Gold Images to the Cloud 53 Mass Deployment of Oracle Software (Database, Real Application Clusters) Supports all versions up to 12.1 including Pluggable Databases Gold Image cloning and standardized software deployment via Profiles Lock down access for controlled and error free deployments DB Provisioning Source DB systems Target DB Systems Software Library Storage Save Gold image (and optionally data) from source systems to EM software library Deploy saved Image and data to target systems with customizations

54 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | EM12c Patch Management Solution Patches, Upgrades complete Database product family Provides proactive Oracle recommendations (CPUs, PSUs,..) Simplified patching flow using Patch Plans Comprehensive pre-flight checks and conflict resolution Support Out of Place, Rolling options for reduced/zero downtime and rollback/switch back Mass automation - multiple targets with multiple patches in a single downtime Extensible framework, Patch Reports and “EMCLI” scripting option *Current support available for Databases only. ** DBaaS on-premise / private cloud End to End Patch Automation Solution for Oracle Databases

55 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | New - Oracle Rapid Home Provisioning Eliminate the need to patch individual Databases Update any number of Databases with a single command Ensure standardization through gold image lineage Create reference homes on Centralized Home Server – Apply patches once on Home Server – Distribute or update on-demand to the Cloud Fast and Efficient 55 Automating Patching for Cloud

56 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Rapid Home Provisioning - commands Create a gold image from – An installed home – An existing workingcopy Create workingcopies from a gold image; optionally create database Add a database to a workingcopy Move an existing database to a different gold image – Individually or all databases configured to this image – Optionally select non-rolling Manage roles and ACLs Manage an RHP server and an RHP client 56

57 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |57 RHP Functionality Rollout Oracle Grid Infrastructure : RHP Server and Client Support for Database Templates Home, Configuration, Data NFS Mount and Local copy Efficient storage Provision and patch User initiated distribution Push or pull Support for Oracle DB Full workflow Support for generic S/W images Local change control Coming Soon: – Support for GI homes – Generic image workflow – Remote home servers – Local changes tracking – Policy based rollout – Database upgrades – Oracle application templates – VM templates

58 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Other Patching Related News Oracle Confidential – Internal/Restricted/Highly Restricted58

59 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Other Patching Related News The dbms_rolling package, introduced in Database , enables automation of minimal downtime patching in a Dataguard environment. Patch a “Leading Group” - the new Primary Database(s), and switch over applications from the “Trailing Group” – the actual Primary Database, to point to the “Leading Group” Includes: – Preparation – Validation and planning – Start Phase – Finish

60 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Other Patching Related News Introducing the “Database Patch for Engineered Systems and DB In Memory” (DBP ENG/IM) with – This patch is a super set of the PSU. It replaces the “Database Bundle Patch for Exadata”. It is intended to be consumed by customers using the Exadata, Exadoop, ZDLRA, Big Data Appliance and In Memory Database features. 2 year error correction grace period for patch sets on the R2 release trains. – Terminal patch set has always been supported through the end of Extended Support Extended Support for Database 11gR1 ends August 2015 Premier Support for Database 11gR2 ends January First year of Extended Support is at no additional cost.

61 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | References PSU known issues MOS note: How to use the MOS Conflict Checker: Document “Database 12c Post Patch SQL Automation “ Document

62 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Related Sessions How and why to Migrate from Schema Consolidation to Pluggable Databases CON7649, – Wednesday, Oct 1st, 11:30 AM - 12:15 PM - Moscone South – 306 Databases to Oracle Exadata: The Saga Continues for Oracle Enterprise Manager–Based Patching – Wednesday Oct 1 st 10:15 AM - 11:00 AM Moscone South CON8121 Rapid Home Provisioning: Deploying and Updating Database Templates in a Cloud [CON8176] – Thursday, Oct 2nd, 9:30 AM - 10:15 AM - Moscone North - 131

63 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Oracle Confidential – Internal/Restricted/Highly Restricted63


Download ppt "Oracle Database Patching Best Practices II Eleanor Meritt, David Price Vice Presidents Oracle Product Development, Sustaining Engineering September 30,"

Similar presentations


Ads by Google