Presentation on theme: "Session S317113: What do I really need to know when upgrading"— Presentation transcript:
1 Session S317113: What do I really need to know when upgrading Thomas Kyte
2 So … What Does Oracle Database 11g Mean To Me? Change
3 Small Change – but think about it… Let’s GoGreen3
4 Small Change – but think about it… ops$tkyte%ORA11GR2> create table t2 as3 select substr(object_name, 1, 1 ) str, all_objects.*4 from all_objects5 order by dbms_random.random;Table created.ops$tkyte%ORA11GR2> create index t_idx on t(str,object_name);Index created.ops$tkyte%ORA11GR2> begindbms_stats.gather_table_stats( user, 'T',method_opt => 'for all indexed columns size 254',estimate_percent=>100 );6 end;7 /PL/SQL procedure successfully completed.4
5 Small Change – but think about it… ops$tkyte%ORA11GR2> select count(subobject_name) from t t1 where str = 'T';…| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time || 0 | SELECT STATEMENT | | | | (0)| 00:00:04 || 1 | SORT AGGREGATE | | | | | || 2 | TABLE ACCESS BY INDEX ROWID| T | | | (0)| 00:00:04 ||* 3 | INDEX RANGE SCAN | T_IDX | | | (0)| 00:00:01 |5
6 Small Change – but think about it… ops$tkyte%ORA11GR2> insert into t2 select 'T', all_objects.*3 from all_objects4 where rownum <= 1;1 row created.ops$tkyte%ORA11GR2> begindbms_stats.gather_table_stats( user, 'T',method_opt => 'for all indexed columns size 254',estimate_percent=>100 );6 end;7 /PL/SQL procedure successfully completed.6
7 Small Change – but think about it… ops$tkyte%ORA11GR2> select count(subobject_name) from t t2 where str = 'T';…| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time || 0 | SELECT STATEMENT | | | | (1)| 00:00:04 || 1 | SORT AGGREGATE | | | | | ||* 2 | TABLE ACCESS FULL| T | | | (1)| 00:00:04 |7
8 ““The Law of unintended consequences holds that almost all human actions have at least one unintended consequence. Unintended consequences are a common phenomenon, due to the complexity of the world and human over-confidence.”
9 What do you have from the past… Online Parameter ChangesOnline Major Memory ChangesOnline Schema EvolutionOnline Index CreatesQuiesceRolling UpgradesOnline Disk reconfiguration (ASM)Online Cross Platform Tablespace TransportFull Database TransportsAnd more….
10 The ability to forget and let it go What do you need to know?Test To ScaleThe ability to forget and let it goSQL Plan ManagementNeverStoppingPlanning Ahead
12 Database Upgrade Process: Steps Planning AheadAnalyze & gather information about environmentDetermine the upgrade path and choose upgrade methodPrepare backup / recovery strategy and clone database to testEstablish performance baseline/metrics before upgradeDevelop a test plan for database, applications, and reportsTest upgraded database with applications and reportsEnsure adequate performance by comparing metrics gathered before and after upgradeRemediate regressions, e.g, tune queries, update database parameters, call Support, etc.Go Live!Forget and let it goASH and AWRTest To ScaleNever StoppingSQL Plan Management
14 SQL Plan Management Phase 1 - Capture Run applications to create a baselineOPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=TRUESQL MANAGEMENT BASEResiding in SYSAUX TS.Will occupy max. 10% of SYSAUX.Weekly job will delete plansnot used in 53 weeks [default].Repeated plans will be added to the SQL Plan Baseline during this phasePlan HistoryParseHJGBPlan BaselineGBHJHJ
15 SQL Plan Management Phase 2 - Selection New Plans are generated (because something changed)But are not trustedOPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=FALSENew plan will be added to the Plan History but it won't be used unless and until it has been verifiedHard ParseNLGBPlan HistoryPlan BaselineHJ
16 SQL Plan Management Phase 3 – Evolution Plans are verified – by testing the performance of the new plan in the backgroundAutomagically or ManuallyPlan HistoryPlan BaselineGBNLDBAHJEqual or better plans can be added to the SQL Plan BaselineInefficient plan willbe kept in theAutomatic Job
17 Your 9i application is already in 11g for whatever reason Upgrade ScenarioYour 9i application is already in 11g for whatever reasonYou’d like to have ‘query plan stability’Coupled with the opportunity to use better plans – do not want to be frozenThe steps would be….
18 SQL Plan Management – Parameterize STSGBNLNow: Different plans createdwith OFE=11will be added tothe Plan Historyfor later verificationOPTIMIZER_FEATURES_ENABLE=11.2.0OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=FALSEThis will cause the optimizer to capture a good approximation of the plans that would be generated in 9.2, but it‘s not a perfect impersonation of the earlier version because it still uses the 11g optimizer code. And, it won‘t capture Rule Based plans. So, it‘s not a perfect solution for people concerned about changing from rule based to cost based optimizer.Repeatable plans will be addedto the Plan Baseline upon 2ndexecution18Plan HistoryPlan BaselineGBHJGBNLGBNLOPTIMIZER_FEATURES_ENABLE=9.2.0OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=TRUE
19 Your application is in 9i You’d like to have ‘query plan stability’ Upgrade ScenarioYour application is in 9iYou’d like to have ‘query plan stability’Coupled with the opportunity to use better plans – do not want to be frozenYou will be changing platforms during the upgrade (not doing a direct upgrade of the database)The steps would be….
20 SQL Plan Management – Outlines Exp/imp outlines toNew systemexp imp expdp impdp DB-Link ...Query outlines can be captured on production and then exported/imported to a test system. Capturing plans with these outlines in place will preserve the original plans in the baseline. Just make sure to execute each statement twice.STSSSPlan HistoryPlan BaselineGBNLHJDBMS_SPM.MIGRATE_STORED_OUTLINE320Capture query outlines on the production system
21 Same Scenario but your application is in 10g Upgrade ScenarioSame Scenario but your application is in 10gYou’d like to have ‘query plan stability’Coupled with the opportunity to use better plans – do not want to be frozenYou will be changing platforms during the upgrade (not doing a direct upgrade of the database)The steps would be….
22 SQL Plan Management – Tuning Pack STSStagingTableexp imp expdp impdp DB-Link ...STSSTS = SQL Tuning Set. This is a mechanism we provide to collect a set of statements and their plans. This part of the DBMS_SQLTUNE package, and requires a Diagnostic and Tuning Pack license.The STS can be put into a staging table, which can then be transported to the new database using one of several mechanisms. We can then unpack the STS and place all the 10.2 plans into the Plan Baseline.Now, when the database finds plans that it thinks are better, those plans will go into the plan history for later evolution and acceptance by the DBA.Package DBMS_SQLTUNE (ADD_SQLSET_REFERENCE, CAPTURE_CURSOR_CACHE_SQLSET, CREATE_SQLSET, CREATE_STGTAB_SQLSET, DELETE_SQLSET, DROP_SQLSET, LOAD_SQLSET, PACK_STGTAB_SQLSET, REMOVE_SQLSET_REFERENCE, SELECT_CURSOR_CACHE, SELECT_SQLSET, SELECT_WORKLOAD_REPOSITORY, UNPACK_STGTAB_SQLSET, UPDATE_SQLSET)Plan HistoryPlan BaselineGBNLHJ10.2 plans will becomethe SQL Plan Baseline322
23 You would like to deploy from development to production.. Upgrade ScenarioYou would like to deploy from development to production..You would like to deploy at a customer site…And you want to start with a stable set of plansUsing better plans only after they have been verifiedThe steps would be….
24 SQL Plan Management - New Application exp imp expdp impdp@CustomerStagingTable@VendorThis requires 11g. If a vendor doesn‘t know for sure how customers will parameterize their database, then the vendor can ship a staging table of query baselines to the destination, and ensure that a default good plan is stored in the customer‘s plan baseline.DBMS_SPM.CREATE_STGTAB_BASELINEStagingTable24DBMS_SPM.UNPACK_STGTAB_BASELINEPlan BaselineGBNLHJ3DBMS_SPM.PACK_STGTAB_BASELINEPlan BaselineGBHJGBNLGBNL
26 Database Replay Overview Replay actual production database workload in test environmentIdentify, analyze and fix potential instabilities before making changes to productionCapture Workload in ProductionCapture full production workload with real load, timing & concurrency characteristics (9i, 10g, 11g)Move the captured workload to test system (11g)Replay Workload in TestMake the desired changes in test systemReplay workload with full production characteristicsHonor commit orderingAnalyze & ReportErrorsData divergencePerformance divergenceOracle Database 11g provides two specific solutions to these aforementioned challenges.Database Replay allows you to test the impact of a system change by replaying real-world workload on the test system before it is exposed to a production system. The production workload (including transaction concurrency and dependency) of the database server is recorded over an illustrative period of time (for example a peak period). This recorded data is used to replay the workload on a test system that has been appropriately configured. You gain a high degree of confidence in the overall success of the database change by subjecting the database server in a test system to a workload that is practically indistinguishable from a production workload.Analysis & Reporting
27 Recording of External Client Requests Supported ChangesClient…Changes Unsupported(there are other tools for that)Database Replay focuses on recording and replaying of the workload that is directed to the RDBMS. Therefore recording of the workload is done at the point indicated in the above diagram. Recording at the RDBMS within the software stack makes it possible to exchange anything below this level and test the new setup using the record and replay functionality.While replaying the workload, the RDBMS performs the actions observed during recording. In other words, during the replay phase the RDBMS code is exercised in a very similar way as it was exercised during the recording phase. This is achieved by recreating all external client requests to the RDBMS. External client requests include all the requests by all possible external clients of the RDBMS.Middle TierChanges SupportedRecording of External Client RequestsDatabase Upgrades, PatchesSchema, ParametersRAC nodes, InterconnectOS Platforms, OS UpgradesCPU, MemoryStorageEtc.Storage
28 Step 1: Workload Capture Production SystemAll external client requests captured in binary filesSystem background and internal activity excludedMinimal overheadAvoids function call when possibleBuffered I/OIndependent of client protocolCan capture on 9i, 10g, and 11g and replay on 11gCapture load for interesting time period, e.g., peak workload, month-end processing, etc.Here you see an illustration of a system that is being recorded. You should always record a workload that spans an “interesting” period in a production system. Typically, the replay of the recording is used to determine whether it is safe to upgrade to a new version of the RDBMS server. During recording, special recording infrastructure built into the RDBMS records data about all external client requests, while the production workload is running on the system. External requests are any SQL queries, PLSQL blocks, PLSQL remote procedure calls, DML statements, DDL statements, Object Navigation requests and OCI calls. Background jobs and in general all internal clients continue their work during recording, without being recorded. The end product is the workload recording contains all necessary information for replaying the workload as seen by the RDBMS in the form of external requests.The recording infrastructure imposes minimal performance overhead (extra CPU, memory and I/O) on the recording system. You should however plan to accommodate the additional disk space needed for the actual workload recording.RAC Note: Instances in a RAC environment have access to the common database files. However, they do not need to share a common general-purpose file system. In such an environment, the workload recording is written on each instance’s file system during recording. For processing and replay all the parts of the workload recording need to be manually copied into a single directory.OverheadPerformance Overhead for TPCC is 4.5%Memory Overhead : 64k/sessionClient…File SystemMiddle TierFile 1File 2…File nStorage
29 Step 2: Process Workload Files Setup test systemApplication data should be same as production system as of capture start timeUse RMAN, Snapshot Standby, imp/exp, Data Pump, etc. to create test systemMake change: upgrade db and/or OS, change storage, migrate platforms, etc.Processing transforms captured data into replayable formatOnce processed, workload can be replayed many timesFor RAC copy all capture files to single location for processing or use shared file systemTest SystemFile 1File 2File n…Capture FilesFile 1File 2…File nMetadataReplay Files
30 Step 3: Replay Workload Test System … Replays workload preserving timing, concurrency and dependencies of the capture systemReplay Client is a special program that consumes processed workload and sends requests to the replay systemClients interpret captured calls into sequence of OCI calls and submit to databaseFor high concurrency workloads, it may be necessary to start multiple clientsReplay ClientsThe workload recording is consumed by a special application called the replay driver, which sends requests to the RDBMS on which the workload is replayed. The RDBMS on which the workload is replayed is usually a test system. It is assumed that the database of the replay system is suitable for the replay of the workload that was recorded. The internal RDBMS clients are not replayed. The replay driver is a special client that consumes the Workload Recording and sends appropriate requests to the test system to make it behave as if the external requests were sent by the clients used during the recording of the workload (see previous example). The use of a special driver that acts as the sole external client to the RDBMS allows for the record and replay infrastructure to be client agnostic.The replay driver consists of one or more clients that connect to the replay system and sends requests based on the workload capture. The replay driver equally distributes the workload capture streams among all the replay clients based on network bandwidth, CPU and memory capability.Why OCI?Powerful and FlexibleSupports a superset of most protocolsFile 1File 2…File nMetadataReplay Files
31 Analysis & ReportingError Divergence: For each call error divergence is reportedNew: Error encountered during replay not seen during captureNot Found: Error encountered during capture not seen during replayMutated: Different error produced in replay than during captureData DivergenceReplay: Number of rows returned by each call are compared and divergences reportedUser: Application level validation scriptsPerformance ReportingCapture and Replay Report: Provides high-level performance informationADDM Report: Provides in-depth performance analysisAWR, ASH Report: Facilitates comparative or skew analysisThere may be some divergence of the replay relative to what was recorded. For example when replaying on a newer version of the RDBMS a new algorithm may cause specific requests to be faster, and so divergence appears as a faster execution. This is considered a desirable divergence. Another example of a divergence is when a SQL statement returns fewer rows during replay-time than those returned during record-time. This is clearly non-desirable and its root cause may be some new index look-up algorithm. The replay will identify this fact.For data divergence the result of an action can be considered as:The result set of SQL query.An update to persistent database state.A return code or an error codePerformance divergence is useful to determine how new algorithms introduced in the replay system may affect overall performance. There are numerous factors that can cause replay divergence. While some of them cannot be controlled, others can be mitigated. It is the task of the DBA to understand the workload runtime operations and take the necessary actions to reduce the level of record and replay divergence.The two types of divergence reporting are listed above. Online divergence should aid the decision to stop a replay that has diverged significantly. The results of the replay before the divergence may still be useful, but further replay would not produce reliable conclusions. Offline divergence reporting is used to determine how successful the replay was after the replay has finished.
33 SQL Performance Analyzer: Workflow ProductionTestMughees:Similar to database replay, SQL Performance Analyzer is another key capability of Real Application Testing but it focused on SQL execution plan regressions. One of the biggest concerns DBA’s have when making any change is its impact on SQL statement performance. SQL Performance Analyzer provides the ability to capture the entire SQL workload from the production environment, replay it in the test environment and then provide a report identifying all SQL statements that improved, those that got worse, and those that remained unchanged. It then optionally allows you to automatically tune the regressed SQL statements using Oracle Database 11g capabilities like SQL Tuning Advisor and SQL Plan Baselines.The end result is that all negative outcomes of a change are quickly identified and addressed using SQL Performance Analyzer so that businesses can continue to embrace change while minimizing risk.Moe: Mughees, What are the unique capabilities of SQL Performance analyzer?Mughees: SQL Performance analyzer offers three unique capabilities not available from other solutions in the market. 1) Low overhead, 2) test with production context and 3)automatic transfer of resolutions to production.Moe: Lets talk about the effort involved here. How much could customers reduce testing effort when using RAT?Make ChangeSteps (1) (2) (3) (4) (5)CaptureSQL (STS)TransportSTSExecute SQLPre-changeExecute SQLPost-changeComparePerf.(6) ReiterateDone?No(7)YesTuned SystemProduction Change / Tuning Deployment(7)
34 SQL Performance Analyzer: Key Differentiators From:To:Automated SQL capture, Negligible overheadManual SQL capture, High overheadNon-production SQL contextProduction SQL contextPartial SQL workloadComplete SQL workloadAutomated analysis in minutesMonths of manual analysisAutomatic regression tuningManual regression tuningLow risk, Low costHigh risk, High cost
35 Real Application Testing: Tools of the Trade SQL Performance AnalyzerDatabase ReplayWhat is it?Predicts SQL performance deviations before end-users can be impacted, helps assess impact of change on SQL response timeReplays real database workload on test system, helps assess impact of change on workload throughputHow it works?Executes each SQL, stored in SQL Tuning Set, in isolation using production context and then compares before and after execution plans and run-time statisticsCaptures workloads and replays it with production characteristics including concurrency, synchronization & dependenciesWhen to use?Unit testing of SQL with the goal to identify the set of SQL statements with improved/regressed performanceComprehensive testing of all sub-systems of the database server using real production workloadSQLConcurrencySQLDependencySpeed up/down
36 More information… Hands on Lab: S318966 Database and Application Testing HOLWed: pmMarriott Golden GateSPA / Database Replay Demo groundsMoscone West: 038/03936
38 Flashback for Rapid Recovery from Human Error Flashback Data Archive:Automatically stores all changes to selected tablesArchive cannot be modifiedView table as of any timeUses:Change TrackingILMLong term history - yearsAuditingComplianceFlashback QueryFlashback TablesFlashback DatabaseFlashback Data Archive and Transaction
39 Restore Points Restore point – specifies a jump label Named Restore PointSimilar to a bookmark"Can be" - but no guaranteeWill be recorded to the control fileGuaranteed Restore PointSimilar to storage snapshotsOverrides the FLASHBACK_RETENTION_TARGETAttention: A guarantee restore point can stop the whole databaseThe guaranteed restore point is useful in upgrades as a fallback scenario (if you‘re coming from 10g). Remember, it was also part of the transient logical standby migration strategy.If you use the guaranteed restore point for a fallback strategy and the upgrade succeeds, you may want to clear that guaranteed restore point to make sure your disk doesn’t fill up and stop the database.SQL> CREATE RESTORE POINT rpt;SQL> FLASHBACK DATABASE TO RESTORE POINT rpt;39SQL> CREATE RESTORE POINT grpt GUARANTEE FLASHBACK DATABASE;SQL> FLASHBACK DATABASE TO RESTORE POINT grpt;
41 Rolling Database Upgrades MajorReleaseUpgradesPatch SetCluster Software & Hardware UpgradesRedoClientsLogsQueueVersion XVersion XXX+112Initial SQL Apply ConfigUpgrade node B to X+141ABABRedoRedoUpgradeX+1X+1XX+14Switchover to B, upgrade A3Run in mixed mode to test
42 Online Application Upgrade Edition-based redefinition Code changes are installed in the privacy of a new editionData changes are made safely by writing only to new columns or new tables not seen by the old editionAn editioning view exposes a different projection of a table into each edition to allow each to see just its own columnsA crossedition trigger propagates data changes made by the old edition into the new edition’s columns, or (in hot-rollover) vice-versa42
44 Editions & object visibility (inherited)Object_3Object_3(inherited)44Object_2Object_2(inherited)Object_1Object_1(inherited)Pre-upgrade editionPost-upgrade editionis child of
45 Editions & object visibility (actual)An editioned object is visible in a child of the edition where it was created.When you “create or replace” it, you get a new actual occurrence that hides the occurrence that was inherited from the parent.The old occurrence is still visible in the parent.Drop and rename work as expected.An editioned object has a new attribute: the name of the edition where it was created. You can see this in the new DBA_Objects.Edition_Name column.The DBA_Objects catalog view family, and all the pre-existing ones, show you what is visible in the current edition – and only that. Just the DBA_Objects family (and a couple of others) has the Edition_Name. This tells you whether a particular object is actual or inherited.There is a new DBA_Objects_AE family that tells you about everything in all editions (hence the suffix). There are a few other _AE view families.Object_3Object_3*(actual)45Object_2Object_2(inherited)Object_1Object_1(inherited)Pre-upgrade editionPost-upgrade editionis child of
47 The following is intended to outline our general product direction The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
51 ‘End-to-End’ Upgrade Lifecycle PreparationUpgradePost-UpgradePhaseUpgrade PlanUpgrade TestingRehearsalProduction UpgradeMonitor & MaintainSub-PhaseMy Oracle Support Upgrade Plan*Real Application TestingProvisioningMonitoringEnterprise Manager – Grid ControlMy Oracle Support ó EM Grid Control Integrated solution can be leveraged throughout full lifecycle*Will be integrated in upcoming release51
54 Documentation Note:785351.1 Upgrade Companion 11g Release 2 54 Companion 11.2: Note:The upgrade companions are in the top 5 downloads from MetaLink. We produced the 10gR2 companion first back in 2007, because customers were experiencing many problems with upgrades and these problems were getting escalated to the executive level.We have an initiative across development, support, consulting, ...many parts of the company...to improve the upgrade process. Every Friday we meet to discuss upgrade issues, improve the Companions, and tackle other upgrade topics.As a result of this initiative, one of the new things we have done as part of the upgrade program is the Upgrade Companion.The Upgrade Companion is a great place to start – if you are just looking at an upgrade – look at one of these.Notes in My Oracle Support. It provides best practices and other considerations to take before you upgrade.Since the Upgrade Companion was released we have found that customers who follow these best practices do not have upgrade problems. The very few escalations we have seen were from customers who skipped significant preparation steps.Note: Upgrade Companion 11g Release 254
55 What are my upgrade paths? Predictable performance post-upgrade 5511.2SQL Plan ManagementAutomated SQL tuningReal Application Testing