Oracle Database Backup-and-Recovery Best Practices and New Features

Oracle Database Backup-and-Recovery Best Practices and New Features
Oracle delivers a complete backup-and-recovery solution comprising Oracle Recovery Manager (RMAN), Oracle Secure Backup, Oracle Data Recovery Advisor, and Oracle Flashback-leveraging disk, tape, and innovative cloud technologies. This session outlines how these features can be used as essential components of a high-availability strategy for superior protection from a wide range of logical and media corruptions. The session also covers the new Oracle Database 11g Release 2 backup-and-recovery capabilities. Hear directly from a major Oracle customer about its backup-and-recovery design and experiences with multiterabyte Oracle Real Application Clusters (Oracle RAC) 11g online transaction processing (OLTP) and archival/reporting databases. Oracle Database Backup-and-Recovery Best Practices and New Features Timothy Chien Principal Product Manager Database High Availability Husnu Sensoy VLDB Expert Turkcell Communication Services

<Insert Picture Here>
Agenda <Insert Picture Here> What Keeps You Awake at Night? Oracle Data Protection Planning & Solutions Oracle Backup & Recovery Solutions Physical Data Protection Recovery Manager Oracle Secure Backup Logical Data Protection Flashback Technologies Recovery Analysis Data Recovery Advisor Putting It All Together: Customer Example Turkcell Backup & Recovery Case Study Q&A

What Keeps You Awake at Night? Data Protection Concerns…
Meeting recovery SLAs? Reducing exposure to data loss? Meeting backup windows? Dealing with long-term backup storage? Management complexity? Budget? …Where do I begin?

Assess Recovery Requirements First Step in Data Protection Planning
Identify and prioritize critical data Design recovery requirements around data criticality Assess tolerance for data loss - Recovery Point Objective (RPO) How frequently should backups be taken? Point-in-time recovery required? Assess tolerance for downtime - Recovery Time Objective (RTO) Downtime: Problem identification + recovery planning + systems recovery Tiered RTO per level of granularity, e.g. database, tablespace, table, row Determine backup retention policy Onsite, offsite, long-term Assess data protection requirements Physical: Disasters, outages, failures, corruptions Logical: Human errors, application errors First step is to start with how critical lost or unavailable data is to your business – how much will the company lose in $x/hr of downtime, not to mention in productivity and labor costs, if a database is down? This cost is important as it will justify HW and storage expenditures to prevent and recover from those outages. Once the business cost and impact is determined for unavailable data, next step is to classify those databases and potentially data sets (recall database consolidation trend), according to criticality. E.g. you may have large DW reporting system that can tolerate 12 hours worth of lost data, as batch loads can be re-run, with several hours of acceptable downtime. This system may fit well with a disk/tape backup strategy. You may also have a critical OLTP system that can tolerate no more than a few mins worth of lost data and several mins of acceptable downtime, e.g. online sales processing, which can cost the company $100K/hr in lost revenue, if unavailable. In this case, traditional backups will not suffice and minimal downtime solution, e.g. Data Guard, should be evaluated. First key requirement is RPO – how much data loss is acceptable? Are there some databases and data sets that require more frequent backups than others? If I need PITR, have I provisioned sufficient disk space for archived logs? Second key requirement is RTO – how much recovery time can I tolerate? Can I tolerate hours, or only minutes? * Recovery Object Granularity (ROG). ROG measures the level of objects that a solution is capable of recovering. We are the best there because we can recover just table or row. The other important characteristics for DW databases is Recovery Event Granularity (REG). REG measures the capability of a recovery management solution to track events and to recover a failed application or missing data to a specific event. E.g., if event is an ETL load, you should be able to easily revert back if ETL load is bad. What are your Recovery Time Objectives (RTO) – i.e. how quickly must you recover from a data failure? RTO may vary by database and/or server Consider combined disk and tape backup strategy for critical databases to meet more pressing RTO requirements Determine desired recovery capability from disk and tape What time period represents most frequent recovery needs? Do you need to retain data on a long-term basis? Review Oracle Suggested Strategy Utilize the benefits of integrated disk and tape backup using technologies such as Fast Recovery Area, RMAN and Oracle Secure Backup

Oracle Maximum Availability Architecture Robust & Integrated Data Protection
Active Data Guard Fully Active Failover Replica Standby Site Production Site Database Database Storage Data Recovery Advisor Intelligent, Guided Recovery Analysis Storage Recovery Manager (RMAN) & Oracle Secure Backup (OSB) Low Cost, High Performance Backup & Recovery Flashback Technologies Correct Errors by Moving Back in Time

Oracle Data Protection Solutions
Backup & Recovery Recovery Time Objective (RTO) Physical Data Protection Recovery Manager (RMAN) Oracle Secure Backup (OSB) Hours/Days Logical Data Protection Flashback Technologies Minutes/Hours Recovery Analysis Data Recovery Advisor Minimizes time for problem identification & recovery planning Disaster Recovery Recovery Time Objective (RTO) Physical Data Protection Active Data Guard Seconds/Minutes

Oracle Backup & Recovery Solutions “Backup and Recovery on Steroids”
Physical Data Protection Recovery Analysis Data Recovery Advisor File System Data UNIX Linux Windows NAS Oracle Databases Logical Data Protection Flashback Technologies Recovery Manager (RMAN)

Agenda <Insert Picture Here> What Keeps You Awake at Night? Oracle Data Protection Planning & Solutions Oracle Backup & Recovery Solutions Physical Data Protection Recovery Manager Oracle Secure Backup Logical Data Protection Flashback Technologies Recovery Analysis Data Recovery Advisor Putting It All Together – Customer Example Turkcell Backup & Recovery Case Study Q&A

Backup & Recovery Foundation Complete Oracle Solution from Disk to Tape
Oracle Secure Backup (OSB) File System Data Tape Backup UNIX Linux Windows NAS Oracle Recovery Manager (RMAN) Oracle Databases Fast Recovery Area Amazon S3 Cloud Storage Oracle Secure Backup (OSB) Cloud Module Oracle Secure Backup is centralized tape backup management software protecting the Oracle database and file systems in distributed UNIX, Linux, Windows and Network Attached Storage (NAS) environments. From one central console, you can easy manage the distributed servers and tape devices within the backup domain. As Oracle is no longer just a database company, Oracle Secure Backup provides tape backup for application files as well as the database. Integrated with Recovery Manager (RMAN), Oracle Secure Backup provides the media management layer for RMAN backups to tape. While tightly integrated with the Oracle database, Oracle Secure Backup is a standalone product offering with an independent release schedule and versioning from the database. OSB 10.1 was released in April 2006 and OSB 10.2 was released in Nov 2007. 3rd Party Media Management Software can be EXPENSIVE, and cost thousands per database server for RMAN integration! OSB will save customer money….Oracle ROI Oracle - Single vendor solution with RMAN+OSB reduces complexity for customer…eliminates finger pointing in multi-vendor solution Oracle backup and recovery for your entire IT environment Multiple media options available to meet the most stringent SLAs Local disk, remote Cloud storage, physical and virtual tape

Oracle Recovery Manager (RMAN) Oracle-integrated Backup & Recovery Engine
Oracle Enterprise Manager Intrinsic knowledge of database file formats and recovery procedures Block validation Online block-level recovery Tablespace/data file recovery Online, multi-streamed backup Unused block compression Native encryption Oracle Secure Backup RMAN Tape Drive Recovery Manager (RMAN) – ensures valid backup & restore Always verifies block checksums on backup & restore Provides optional logical block validation (e.g. missing row piece) Checks on-demand for backup / restore corruptions without creating backups / restores (BACKUP VALIDATE / RESTORE VALIDATE) Provides online recovery of individual block corruptions or all identified corruptions with Block Media Recovery (RECOVER BLOCK) Integrated disk, tape & cloud backup leveraging the Fast Recovery Area and Oracle Secure Backup Fast Recovery Area Cloud Database

Integrated backup-storage tiering
Oracle Fast Recovery Area Automatic Disk-to-Disk (D2D) Backup & Recovery Fast Recovery Area – Integrated D2D backup and recovery Favorable disk economics – low-cost disks used for recovery area Oracle makes it even better with ‘ ‘restore-free recovery’: switch datafile 4 to copy; recover datafile 4; Fast incremental backups Backs up only changed blocks Changed blocks are tracked using a very efficient algorithm, e.g. 20X faster Nightly incremental backup rolls forward recovery area backup No need to do full backups recover copy of database with tag ‘ORCL’; Database Area Nightly Apply Validated Incremental Fast Recovery Area Weekly Archive To Tape RMAN, which is Oracle’s Backup & Recovery solution, fully automates disk based backup and recovery in Oracle Database10g, using essentially a tiered storage configuration. You can use your mission-critical million-dollar storage array to hold database processing related files – using an area called Database Area, while using a more cost-effective, but performant storage array – maybe comprised of ATA/SATA disks, called the Recovery Area. Oracle manages this transparently for you. With an Oracle suggested backup strategy, you can start by doing a full backup of your database, and then setting up an automated policy to do nightly incremental backups. These backups could be stored in the Recovery Area. Two things to note here. Firstly – while doing incremental backups – only the changed data blocks are backed up – thereby saving considerable storage space and enabling what in the storage industry is referred to as de-duplication – i.e. eliminating the need to store multiple copies of the same block. Also, with a mechanism called Block Change Tracking – the changed blocks are tracked very efficiently. Secondly – setting up an automated RMAN policy, these nightly incremental backups could be used to roll forward recovery area backup and automatically create a full backup, thereby eliminating the need to manually do a full backup. Note that while we recommend the use of disk here as a very viable recovery mechanism, we are not discounting the value of tape. Tape continues to remain as a very effective backup media – and in this configuration as well, an automated policy could be setup to have old data age out tapes for archival purpose. Net net – this mechanism enables much faster backups – just propagate changes to recovery area, and also much faster restores – just copy backup files from Recovery Area, or simply use the copy in the Recovery Area. ================ An example of block change tracking improving incremental perf by 20x: E.g. 500 GB database, 25 GB changes since level 0, one channel - 9i: 10000s (~2.8 hrs) for inc backup - scanning full 500 GB of changed+unchanged blocks - 10g w/ BCT: 500s (~8.3 min) for inc backup - scanning only 25 GB of changed blocks Integrated backup-storage tiering

RMAN New Features Oracle Database 11g Release 2
Automatic Block Repair Allows corrupt blocks on the primary database to be automatically repaired from physical standby database, as they are detected. In-line and transparent. User sees brief wait from query on corrupt block while it is being repaired. Can also be performed on-demand via RECOVER command Requires Active Data Guard (real-time query on physical standby database). Automatic Block Repair Queries Primary database Active Data Guard Standby

Backup compression: popular way to save on storage costs Multiple RMAN backup compression levels Choose compression levels & backup throughput [BASIC] | HIGH | MEDIUM | LOW HIGH – reduces backup size by 40%+ depending on data type LOW – least impact on backup throughput MEDIUM – best balance between compression and throughput HIGH | MEDIUM | LOW require Advanced Compression Option LOW - corresponds to LZO (11gR2) – smallest compression ratio, fastest MEDIUM - corresponds to ZLIB (11gR1) – good compression ratio, slower than LOW BASIC (free) - corresponds to BZIP2 (10g style compression) - ~compression ratio as MEDIUM, but slower HIGH - corresponds to unmodified BZIP2 (11gR2) – highest compression ratio, slowest

In previous releases, DUPLICATE required RMAN client connections to source and clone databases. With enhanced DUPLICATE, connection to source database not needed for environments where network connection is not available. Source Database Clone Database Firewall Restriction RMAN duplicate does not require connection to source (i.e. target) database. This was to address customer feedback that connection to source database from clone db site may be blocked by network (per security policy) and that any problem with this connection causes duplicate operation to fail. When catalog connection is available: DUPLICATE utilizes existing backup on disk or tape, as normal When catalog connection is not available: Provide disk backup location of all needed backup sets, datafile copies, archived logs, and control file copies. SQL Net Connections Restore Processes RMAN Client Disk/Tape Backup

Additional RMAN New Features
Benefit Backup Fast Recovery Area to disk location Protect Fast Recovery Area with on-disk backup of its RMAN backups, archived logs, and controlfiles. Extended tablespace point-in-time recovery (TSPITR) capabilities Recover a dropped tablespace. Perform multiple tablespace point-in-time recoveries, without requiring recovery catalog Resumable DUPLICATE DUPLICATE can resume processing from most points of failure, reducing overall time. CONVERT DATABASE can skip unneeded datafiles Reduces overall conversion time by only processing the required UNDO-containing data files. SET NEWNAME FOR TABLESPACE | DATABASE Simplifies renaming of datafiles for RESTORE, DUPLICATE, and TSPITR operations. Resumable duplicate - bug for details.

RMAN Best Practices

RMAN Best Practices Fast Recovery Area (FRA) guidelines
Place FRA on separate storage & store backups, in addition to copy of control file, redo logs, and archived logs, to protect all needed recovery-related files from production outages. When estimating FRA size, if you want to keep: Control file backups and archived logs Estimate archived logs generated between successive backups on the busiest days and multiply total size by 2 to account for activity spikes. Archived logs and Flashback logs Multiply the archived log size between backups by 4, assuming Flashback retention = time between archived log backups. Incremental backups Add in their estimated sizes On-disk image copy backup Add in size of the database minus the size of temp files The Fast Recovery Area (FRA) is one location to hold recovery-related files, e.g. RMAN backups, Flashback logs, archived logs, multiplexed copy of online redo, multiplexed copy of controlfile, specified with a location (directory or ASM disk group) and space quota (upper limit). The FRA automatically deletes unneeded files when there is space pressure; ‘unneeded files’ are those that are either (1) backed up to tape via RMAN, or (2) obsolete according to RMAN retention policy. Calculating an appropriate initial FRA size depends on what the user wants to keep, e.g. e.g. one may just want Flashback logs and archived logs, and no RMAN backups. The following guidelines assist with determining the initial size. When keeping only controlfile autobackups and archived logs, one can find an approximate FRA size estimate by finding the total size of archived logs generated between successive backups on the busiest days (controlfile autobackups are generally small relative to archived logs). This is because once the archived logs are backed up, they are considered ‘reclaimable’ and will be deleted under space pressure, so you only need enough space to hold archived logs between two successive backups. This estimate is multiplied by 2 to accomodate unexpected redo spikes. When keeping both archived logs and Flashback logs, you can multiply the archived log size by 2 to get an initial estimate. This is because Flashback logs are generally created in proportion to archived redo logs generated during same retention period. So, multiply the archived log size by 4 (2x for archived logs + 2x for Flashback logs) to accommodate unexpected redo spikes, which also affect Flashback log sizes. There is no formula to estimate incremental backup sizes – they are really dependent on the amount of changes between backups. One can test run an incremental strategy to determine representative incremental sizes for a period of time, and then include those in calculation of production FRA size. Finally, if an on-disk image copy backup is to be kept in the FRA, then add in the size of the database minus size of temp files (RMAN does not backup temp files). An on-disk image copy backup allows for faster restore vs. tape, or can just be used as-is in place of the production storage data file.

RMAN Performance Factors Balancing Backup and Restore Requirements
Consideration Performance Effect Incremental Backup Strategy Multiplexing Hardware/Network/ Storage Incremental backup strategy improves backup performance, with trade-off in recovery performance Enable block change tracking for fast incremental backups Cumulative vs. differential incremental backups ‘Incremental forever’ requires an initial full then incrementals thereafter Fast recovery: Current image copy of database readily available Backup ‘x’ files in parallel per channel, improving backup performance RMAN multiplexing level = min(FILESPERSET, MAXOPENFILES) Exception: Set MAXOPENFILES = 1 for SAME or ASM datafiles Set # of RMAN channels = # of tape drives, so that media management multiplexing is not used for RMAN backups Setting # of RMAN channels > # of tape drives will impact restore, due to interleaved backup pieces on single tape Multiple channels per tape drive can substantially slow restore performance (I.e. media management multiplexing) RMAN channel represents a single backup file stream. A channel can read multiple datafiles or archived logs to backup into a multiplexed backup set, or can read one file at at a time for image copy backup. Increasing RMAN channels increases backup parallelism, thereby potentially reducing overall backup time. RMAN multiplexing is the number of datafiles or archived logs read by one channel at any time. The default is min (FILESPERSET, MAXOPENFILES). FILESPERSET defaults to 64 and specified at BACKUP command. MAXOPENFILES defaults to 8 and is specified at CONFIGURE CHANNEL command. So multiplexing=8 by default, which means a maximum of 8 files will be read by one channel at any time. For SAME/ASM storage, MAXOPENFILES should be set to 1, since all files are striped across available disks and reading one file/channel will optimally read from each disk. Do not utilize media management multiplexing for the Oracle database, as RMAN backup pieces will not be efficiently restored, due to interleaving of pieces on same tape volume. I.e. the tape may need to move forward/rewind, depending on which pieces are needed by restoration. Maxopenfiles = 1 for ASM…the files are already stripped appropriately across all disks so they will already be efficiently read by RMAN eliminating the need for higher multiplexing values. Assess host resources, production disk I/O, HBA/network, tape drive throughput Minimum performant component of these will be performance bottleneck

Data Warehouse B&R Best Practices
Exploit partitioning and read-only tablespaces Older partitions can be moved to read-only tablespaces Backup read-only tablespaces once, then periodically, depending on tape retention policy Divide full backup workload across multiple days Leverage database & backup compression Save time with tablespace level backups Backup index tablespaces less frequently than data tablespaces Backup scarcely used tablespaces less frequently Reduce restore time for most critical tablespaces, by grouping them together in separate backups Take incremental backup when NOLOGGING operations finish to ensure recoverability Divide full backup workload across multiple days E.g. BACKUP DATABASE NOT BACKED UP SINCE ‘SYSDATE-3’ DURATION 06:00 PARTIAL MINIMIZE TIME; Above command will start full backup and do as much work as possible in 6 hour backup window. Next day, same command will pickup with last file not backed up and resume full backup, again in 6 hour backup window. So, over course of a few days, entire database will be backed up.

Test, Test, Test Recovery…
Recovery Scenario Oracle Technologies Media Failure RMAN – restore all files to new storage location Block Corruption RMAN Validate, Block Media Recovery, Trial Recovery, LogMiner User/Logical Error Flashback Technologies, RMAN TSPITR, LogMiner Disaster Data Guard; RMAN -- restore all files to new host/storage Media Failure Restore database files to a new location Validate restoring database, tablespace, data file Block Corruption Validate database Validate data file block corruption repair with Block Media Recovery Validate archived log apply with Trial Recovery (detect corrupt and missing logs) User Error Validate correction of user error using Flashback Query (Flashback Table, Flashback Drop, Flashback Database also available in Oracle Database 10g) Perform tablespace point-in-time recovery (TSPITR) to recover original data prior to an erroneous transaction Research corrupt or erroneous transactions with LogMiner Disaster Validate restoring all files to another host Test switchover/failover procedures to standby database in Data Guard configuration Data Recovery Advisor – built-in database failure diagnosis, analysis, & repair tool

Additional Resources RMAN Step-by-Step Performance Tuning (NEW)
Very Large Database Backup & Recovery Best Practices Best Practices using Recovery Manager with Oracle Data Guard and Oracle Streams Because RMAN is server managed backup, there are several components in the environment that can affect overall performance. The backup flow is as follows: 1) RMAN uses one or more server processes on the database host to generate backups 2) Database blocks are read from production disk and into a set of input memory buffers 3) Blocks are validated and written to a set of output memory buffers 4) Blocks are composed into backup pieces and written to either disk (default) or tape (via Oracle Secure Backup or 3rd party media manager) E.g If host CPU/memory cannot support these additional processes and network+backup media throughput is adequate, then consider adding host resources. Refer to the optimization paper and Fannie Mae case study for tuning details.

Oracle Secure Backup

Oracle Secure Backup (OSB) Enterprise Tape Backup Management
Protects Entire IT Environment Oracle Database 11g Release 2 to Oracle9i 25 – 40% faster tape backup Heterogeneous file systems (UNIX/ Linux / Windows) and NAS devices Built-in Oracle Integration Centralized management in distributed environments Over 75% less expensive than comparable products Oracle Enterprise Manager Oracle Secure Backup File System Data Oracle Database RMAN Integration Oracle Secure Backup is centralized tape backup management software protecting the Oracle database and file systems in distributed UNIX, Linux, Windows and Network Attached Storage (NAS) environments. From one central console, you can easy manage the distributed servers and tape devices within the backup domain. As Oracle is no longer just a database company, Oracle Secure Backup provides tape backup for application files as well as the database. Integrated with Recovery Manager (RMAN), Oracle Secure Backup provides the media management layer for RMAN backups to tape. Complete data protection for your entire environment, OSB provides an alternative to expensive media management products increasing ROI for your customers Oracle investment. Using OSB, tape backup management is automated through user-defined policies through the life cycle of the backup tape from first write to tape duplication (if any) to rotation between multiple locations (vaulting) and finally re-use when upon expiration of the tape. File system and database backup may be encrypted per user-defined policy or on a one-off basis using OSB native encryption capability or LTO-4 tape drive encryption….OSB seamlessly manages encryption keys for both native encryption or hardware (LTO-4 drives). OSB is integrated with Enterprise Manager 10gR2…With EM Grid , both OSB file system backup operations, database backup operations and media management can be managed through Grid. In addition to EM, OSB backup management may be managed using a command line interface or native web tool. Oracle database backups provide the fastest database backup to tape (about 25 – 40 % faster) due to optimizations, particularly: Unused block compression: Only used data blocks are backed up (Oracle Database 10g R2) Backup undo optimization: Only "active" undo data is backed up (Oracle Database 11g R1) NOTE: OSB supports 6 platforms and 3 NAS devices. Virtual Tape Library (VTL) Tape Library

Oracle Secure Backup Cloud Module Offsite Database Backups in the Cloud
Database Files / Fast Recovery Area RMAN Oracle Secure Backup Cloud Module Amazon S3 Compression / Encryption Oracle Secure Backup Cloud module: Backup databases to Amazon Cloud Complements local disk and/or tape backup Eliminates IT management overhead of a disaster recovery site Backed by Amazon S3 uptime SLAs $3,500 per RMAN channel More information: Works with Oracle9i R2 and higher DB versions

Logical Data Protection Fast ‘Rewind’ of Logical Errors

Flashback Technologies Error Detection & Correction
Traditional Recovery Flashback Technologies Error Detection & Correction Recovery Time Flashback revolutionizes error recovery View ‘good’ data as of a past point-in-time Simply rewind data changes Time to correct error equals time to make error Flashback Correction Time = Error Time + f(DB_SIZE) Low impact Excellent tool for configuring QA, Dev and Training databases Flashback is easy – simple commands, no complex procedure Flashback enables easy navigation through time See all rows at a given time See all changes to a row See all changes made by a transaction Flashback enables easy correction of errors Row level Table level Database level Flashback applies to all types of users End users Developers Administrators Flashback is much faster and easier than traditional recovery ================= Flashback impact The environment for the OLTP < 2% overhead (actually < 1% in my tests) was a single instance database running the Swingbench Order Entry workload with on Linux 32-bit, RHEL ( ELhugemem) and the I/O subsystem was: Array Serial # Disks Memory EMC Clariion CX700 # Trays - 15 disks per Tray - 73Gb 4Gb per SP (512Mb Read Cache / 2631Mb Write Cache) EMC Clariion CX500 # Trays - 15 disks per Tray - 73Gb 2Gb per SP (365Mb Read Cache / 1107Mb Write Cache) where the cx700 was the data ASM disk group and the cx500 was the ASM Fast Recovery area disk group. So the FRA had 28 spindles. Other qualitative reasons for low FB logging impact: - Only one before-image is logged per 30 min interval, regardless of number of changes to the block - No extra block reads are required when writing to flashback logs (note: except for direct insert loads, which is fixed in 11.1 for single instance) - Only data file block changes are tracked, not all database files (e.g. online redo, controlfile, etc.) - In general, no process needs to wait for flashback log I/O in a well configured OLTP system.

Error Investigation with Flashback
Flashback Query Query all data at point in time select * from Salary AS OF ‘12:00 P.M.’ where … Flashback Version Query See all versions of a row between times See transactions that changed the row Tx 3 select * from Salary VERSIONS BETWEEN ‘12:00 PM’ and ‘2:00 PM’ where … Oracle9i introduced Flashback Query to provide a simple, powerful and completely non-disruptive mechanism for recovering from human errors. It allows users to view the state of data at a point in time in the past without requiring any structural changes to the database. Oracle Database 10g extended the Flashback Technology to provide fast and easy recovery at the database, table, row, and transaction level. Flashback Technology revolutionizes recovery by operating just on the changed data. The time it takes to recover the error is now equal to the same amount of time it took to make the mistake. Flashback Query: Ensure that the database is using an undo tablespace. The setting the UNDO_MANAGEMENT initialization parameter to AUTO specifies this. Set the UNDO_RETENTION initialization parameter (secs, default=900) to a value that causes undo to be kept for a length of time that allows success of your longest query back in time or to recovery from human errors. To guarantee that unexpired undo will not be overwritten, set the RETENTION GUARANTEE clause for the undo tablespace. Flashback Version Query Pseudocolumn Description: VERSIONS_STARTSCN, VERSIONS_STARTTIME - Starting System Change Number (SCN) or TIMESTAMP when the row version was created. This identifies the time when the data first took on the values reflected in the row version. You can use this to identify the past target time for a Flashback Table or Flashback Query operation. If this is NULL, then the row version was created before the lower time bound of the query BETWEEN clause. VERSIONS_ENDSCN, VERSIONS_ENDTIME - SCN or TIMESTAMP when the row version expired. This identifies the row expiration time. If this is NULL, then either the row version was still current at the time of the query or the row corresponds to a DELETE operation. VERSIONS_XID - Identifier of the transaction that created the row version. VERSIONS_OPERATION - Operation performed by the transaction: I for insertion, D for deletion, or U for update. The version is that of the row that was inserted, deleted, or updated; that is, the row after an INSERT operation, the row before a DELETE operation, or the row affected by an UPDATE operation. Note: For user updates of an index key, a Flashback Version Query may treat an UPDATE operation as two operations, DELETE plus INSERT, represented as two version rows with a D followed by an I. The following statement queries the FLASHBACK_TRANSACTION_QUERY view for transaction information, including the transaction ID, the operation, the operation start and end SCNs, the user responsible for the operation, and the SQL code to undo the operation: SELECT xid, operation, start_scn,commit_scn, logon_user, undo_sql FROM flashback_transaction_query WHERE xid = HEXTORAW(' D'); Flashback Transaction Query See all changes made by a transaction Tx 2 select * from FLASHBACK_TRANSACTION_QUERY where xid = HEXTORAW(‘ D’); Tx 1 All above are based on available UNDO

Error Correction with Flashback
Database Flashback Database – restore database to any point in time Flashback Table – restore contents of tables to any point in time (undo-based) Flashback Drop – restore accidentally dropped tables (based on free space in tablespace) Flashback Transaction – back out transaction and all subsequent conflicting transactions (redo-based) Customer Order Flashback Database – restore database to point-in-time Imagine a bad batch job that has impacted your entire database, incorrectly modifying data across many tables and even certain PL/SQL procedures. The only corrective action would be to import a previous good data set, provided you have them, or to restore a good backup and roll the database forward to a point-in-time prior to the time of the batch job. Both can potentially entail long recovery times into the hours or days. Oracle Database 10g features Flashback Database, a unique database point-in-time recovery capability, which allows the database to be ‘rewound’ to a prior point-in-time within seconds or minutes. Because only the affected data blocks are restored and recovered, it’s faster than traditional recovery methods. The command is simply ‘flashback database to my point-in-time’ We accomplish this fast recovery by utilizing flashback logs which record old block versions. When writes are issued to disk and flashback database is enabled, the old block version is written to the flashback log while the new block version is written to datafiles. When a flashback database command is issued, only the changed blocks are retrieved from the flashback logs and then recovered with appropriate archived logs to the required point-in-time. You control how far back to retain flashback logging to support your recovery requirements. For example you might enable flashback database for 24 hrs, and then rely on backups for recovery past 24 hrs. Flashback Table – recover contents of tables to point-in-time (undo-based) Flashback Drop – restore accidentally dropped tables (based on free space in tablespace) Flashback Transaction – back out transaction and all subsequent conflicting transactions (redo-based)

Flashback Database Continuous Data Protection (CDP)
Fast point-in-time recovery strategy Eliminate the need to restore a whole database backup Continuous data protection for database Optimized, before-change block logging Restores just changed blocks Replay log to restore DB to desired time It’s fast - recover in minutes, not hours It’s easy - single command restore Flashback Database to ‘2:05 PM’ Disk Write New Block Version Old Block Version Imagine a bad batch job that has impacted your entire database, incorrectly modifying data across many tables and even certain PL/SQL procedures. The only corrective action would be to import a previous good data set, provided you have them, or to restore a good backup and roll the database forward to a point-in-time prior to the time of the batch job. Both can potentially entail long recovery times into the hours or days. Oracle Database 10g features Flashback Database, a unique database point-in-time recovery capability, which allows the database to be ‘rewound’ to a prior point-in-time within seconds or minutes. Because only the affected data blocks are restored and recovered, it’s faster than traditional recovery methods. The command is simply ‘flashback database to my point-in-time’ We accomplish this fast recovery by utilizing flashback logs which record old block versions. As you can see in the diagram, when writes are issued to disk and flashback database is enabled, the old block version is written to the flashback log while the new block version is written to datafiles. When a flashback database command is issued, only the changed blocks are retrieved from the flashback logs and then recovered with appropriate archived logs to the required point-in-time. You control how far back to retain flashback logging to support your recovery requirements. For example you might enable flashback database for 24 hrs, and then rely on backups for recovery past 24 hrs. Flashback also critical for fast reinstatement of new standby after failover & snapshot standby. “Rewind” button for the Database Data Files Flashback Log

Flashback Technologies New Features Oracle Database 11g Release 2
Increased Availability Enable Flashback Database while database is open Test Flashback without having to take downtime Better Manageability Monitor Flashback Database progress with v$session_longops Progress percentage can be found with (SOFAR / TOTALWORK) Minimize System Impact Optimized Flashback logging for batch/insert intensive loads Potentially reduce Flashback logging impact to ~2% Extended Dependency Tracking Flashback Transaction supports foreign key dependency tracking

Best Practices – Undo-based Flashback Flashback Query, Flashback Table
Use Undo Advisor (available through Enterprise Manager) to get recommendations on available undo retention for various sizes. Use fixed size undo Undo retention automatically tuned for best possible retention based on tablespace size and current system load. Be aware of DDL restrictions – not possible to query in the past if table structure is modified (e.g. drop/modify column, move table, etc.) Further details: Some DDLs that alter the structure of a table, such as drop/modify column, move table, drop partition, and truncate table/partition, invalidate any existing undo data for the table. It is not possible to retrieve data from a point earlier than the time such DDLs were executed. Trying such a query results in error ORA This restriction does not apply to DDL operations that alter the storage attributes of a table, such as PCTFREE, INITRANS, and MAXTRANS. Use DBMS_FLASHBACK package to set Flashback Query SCN at session-level for all ‘selects’ (i.e. no ‘as of’ clause needed) – this is useful for ETL/DW environments where queries that run during the day may also run during nightly batch job, so pre-batch job SCN is set for those sessions to ensure application consistent results.

Best Practices – Flashback Database
Tune FRA storage Use ASM, configure enough disk spindles, etc. Use physical standby database to test Flashback logging Use V$FLASHBACK_DATABASE_LOG to size log space, after running workload > duration of Flashback retention period. Create Guaranteed Restore Point (GRP) without enabling Flashback logging Saves disk space for workloads where same blocks are repeatedly updated Drop GRP to immediately reclaim space Further details: Metalink Note Flashback Database Best Practices & Performance Restore point is a user defined name that is associated with a database point in time used in conjunction with Flashback Database, Flashback Table and RMAN. They can be created in SQLPlus or EM. Guaranteed Restore Point is a special type of restore point that ensures flashback logs are kept until the restore point is used or deleted. When FB logging is turned on, flashback logs are generally created in proportion to archived redo logs generated during same retention period When FB logging is turned off and GRP is created, each changed block is only logged once to maintain GRP vs. continuous logging of blocks with a configured Flashback retention. - E.g. Create GRP prior to nightly batch job for fast recovery of batch issues, then delete GRP next day to reclaim space Flashback retention should be set >= 60 minutes (also for Data Guard Fast-Start Failover environment). This is because Oracle writes a metadata marker (used for FB DB operation) into FB logs every 30 mins and so, setting retention under 60 mins where there is space pressure could delete a needed marker and thus render FB DB unusable for some portion of time. Setting FB retention >= 60 mins guarantees that we will have at least 2 markers always available in FB logs. Maintaining flashback logs imposes comparatively limited overhead on an Oracle database instance. Changed blocks are written from memory to the flashback logs at relatively infrequent, regular intervals, to limit processing and I/O overhead. To achieve good performance for large production databases with Flashback Database enabled, Oracle recommends the following: - Use a fast file system for your Fast Recovery area, preferably without operating system file caching. Files the database creates in the Fast Recovery area, including flashback logs, are typically large. Operating system file caching is typically not effective for these files, and may actually add CPU overhead for reading from and writing to these files. Thus, it is recommended to use a file system that avoids operating system file caching, such as ASM. - Configure enough disk spindles for the file system that will hold the Fast Recovery area. For large production databases, multiple disk spindles may be needed to support the required disk throughput for the database to write the flashback logs effectively. - If the storage system used to hold the Fast Recovery area does not have non-volatile RAM, try to configure the file system on top of striped storage volumes, with a relatively small stripe size such as 128K. This will allow each write to the flashback logs to be spread across multiple spindles, improving performance - For large, production databases, set the init.ora parameter LOG_BUFFER to be at least 8MB. This makes sure the database allocates maximum memory (typically 16MB) for writing flashback database logs. The overhead of turning on logging for Flashback Database depends on the mixture of reads and writes in the database workload. The more write-intensive the workload, the higher the overhead caused by turning on logging for Flashback Database. (Queries do not change data and thus do not contribute to logging activity for Flashback Database.) Flashback related FRA sizing ->

Recovery Analysis Intelligent, Guided Recovery

Data Recovery Advisor The Motivation
Investigation & Planning Oracle provides robust tools for data repair: RMAN – physical media loss or corruptions Flashback – logical errors Data Guard – physical problems However, problem diagnosis and choosing the right solution can be error prone and time consuming Errors more likely during emergencies Data Guard – There are two options using a standby database that can be used to repair block corruption on the primary database: • Extract the rows from the block that is corrupted on the primary by using Data Pump or other means to select the data from a table. The data is then re-inserted into the primary database. • Copy the standby database datafile(s) to the primary database. Once the file is restored on the primary database, archived logs are applied to bring it consistent with the rest of the database. If the primary database corruption is widespread due to a bad controller or other hardware/software problem, then you may want to switchover to the standby database while repairs to the primary database server are made. Recovery

Data Recovery Advisor (DRA)
Oracle Database tool that automatically diagnoses data failures, presents repair options, and executes repairs at the user's request Determines failures based on symptoms E.g. an “open failed” because datafiles f045.dbf and f003.dbf are missing Failure Information recorded in diagnostic Automatic Diagnostic Repository (ADR) Flags problems before user discovers them, via automated health monitoring Intelligently determines recovery strategies Aggregates failures for efficient recovery Presents only feasible recovery options Indicates any data loss for each option Can automatically perform selected recovery steps Accessed via RMAN or EM Data Recovery Advisor can diagnose failures such as the following: Components that are not accessible because they do not exist, do not have the correct access permissions, are taken offline, and so on Physical corruptions such as block checksum failures, invalid block header field values, and so on Logical corruptions caused by software bugs Incompatibility failures caused by an incorrect version of a component I/O failures such as a limit on the number of open files exceeded, channels inaccessible, network or I/O error, and so on Configuration errors such as an incorrect initialization parameter value that prevents the opening of the database Intelligently determines recovery strategies Aggregates failures for efficient recovery – propose single script with appropriate commands in order, to cover all failures if possible vs. multiple scripts Presents only feasible recovery options, e.g. propose RMAN restore script only if needed backups are available. Can also check for available Flashback logs and standby database (in Grid Control) for recovery options. Indicates any data loss for each option, e.g. if point-in-time recovery is only option (due to missing backups or logical error) Reduces downtime by eliminating confusion

Data Recovery Advisor Wizard

Data Recovery Advisor – View Failures

Data Recovery Advisor – Manual Repair

Data Recovery Advisor – Recovery Advice

Data Recovery Advisor – Summary

Putting It All Together.. Customer Example
Requirement Service Level Agreement RPO RTO Tier 3 Tier 2 Tier 1 Disaster Recovery Retention Policy Backup Redundancy Oracle Solution Archived Log Mode RMAN, OSB, DRA Flashback Table Flashback Database Data Guard OSB Fast Recovery Area, OSB Any point in time within recovery window <1 hour for tablespace/datafile recovery <3 hours for full database recovery <30 min for row/table recovery (within last 3 hrs) <1 hour for database recovery from logical errors (within last 2 hrs) <15 min for any database outage Failover to standby database at secondary site Backups sent offsite Onsite backups - 3 day recovery window Offsite backups - 1 year tape retention Two backup copies on tape

Recovery SLAs Customer Example
Oracle Solution - RMAN + OSB + Data Guard + DRA One-time image copy backup to Fast Recovery Area (FRA) Daily differential incremental backup to FRA Image copy rolled forward daily until “sysdate – 4” FRA sized for one image copy backup + 4 incrementals + 4 days of archived logs Daily backup of FRA to tape via OSB (retained for 1 month) Daily vaulting of tape backups to offsite location (retained for 1 year) Real-time, synchronized physical standby database in Maximum Performance mode for disaster recovery Leverage DRA for real-time detection and analysis of failures

Recovery SLAs Customer Example
Oracle Solution – Flashback Technologies Size UNDO tablespace for 3 hour retention period Set Flashback Database target retention time to 2 hours Provision Flashback log space in FRA, based on 2 hour workload RMAN channel represents a single backup file stream. A channel can read multiple datafiles or archived logs to backup into a multiplexed backup set, or can read one file at at a time for image copy backup. Increasing RMAN channels increases backup parallelism, thereby potentially reducing overall backup time. RMAN multiplexing is the number of datafiles or archived logs read by one channel at any time. The default is min (FILESPERSET, MAXOPENFILES). FILESPERSET defaults to 64 and specified at BACKUP command. MAXOPENFILES defaults to 8 and is specified at CONFIGURE CHANNEL command. So multiplexing=8 by default, which means a maximum of 8 files will be read by one channel at any time. For SAME/ASM storage, MAXOPENFILES should be set to 1, since all files are striped across available disks and reading one file/channel will optimally read from each disk.

Agenda <Insert Picture Here> What Keeps You Awake at Night? Oracle Data Protection Planning & Solutions Oracle Backup & Recovery Solutions Physical Data Protection Recovery Manager Oracle Secure Backup Logical Data Protection Flashback Technologies Recovery Analysis Data Recovery Advisor Putting it All Together – Customer Example Turkcell Backup & Recovery Case Study Q&A

Remember? Data Protection Concerns…
Meeting recovery SLAs? Reducing exposure to data loss? Meeting backup windows? Dealing with long-term backup storage? Management complexity? Budget? Solution…

Oracle Backup & Recovery Solutions Complete & Targeted Recovery
Recovery Analysis Data Recovery Advisor Logical Data Protection Flashback Technologies Physical Data Protection Recovery Manager Oracle Secure Backup Leverage Integrated Oracle Backup & Recovery Solutions Physical Data Protection Recovery Manager Comprehensive physical backup and recovery Oracle Secure Backup Centralized tape backup for database and file system data Fastest Oracle database backup to tape Low-cost, per-tape drive licensing Innovative database backup to Cloud storage Logical Data Protection Flashback Technologies Fast, point-in-time database ‘rewind’ Fine-grained recovery at row, transaction, and table Recovery Analysis Intelligent, guided recovery with Data Recovery Advisor

OTN Resources Recovery Manager:
Oracle Secure Backup Flashback Technologies Oracle Cloud Computing Center Oracle Maximum Availability Architecture

HA Sessions, Labs, & Demos by Oracle Development
Sunday, 11 October – Hilton Hotel Imperial Ballroom B 3:45p Online Application Upgrade Monday, 12 October – Marriott Hotel Golden Gate B1 11:30a Introducing Oracle GoldenGate Products Monday, 12 October – Moscone South 1:00p Oracle’s HA Vision: What’s New in 11.2, Room 103 4:00p Database 11g: Performance Innovations, Room 103 2:30p Oracle Streams: What's New in 11.2, Room 301 5:30p Comparing Data Protection Solutions, Room 102 Tuesday, 13 October – Moscone South 11:30a Oracle Streams: Replication Made Easy, Room 308 11:30a Backup & Recovery on the Database Machine, Room 307 11:30a Next-Generation Database Grid Overview, Room 103 1:00p Oracle Data Guard: What’s New in 11.2, Room 104 2:30p GoldenGate and Streams - The Future, Room 270 2:30p Backup & Recovery Best Practices, Room 104 2:30p Single-Instance RAC, Room 300 4:00p Enterprise Manager HA Best Practices, Room 303 Tuesday, 13 October – Marriott Hotel Golden Gate B1 11:30a GoldenGate Zero-Downtime Application Upgrades 1:00p GoldenGate Deep Dive: Architecture for Real-Time Wednesday, 14 October – Moscone South 10:15a Announcing OSB 10.3, Room 300 11:45a Active Data Guard, Room 103 5:00p Exadata Storage & Database Machine, Room 104 Thursday, 15 October – Moscone South 9:00a Empowering Availability for Apps, Room 300 12:00p Exadata Technical Deep Dive, Room 307 1:30p Zero-Downtime DB Maintenance, Room 103 Demos Moscone West DEMOGrounds Mon & Tue 10:30a - 6:30p; Wed 9:15a - 5:15p Maximum Availability Architecture (MAA), W-045 Oracle Streams: Replication & Advanced Queuing, W-043 Oracle Active Data Guard, W-048 Oracle Secure Backup, W-044 Oracle Recovery Manager & Flashback, W-046 Oracle GoldenGate, 3709 Hands-on Labs Marriott Hotel Golden Gate B2 Monday 11:30a-2:00p Oracle Active Data Guard, Parts I & II Thursday 9:00a-11:30a Oracle Active Data Guard, Parts I & II

Turkcell Backup & Recovery Strategy
Hüsnü Şensoy Turkcell Telecommunication Services VLDB Expert Oracle ACE Director Member of Global DWH Leaders & Oracle CAB Oracle Magazine Editors’ Choice Award DBA of the Year 54

Agenda Backup & Recovery Strategies for Oracle Databases
Motivation behind those strategies Revisiting “Incrementally Updated Backup” Revisiting “FRA” How to bring your database back without restore ? Sick backup will not work Centralized scheduling & monitoring 11g Release 2 Backup & Recovery New Features with real Telco data warehouse data Brand new compression algorithms Summary

Turkcell Overview Leading GSM operator of Turkey established in February 1994. Third GSM operator in Europe in terms of subscriber (+36 million). First and only Turkish company ever to be listed on New York Stock Exchange. Member of Board of Directors of GSMA since 2003. 25th company of INFOTECH 100 list.

Backup & Recovery Strategies for Oracle Databases
Turkcell Backup & Recovery Strategy Backup & Recovery Strategies for Oracle Databases

Design Considerations
Define your backup & recovery policies upfront A well documented strategy that can be used to bring everything back KISS: Even a junior DBA should be able to bring your database back. Standardize, standardize, standardize… Be prepared to justify the cost in terms of business impact of downtime

Proactively validate database and backup integrity Physical errors Logical inconsistencies Transmission errors Do you perform regular full recoveries to separate host and storage?

Centralized backup reporting: Is there a single point of access for all my databases’ backup logs ? What is the average backup duration for database X ? How do brand new tape drives affect backup performance ?

What Type of Architecture ?
What’s in there ? 7 RAC databases More than 20 services 20 Gbit/s 120 Intel Cores 640 GB Memory APPDB VASCMT VASSE VASNIF BSSOSS VASRES BSSARCH 25 TB DATA FRA ARCHIVE

How Do We Backup ? Incrementally Updated Backup Strategy
Initial image copy backup to FRA Fast incremental backups thereafter Image copy is rolled forward with incremental backup on regular basis to create full on-disk backup Full database backup times only depend on the amount of blocks changed since last incremental backup. The longest backup time is only ~30 minutes, with ZLIB backup compression and logical block checking turned on. run{ backup as compressed backupset check logical incremental level 1 for recover of copy with tag DAILY_COPY database filesperset 1; recover copy of database with tag DAILY_COPY; } This is the shortest, cleanest, and most elegant backup script that I have seen in all my years at Turkcell.

Setting Up Flash Recovery Area (Oracle Database 11g Release 1)
Self managed & organized logical storage area. Setup as part of Universal Installer wizard. Redo log copy, control file copy, archived logs, and Flashback logs are automatically stored there. RMAN automatically utilizes FRA for all disk backups. Or, just enable it by setting two init.ora parameters : db_recovery_file_dest_size db_recovery_file_dest

Flash Recovery Area ASM is the best infrastructure to be used as FRA destination: Raw device performance. No other solution (except Sun ZFS file system with its online FS check capability) will practically let you implement large storage pools as ASM does. Ease of management. ASM allows you to provision the same diskgroup to multiple FRA destinations. DB1 FRA DB2 FRA DB3 FRA DB4 FRA ASM Diskgroup (+FRA)

Restore-Free Recovery
Create a pfile whose control_file parameter just points the FRA copy of controlfile Mount DB Switch database to copy Recover database Open database

What Are the Commands? From hours to minutes Step1 Step 2 Step 3
SQL> startup pfile=/home/oracle/init.ora nomount; ORACLE instance started. … Step 2 RMAN> switch database to copy; using target database control file instead of recovery catalog datafile 1 switched to datafile copy "+FRA/disaster/datafile/system " datafile 9 switched to datafile copy "+FRA/disaster/datafile/undotbs " Step 3 RMAN> recover database; Starting recover at 07-FEB-09 using channel ORA_DISK_1 starting media recovery media recovery complete, elapsed time: 00:00:03 Finished recover at 07-FEB-09 Step 4 RMAN> alter database open; database opened From hours to minutes

Backup Validation Backups on disk or tape might be damaged due to
Physical problems on media (fabric problems, dust, cosmic rays, etc) Media library errors (error in checksum computation) How you can increase the probability that your backups are healthy ?

Possible Solutions Prevent Errors at Backup Time
Protective action for possible problems. It will slightly effect your backup time (check logical) Multiplex Backups Good if the error is a temporary/stochastic one. It has an additional cost of time and media. Probe Backups Cost effective method for the majority of the problems Not 100% coverage for any kind of errors. Restore them Most effective one among all Most costly one among all.

RMAN Backup Validation
RMAN> backup check logical validate datafilecopy all filesperset 1; This will report For any inconsistent data, index, or other type of blocks. Number of total and empty blocks examined. Highest change number of each datafile copy.

Centralized Scheduling & Monitoring
Develop standard backup job scheduling and monitoring routines. This enables you to: See all backup schedules at once Check details of previously completed backups (duration, logs, etc.) Easily modify backup scripts and bulk deploy them.

Grid Control Backup Jobs
Manage backup of all databases of the cluster by using just one screen

ORACLE DATABASE 11g RELEASE 2 RMAN COMPRESSION
Turkcell Backup & Recovery Strategy ORACLE DATABASE 11g RELEASE 2 RMAN COMPRESSION

11g Release 2 RMAN Compression
Pre-Compression Block Processing Binary Compression Basic Advanced HIGH MEDIUM LOW

Test Setup Hardware Sun Solaris 10
CPU: 8 Intel Xeon 3.00GHz/processor Memory: 16 GB HBA: 2x 2Gbit QLogic HBA Data Marketing data from Turkcell data warehouse 2.2 billion records (140G) No segment compression PCTFREE 1 16K block size tablespace Number of Channels 8 RMAN Channels Compression Types No compression BASIC LOW MEDIUM HIGH Collected Metrics Compression Ratio Duration I/O Throughput CPU Utilization Need machine specs

Test Summary In Oracle Database 11g Release 2, RMAN extends its compression capabilities to fit any CPU power and I/O throughput combination. MEDIUM compression level can backup faster than BASIC with similar compression ratio and 3X faster with 50% less CPU utilization. Even if you don’t have need to reduce backup sizes, LOW/MEDIUM compression level might be faster than uncompressed backup depending on your I/O throughput, by significantly reducing the amount of data/sec written by RMAN.

Best Practices Summary
A well defined, documented, standard, manageable, and fast backup & recovery strategy is a MUST if you manage tens (even hundreds) of databases. Whatever solution you pick, the indicator of a good backup & recovery strategy is simple: It shouldn’t depend on the size of database. FRA over ASM and RMAN satisfies these requirements with zero cost.

Oracle Database Backup-and-Recovery Best Practices and New Features

Similar presentations

Presentation on theme: "Oracle Database Backup-and-Recovery Best Practices and New Features"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Oracle Database Backup-and-Recovery Best Practices and New Features

Similar presentations

Presentation on theme: "Oracle Database Backup-and-Recovery Best Practices and New Features"— Presentation transcript:

Similar presentations

About project

Feedback