1. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Backup & Recovery with RMAN LCG 3D Workshop, Bologna June 12 th, 2007 Jacek Wojcieszuk LCG

2. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 2 Agenda Types of failures and backups in Oracle RMAN Architecture Manual vs. RMAN backups On-tape backups with RMAN RMAN Configuration RMAN backup strategies RMAN backups at Tier0 Exemplary recovery scanarios Demo

3. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 3 Types of failures Instance Failure –Usually connected with an Oracle process failure Media Failure –Disk failure, storage array controller failure etc. Block Corruption –Usually caused by bugs in Oracle software Human error –In most cases accidentally deleted/updated data –Database user or DBA Disaster –Fire, flood, earthquake, plane crash etc.

4. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 4 Backup options in Oracle Physical backups –Cold (off-line) backups Full database only Require downtime Do not provide flexibility for point in time recovery (PiTR) –Hot (on-line) backups Different types of backups: full, incr. (cumulative, differential), archivelogs Different scopes: full database, tablespace(s) or datafile(s) Do not require database downtime Can be used to recover full database, single/multiple tablespace(s)/datafile(s) or a corrupted block Database can be recovered to any point in time within assumed backup retention period

5. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 5 Backup options in Oracle (2) Logical backups –Logical copy of data in the database –Support for different backup granularity –Can be taken either with legacy Export/Import tools or with Data Pump (10g) Standby systems (Data Guard) –Physical and logical standby databases

6. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 6 How hot backups are possible? SGA Server proc. DBWRLGWR RedoLog Db file Log archiver Archived redo logs Backup proc. Tapes or disks system

7. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 7 RMAN architecture Target Database RMAN Client Target Database Target Database Auxiliary Database RMAN Catalog Schema Catalog Database

8. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 8 Types of RMAN hot backups Copy or backupset Full database backup Incremental backups (in 10g 2 levels available: 0 and 1) –Cumulative, differential –Can be used to update a copy of the database Archivelog backups Tablespace(s), datafile(s) backups Differential incremental Cumulative incremental

9. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 9 Manual vs. RMAN backups RMAN advantages: –Supports incremental backup strategies –RMAN on-line backups are not so heavy for the system as manual on-line backups –RMAN can detect corrupted blocks –RMAN automatically track database structure changes –Provides easy, automated backup, restore and recovery operations –Keeps invenotory of taken backups –Can seamlessly work with third party media managers Disadvantage: something new to learn –RMAN concepts and command syntax sometimes are not intuitive

10. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 10 On-tape backups with RMAN RMAN allows to take on-disk backups out of the box –Flash recovery area, if configured, further simplifies such backups –On disk backups are interesting but usually not sufficient for a disaster recovery On-disk backups can be manually sent to tapes –Recovery can be very troublesome RMAN can seamlessly work with third party Media Managers –Media Manager Library (MML) is required –Different configuration tasks for different MMLs

11. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 11 On-tape backups with RMAN (2) Many vendors of Media Management software provide MMLs Most popular are: –Tivoli Storage Manager –Veritas NetBackup –EMC NetWorker Oracle Secure Backup –Very new and probably not yet reliable enough

12. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 12 RMAN Configuration RMAN can be preconfigured –Configuration is stored in the control file and in the recovery catalog (if used) –Can facilitate backup automation Most useful settings: SettingDefaultRecommended Controlfile autobackupoffon Retention policyto redundancy 1to recovery window of 31 days Device typedisk parallelism 1...disk|sbt prallelism 2... Default device typeto diskto sbt Backup optimizationoff Channel device typenonesbt parms=‘...’ Maxsetsizeunlimiteddepends on your database size

13. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 13 RMAN Configuration (2) Example: configure RETENTION POLICY TO RECOVERY WINDOW OF 31 DAYS; configure DEFAULT DEVICE TYPE TO 'sbt'; configure DEVICE TYPE 'sbt' PARALLELISM 2; configure CHANNEL DEVICE TYPE ‘sbt’ parms='ENV= (TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin/tdpo.opt)'; configure DEVICE TYPE DISK PARALLELISM 2; configure MAXSETSIZE TO 200 G; configure archivelog backup copies for device type 'sbt' to 1; configure controlfile autobackup on; The SHOW ALL command lists all RMAN configuration settings To clear a given settings append CLEAR at the end of the CONFIGURE command

14. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 14 RMAN backup strategies RMAN allows many types of backups It possible to build own backup strategy that suits given database best There are also Oracle recommended backup strategies: –Incremental backup strategy: level 0 backups + level 1 backups (cumulative and/or differential) and archivelog backups inbetween. –Incrementally updated database copy: 1 backup as copy of the whole database + incremental backups used to update the copy + archived redo logs.

15. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 15 Backup strategy used at Tier0 Both Oracle-recommended strategies implemented for all production systems Incremental backup strategy: –Backups go to tapes –Weekly or biweekly level 0 backups (depending on the DB size) –A level 1 cumulative backup inbetween –Daily incremental level 1 differential backups –Archivelog backup every 30 minutes Incrementally updated DB copy strategy: –daily incremental differential backups applied with 2 days of delay –Copies, incremental backups and archived redo logs stored in the Flash Recovery Area

16. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 16 Backup operations at Tier0 Central machine to schedule and run all the backups Central RMAN catalog exported on regular basis Examples of RMAN commands being used: run { crosscheck archivelog all; backup force tag ‘some_tag_0T' incremental level 0 check logical database force format '%d_%T_%U_lvl0T' plus archivelog format '%d_%T_%U_lvl0Tarch'; delete noprompt force archivelog all completed before „sysdate - 2"; } run { crosscheck archivelog all; backup tag ‘some_tag_1D' incremental level 1 database format '%d_%T_%U_lvl1D' plus archivelog format '%d_%T_%U_lvl1Darch'; delete noprompt force archivelog all completed before ‘sysdate – 2’; }

17. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 17 Backup operations at Tier0 (2) run { backup tag ‘some_tag_AR' archivelog all format '%d_%T_%U_arch'; delete noprompt force archivelog all completed before ‘sysdate - 2’; } run { backup maxsetsize 2047G tag ' DB_Copy_tag' device type disk incremental level 1 for recover of copy with tag ‘DB_Copy_tag’ database; recover device type disk copy of database with tag DB_Copy_tag until time ‘sysdate -2 ‘; delete force noprompt backupset device type disk tag DB_Copy_tag completed before ‘sysdate -2’; }

18. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 18 Complete database recovery Needed when: –All datafiles are lost or the SYSTEM tablespace datafiles are lost –At least one member of each redo log group survived Requires: –Control file recovery (if it’s lost) –Datafile restore from a backup –Database recovery using incremental backups and/or archived redo logs and online redo logs startup mount run { allocate channel c1 device type disk|sbt; allocate channel c2 device type disk|sbt; restore database check readonly; recover database; alter database open; }

19. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 19 Database point-in-time recovery Needed when: –all datafiles are lost –All copies of the current control file are lost –Or all online redo log group members are lost The most typical recovery in case of systems implementing SAME approach If done after a disaster it has to be preceded by: –Hardware configuration –OS and Oracle software installation –Re-creation or restore from non-RMAN backup of listener.ora, tnsnames.ora and other important configuration files –ASM instance and diskgroup configuration (if needed) –MML installation and configuration –... Requires –Spfile restore –Controlfile restore –Datafiles restore and recovery

20. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 20 run { set until time = "to_date('24-AUG-2006 00:00:00','dd-mon-yyyy hh24:mi:ss')"; allocate channel c1 device type disk|sbt; allocate channel c2 device type disk|sbt; restore database check readonly; recover database; alter database open resetlogs; } set dbid=xxxxxxx startup nomount; run { allocate channel c1 device type disk|sbt; restore spfile to ‘some_location’ from autobackup; recover database; alter database open resetlogs; } shutdown immediate; startup nomount; run { allocate channel c1 device type disk|sbt; restore controlfile from autobackup; alter database mount; }

21. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 21 Tablespace point-in-time recovery Needed –Mainly to address a human error Oracle makes efforts to automate it –Can be done with few clicks in OEM Requires –Point in time recovery of the whole database (an auxiliary instance is created) –Export/import of selected tablespaces schemas or objects

22. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 22 Block media recovery Needed when: –Database reports either single or multi block corruption Can be done with an open database Only the table(s) containing corrupted blocks are not available Database corruptions can be discovered with RMAN backup validate database command Corrupted blocks can be found in V$DATABASE_BLOCK_CORRUPTION run { allocate channel c1 device type disk|sbt; blockrecover datafile 19 block 44; } ORA-01578: ORACLE data block corrupted (file # 19, block # 44) ORA-01110: data file 19: ‘d:\oracle\oradata\data\mydb_maintbs_01.dbf’

23. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 23 Single/multiple datafile/tablespace recovery Needed when –Single/multiple tablespaces or datafiles have been lost –SYSTEM tablespace is intact –Controlfiles and online redo logs are intact Requires –To put offline datafiles and tablespaces being recovered –The database can be open and available to users run { sql „alter tablespace users offline”; allocate channel c1 device type disk|sbt; restore tablespace users; recover tablespace users; sql „alter tablespace users online”; }

24. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 24 Demo

25. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 25 Hands on preparation https://twiki.cern.ch/twiki/bin/view/PSSGroup /HandsOnExerciseshttps://twiki.cern.ch/twiki/bin/view/PSSGroup /HandsOnExercises Please start a full backup of your database, if possible use a tape system

26. CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LCG 3D Workshop, Bologna, June 2007 - 26 More info Oracle Database 10g RMAN Backup & Recovery (by Mathew Hart and Robert G. Freeman) Oracle Documentation –Backup and Recovery Basics –Backup and Recovery Advanced User's Guide –Backup and Recovery Reference