1. 1 Recovery and Backup RMAN TIER 1 Experience, status and questions. Meeting at CNAF June 12-13 of 2007, Bologna, Italy Carlos Fernando Gamboa, BNL Gordon L Brown, RAL

2. 3D LCG Database Administration workshop at CNAF - June 12-13, 20072 Table of Content Factors that define a backup policy Backup strategy (from previous DBA SARA workshop) Site status RMAN configuration RMAN implementation problems Experience Questions Example (Presented by Gordon L Brown)

3. 3D LCG Database Administration workshop at CNAF - June 12-13, 20073 Factors that define a backup policy Maximum % of data that can be lost - Retention Policy: How long are the backups retained and how far is acceptable to recover the database? - How many copies per file must be retained? Time required to recover the service - Faster recovery through disk media but more expensive than tape. - Minimum % downtime of the service during recovery Cost/Benefit of backup policy and recovery strategy Site resources

4. 3D LCG Database Administration workshop at CNAF - June 12-13, 20074 CERN backup strategy RMAN - backups to tape Level 0, every 2 weeks level 1 cumulative, twice per week level 1 differential, every day (except when the cumulative backup is done) archivelogs backups, every 30 minutes Retention: recovery window of 31 days (may change) RMAN - backups to disk: Daily refreshed with incremental recovery Image copy delayed from production (2 days) Allows for very fast recovery for many failure scenarios Regular tests of recovery procedure Open point: disaster recovery / dataguard From: Gordon D. Brown e-Science, RAL3D Workshop, SARA, Netherlands 20/21 March 2007

5. 3D LCG Database Administration workshop at CNAF - June 12-13, 20075 Summary Sites RMAN implementation SiteCurrent StatusIncremental Level 0 Incremental level 1 cumulative Incremental level1 differential Archivelogs BNL-Media:Disk -Waiting for disk storage equipment (IBM DS3400 3.6TB ) -Evaluating Tape resources -Retention Policy 7 days Weekly2 days/week3 days/week (planned) Hourly PIC-Media:Disk -Waiting DBA workshop’s input before deploying Bi weekly (Planned) Daily (Planned) Every 30 minutes (Planned) FTS already deployed Weekly1 day/week (Thursday) 2 days/week (Friday and Saturday) -Full Saturday -Hourly ASGCFull Backup weeklyIncremental (after may) Not define time NDGF-Media: Disk -Full backup DailyEvery hour CNAF-Media: Disk -Monthly full backup (some cases) -Catalog database which they stored the script executed by the target databases. Weekly - kept for 3 weeks Daily -kept for 3 weeks -Kept for 3 days

6. 3D LCG Database Administration workshop at CNAF - June 12-13, 20076 SiteCurrent StatusIncremental Level 0 Incremental Level 1 cumulative Incremental Level1 differential Archivelogs GRIDKA-Catalog in control file with keep time 31 -Control scripts on RMAN logs to monitor backup - Installation of own GridControl on a separate OEM- Server -Schedule all RMAN Backups from this Server -Use of Tivoli Data Protector for Oracle to write directly to tape is not fully discussed. -Use of Nagios for monitoring backup -Weekly-Daily -Daily? TRIUMF-Backup to disk: Planning to migrate to tape on late summer -The backup server is auto mounted on the nodes of the RAC. -NFS daemon configured to avoid concurrent backups of other system when RMAN is backing up. -Weekly-Planned using-Planned RAL- Use a central repository for storage of the backup/restore control data -Backups are validated daily. Team is notified if the validation of the backup fails on a daily report. -Backups are kept on disk for two weeks and the oldest is replaced after the new backup is successfully validated. -Daily secure of backups are copied to tape. -RMAN jobs run as CRON jobs "daily report" job as a scheduled job in Grid Control. -Weekly-Daily Summary sites RMAN implementation

7. 3D LCG Database Administration workshop at CNAF - June 12-13, 20077 RMAN Implementation Problems Control file block problem An archivelog file was not found on the control file. Different archivelog format on each node of the RAC. Service interruption to change archivelog destination.

8. 3D LCG Database Administration workshop at CNAF - June 12-13, 20078 Sites experience with RMAN RMAN deployment on FTS. RMAN basic configuration through OEM. Most of the sites are validating RMAN scripts for the 3D database backup and learning how to use this tool. TRIUNF experience with compression - It seems too heavy on the oracle RAC node, and does not seem to compress really efficiently. - Running bzip2 on a 1.9 GB compressed RMAN file results in a file that is further compressed to 1.6 GB.

9. 3D LCG Database Administration workshop at CNAF - June 12-13, 20079 Sites Experience with RMAN (cont.) Backup process “Testing is tedious; with our currently-configured backup server a full backup takes almost 2 hours”. CNAF has experienced restore with single instance with local file system (ext3) and with ASM for the recovery area in remote storage system.

10. 3D LCG Database Administration workshop at CNAF - June 12-13, 200710 Questions Integrate RMAN / TIVOLI Cost / benefit ratio? OEM - Cron jobs to implement backups or configure them by using OEM? - How to configure OEM to send alerts when service is down? RMAN implemented as cron jobs - Can cron jobs use ASM disk as backup destination when using RMAN scripts? - Is it necessary to implement RMAN scripts on each node to archivelog and delete them?

11. 3D LCG Database Administration workshop at CNAF - June 12-13, 200711 Questions (cont.) How important is to implement an independent RMAN catalog Db? Is there any reason to limit the size of an RMAN-generated file for disk-based backup? Are there any backup options that we should use that help to optimize RMAN restores? What is the recommended number of backups that sites retain? Has someone experience to run grid control as a VMWare Virtual Machine? Examples for use dmbs_scheduler to manage RMAN jobs node- independent on different Oracle RAC?

12. 3D LCG Database Administration workshop at CNAF - June 12-13, 200712 Thanks to the people that contributed to this presentation. Luis Diaz Denice Deatrich Dave Wei Federico Bonifazi Olli Tourunen Doris Wochele Gordon L. Brown