2. Oracle ZFS Storage Protecting Exadata Databases
Greg Drobish
Application Integration Engineering

3. Session Agenda 1
Exadata Overview Data Protection with ZFS Storage Oracle Recovery Manager (RMAN) Configuration Best Practices 2 3 4

4. Exadata Architecture
Complete | Optimized | Standardized | Hardened Database Platform
Standard Database Servers
8x 2-socket servers  192 cores, 2TB DRAM or
2x 8-socket servers  160 cores, 4TB DRAM
Unified Ultra-Fast Network
40 Gb InfiniBand internal connectivity  all ports active
10 Gb or 1 Gb Ethernet data center connectivity
Scale-out Intelligent Storage Servers
14x 2-socket servers  168 cores in storage
168 SAS disk drives  672 TB HC or 200 TB HP
56 Flash PCI cards  44 TB Flash + compression
Fully Redundant

5. Exadata Database Machine
Most advanced hardware
Fully scale-out servers and intelligent storage with unified InfiniBand connectivity and PCI flash
Most advanced software
Database optimized compute, storage, and networking algorithms dramatically improve performance and cost
The ultimate platform for all database workloads
OLTP, Warehousing, Database as a Service
Standardized, optimized, hardened end-to-end

6. Pre-built and Optimized Out-of-the-Box
100%
Custom Configuration
Test & debug failure modes
Measure, diagnose, tune and reconfigure
Performance Achievement
Performance Achievement
Multi-vendor finger pointing
Assemble dozens of components
Time (Days)
Time (Months)

7. Exadata Data Protection with the Oracle ZFS Storage Appliance

8. ZFS Storage: An Ideal Solution
Superior Performance
Enhanced Reliability
Extreme Network Bandwidth
Powerful Features
Simplified Management
Cost Effective Configuration

9. Oracle ZS3-4 delivers world-record performance
17,224 SPC-2 MBPSTM
$22.5 SPC-2 Price PerformanceTM
15,423 SPC-2 MBPSTM
$134.2 SPC-2 Price PerformanceTM
IBM DS8870
13,147 SPC-2 MBPSTM
$95.3 SPC-2 Price PerformanceTM
VSP
13,147 SPC-2 MBPSTM
$131.2 SPC-2 Price PerformanceTM
P9500 XP
Results as of September 10, 2013, for more information go to SPC-2. Results for Oracle ZFS Storage ZS3-4 are 17, SPC-2 MBPS™, $22.53 SPC-2 Price-Performance. Full results  at Results for IBM DS8870 are 15, SPC-2 MBPS, $ SPC-2 Price-Performance. Full results at ww.storageperformance.org/results/benchmark_results_spc2#b Results for Hitachi VSP are 13,147 SPC-2 MBPS, $95.38 SPC-2 Price-Performance. Full results at Results for HP P9500 XP Disk Array are 13, SPC-2 MBPS, $88.34 SPC-2 Price-Performance. Full results at

10. Oracle ZFS Storage Appliance for RMAN Backup Benefits
Performance
Capacity
Simplification
World Record Performance
DTrace Analytics
Hybrid Columnar Compression
Immensely Scalable Architecture
Snap Management Utility
Enterprise Manager Integration

11. Oracle Recovery Manager (RMAN)

12. Oracle Data Protection Technologies All About RTO & RPO
Technology
Protection Against Type of Failure
Downtime Recovery Time Objective , RTO
Data Loss Exposure Recovery Point Objective , RPO
Oracle Secure Backup
Physical
Hours/Days
Days/Hours
(from last backup)
Recovery Manager (RMAN)
Hours (from last backup)
Flashback Technologies
Logical
Minutes/Hours
(from point-in-time)
Data Guard
Site Failure / DR
Seconds/Minutes
Zero/Seconds

13. What is Recovery Manager (RMAN)?
Comprehensive built-in backup & recovery tool for Oracle Databases
Operations
Backup: Backup complete or partial Oracle database components
Restore: Restore complete or partial components from the backup (depends on the type of failure)
Recover: Make the restored data consistent
Smallest Database Unit
Data Block
Extent  Segments  Tablespaces  Data Files (Physical)

14. RMAN Backup Options Image Copy Backup Backup Set Backup
RMAN disk destination only
Block to block copy of database except for temp files
Level 0 backup (Base Image) used with an Incrementally Updated Backup Strategy (synthetic fulls)
Cannot be compressed or encrypted by RMAN
Most common use cases:
Incrementally Updated Backup Strategy
Cross platform migration
ZFS cloning solution (Snap Management Utility)
Backup Set Backup
RMAN disk or tape destinations
Most widely deployed backup type
Can leverage RMAN compression or encryption capabilities
Smaller backup size than an image copy backup due to RMAN null and unused block compression
Most common use cases:
Full / incremental backup strategy
Tape or cloud backups
Long-term retention on disk
IO considerations, RMAN backups should be able to fully utilize read I/O (whether copies or backup sets). There is addl IO on the incremental update operation, but it's usually minimal relative to the full & incrementals.

15. RMAN Incremental Backup Strategies
Built-In Backup Deduplication
Incremental Forever Strategy
Uses Image Copy method
Initial RMAN image copy backup to disk
Subsequently, backup only the changes (incremental)
Merge the incremental with the base image
Allows “SWITCH TO COPY” capability
On-disk retention of 1-7 days
RMAN compression applicable to the incremental and archived log backups
RMAN “BACKUP RECOVERY AREA” or BACKUP COPY OF to make addition copies
Full / Incremental Strategy
Uses Backup Set method
RMAN backup set backup operation to:
Disk - retention from days to weeks
Tape – retention from weeks to years
Cloud – retention from weeks to years
RMAN compression applicable to full , incremental and/or archived log backups
RMAN “BACKUP BACKUPSET” to make additional copy of the backup to disk or tape
RMAN can create multilevel incremental backups. Each incremental level is denoted by a value of 0 or 1. A level 0 incremental backup, which is the base for subsequent incremental backups, copies all blocks containing data. You can create a level 0 database backup as backup sets or image copies.
The only difference between a level 0 incremental backup and a full backup is that a full backup is never included in an incremental strategy. Thus, an incremental level 0 backup is a full backup that happens to be the parent of incremental backups whose level is greater than 0.
A level 1 incremental backup can be either of the following types:
A differential incremental backup, which backs up all blocks changed after the most recent incremental backup at level 1 or 0
A cumulative incremental backup, which backs up all blocks changed after the most recent incremental backup at level 0
Incremental backups are differential by default.
Note:
Cumulative backups are preferable to differential backups when recovery time is more important than disk space, because fewer incremental backups must be applied during recovery.
The size of the backup file depends solely upon the number of blocks modified, the incremental backup level, and the type of incremental backup (differential or cumulative).
High Availability
Space Optimization

16. Exadata Backup with ZFS Storage Configuration Best Practices

17. ZFS Storage Appliance Backup Test Results
Clustered 7420
1 TB DRAM (512GB per controller)
InfiniBand Connectivity
ZFS Storage OS8 Software
No Read Cache SSD
Controllers
8 High Capacity Disk Shelves
7200 RPM drives
No Write Log SSD
Enclosures
Peak Sustained Backup:
23 TB/hr (backupset)
26 TB/hr (imagecopy)
Peak Sustained Restore:
15 TB/hr (backupset)
17 TB/hr (imagecopy)

18. Direct NFS
Oracle Intelligent Storage Protocol (OISP) integrated with DNFS in 12c dynamically tunes record size and logbias variables on the ZFS Storage Appliance to optimize performance
OISP
DNFS provides integrated performance tuning when backing up an Oracle database to a ZS3
Bypasses the operating system
Data is cached just once in user space with no 2nd copy in kernel space
Distribute throughput across multiple network interfaces and stripe buffers over up to 4 addresses
Example: Direct NFS Striping
Number of Addresses 1 2 4
Backup
4209 MB/s
5261 MB/s
5417 MB/s

19. Superior RAID-Z Performance
Disk Capacity Utilization
Mirrored
42.2%
Single Parity
69.3%
Double Parity
76.7%
High performance RAID-Z gives customers the choice of mirrored or striped parity based on their needs
Variable stripe width means every write is a full stripe write
RAID-Z reduces the IOPS overhead
RAID-Z Parity A1 A2 A3 Ap B1 B2 Bp B3 C1 Cp C2 C3
Disk1
Disk2
Disk3
Disk4

20. RAID-Z1 vs Mirror: Calculating IOPS Overhead
Example Backup – Why is RAID-Z1 Faster?
Single Parity
3419 MBPS
Mirrored
3337 MBPS
IOPS per disk correlate to AWR average wait time
Single Parity
Single Parity
158.2 IOPS per every 100 MBPS
Average disk IO size: KB
Mirrored
250.6 IOPS per every 100 MBPS
Average disk IO size: KB
IOPS per disk correlate to AWR average wait time
Mirroring
Mirroring generates 58% more IOPS
5x higher DB wait time
Conclusion

21. Choosing a Storage Profile
Mirrored, Single Parity or Double Parity
Mirrored: 1:1
Best uses: Restore performance, Incremental forever backup strategies, cloning for dev/test provisioning
Single Parity: Narrow stripe width (3+1)
Best uses: Backup performance, Traditional RMAN backup/restore strategies
Double Parity: Wide stripe width (variable)
Best uses: Only large sequential streaming IO, Optimal useable capacity
Lengthy resilvering times are possible

22. Optimization For Image Copy or Backupset
Incrementally updated backup, synergies with cloning, superior performance, generally accessible
Backupset:
Tape support, bypasses unused datafile blocks, RMAN default, multi-section backup
More channels
Align number of channels to datafiles in backupset
Image Copy Tuning
Fewer channels
Use multi-section backup
Backupset Tuning
General Performance Recommendations
Image Copy
4-10 channels per database server, align channels to datafiles
Backupset
2-4 channels per database server, specify section size (32-100GB)
Incremental Merge
Limit read-modify-write penalty (mirrored storage, write-optimized flash)

23. Save Money on SSD, Spend it on HDD
All writes whether synchronous or asynchronous are first written to the ARC
Incoming Writes
SSD is critical for achieving optimal transactional and mixed-IO performance
Dedicated RMAN workloads may perform better without SSD
Synchronous writes are committed to the ZIL before an acknowledgement is returned to the client
Synchronous Writes
Transaction groups transfer data from the ARC to HDD
TXG

24. SSD Is Not Always Faster
Example Backup (Backupset)
No Write SSD
2590 MBPS
4 Mirrored Write SSD
846 MBPS
RMAN backupsets require synchronous semantics but are not very latency sensitive
IO queuing can create a bottleneck and reduce throughput
Set logbias=throughput will bypass SSD write cache (if present) and write to HDD
Analytics shows that IO queuing on the 4 SSDs is creating a bottleneck
IO Queuing
Bandwidth intensive workloads benefit from wide IO distribution across the storage pool
Sharing the Load

25. More HDD Improves Performance
Peak sustained throughput increases as more disk are added to the ZFS Storage Appliance configuration
For optimal performance always use a single storage pool per controller for RMAN backup/restore
ZS3 RMAN Performance Scaling Adding Disk Shelves: 7200 RPM High Capacity Drives
Single Parity RAID
1 shelf
2 shelves
3 shelves
4 shelves
8 shelves
Backup (Image copy)
5.1 TB/hr
9.8 TB/hr
13.8 TB/hr
20.5 TB/hr
26 TB/hr
Restore (Image copy)
1.8 TB/hr
4.0 TB/hr
6.3 TB/hr
11.4 TB/hr
17 TB/hr

26. Optimizing for Incrementals Forever
Recommended in situations with very large databases and small daily change rates
Use mirrored storage and a significant amount of write-optimized flash
Use a two share configuration to separate the image copy from the incremental backupsets
Synchronous write bias should be set to latency for image copy share
Align ZFS record size to db_block_size for image copy to reduce read-modify-write overhead

27. For More Information – MOS 1354980.1
FAQ: Exadata RMAN Backup with The Oracle ZFS Storage Appliance
MOS Document
Engineered Systems Backup Utility (ESBU)
ESBU Download Page

28. Oracle Confidential – Internal/Restricted/Highly Restricted