1. Oracle 11g Real Application Clusters Advanced Administration
Natik Ameen Session #352

2. Introduction Natik Ameen, Senior Oracle DBA
DBA for 8 years, with 4 years on RAC 9i, 10g
Financial, Gaming and Aviation industries
Intended for everyone who has basic knowledge of RAC

3. Objective RAC overview ASM Nodeapps Metrics collection Wait events
Best Practice
11g New features

4. RAC Overview Clustered nodes Cluster Interconnect Virtual IP
Shared storage
NodeApps

5. RAC Architecture
Clustered nodes
Interconnect Network
Shared storage

6. SGA Structure & Processes
Global Resource Directory : Maintains mapping of data in SGA
Global Cache Server: Synchronizes Buffer cache across instances
Global Enqueue Services: Maintains coherency in data dictionary & library caches
Lock Manager Server: Transports blocks for Cache Fusion between nodes.

7. Cache Fusion
What is it
Concurrency & Consistency
Scalability

8. Interconnect Network Main component for “Cache Fusion”
IC uses private network
User Datagram Protocol(UDP)
Reliable Datagram Socket Protocol(RDS) >
Typical bandwidth utilization 20-30%, saturated > 70%

9. ASM Overview
ASM Instance
Disk Group
Disk Files
ASMLib

10. ASM Instance Scaled down version of DB instance
DB instance access DB files

11. Disk Groups
Striping
Mirroring
Failure groups

12. ASM Files
Name pre-pended with ‘+’

13. ASMLib
Kernel driver with efficient access to DB files

14. NodeApps Listener Oracle Notification Services
Fast Application Notification
Fast Connection Failover

15. Metrics Collection
AWR Reports
OS Utilities
Bench Marking

16. AWR Report The RAC specific statistics include:
Instance Global Cache Load Profile
Global Cache Efficiency percentages
Messaging Statistics
Top % CR and Current Block Segments

17. Cache Fusion data block & messaging traffic
Calculate network traffic

18. RAC Wait Events Most of these wait events occur during Cache Fusion
GC cr/current block 2-way, 3-way
GC cr/current block congested
GC cr/current block busy
GC current grant busy
GC cr/current block request
GC cr block lost

19. GC wait events Classified under “cluster wait lass”.
Characterized as “current” or as “CR”.
“current” classified as all blocks read into memory for the very first time
CR denotes block for read access
Master node maintains complete information on the resource, others maintain information on locally held locks

20. gc current block 2-way
Instance A requests block from master instance B
If the block is available on B then it is sent to A

21. gc current block 3-way
Instance A requests block from master instance B
B does not have block but directs instance holding block
or directs request to disk
Three hops maximum, not dependant on number of nodes

22. GC cr/current block congested
Repeated requests by foreground processes, not serviced by LMS
Indicates LMS not able to keep up
Queue lengths & scheduling delays in OS, can cause LMS delays
GC cr/current block busy
Delay for some reason, before block sent to requestor
Could be due to I/O saturation
GC current grant busy
Permission to access the block granted, but blocked by other requests ahead of it
GC cr/current block request
Wait time, when cr or current block is being retrieved
GC block lost
Lost blocks possibly due to faulty interconnect network

23. Where to get more information
Other sessions here at IOUG-A Live!
RAC Special Insterest Group(RAC SIG)
Consulting companies (depending on specialization)

24. Questions? Lots of things to cover
If we didn’t cover something you wanted to hear, please contact me

25. Oracle 11g Real Application Clusters Advanced Administration
THANK YOU!
Please fill out evaluations!