1. Wait Events in RAC Session 362
Arup Nanda
Longtime Oracle DBA
arup.blogspot.com

2. What is This? RAC Performance Tuning
I teach a course – Performance Tuning in RAC
Wait events are useful for understanding the bottlenecks
All single instance wait events are applicable to RAC
RAC has some special cases
This session talks about those RAC-specific wait events
This is just a subset of the events; not a comprehensive list
Wait Events in RAC

3. What’s a Wait? A process in Oracle can only be in three states
Doing something Useful (consuming CPU) ….. U
Idle, waiting for some work to be assigned ….. I
Waiting for something, e.g ….. W
a block from disk
a lock
a latch (could be waiting on CPU)
Response time = U + I + W
We must accurately measure each component time before we decide what and how to tune
Wait Events in RAC

4. Cluster Coordination Checkpoint! Checkpoint! Node 1 Node 2
Buffer Cache
Buffer Cache
SCN1
SCN2
DBWR
DBWR
Checkpoint!
LMS
LMS
Checkpoint!
Database
DBWR must get a lock on the database block before writing to the disk. This is called a Block Lock.
RAC for Beginners

5. Cache Fusion Instance 1 Instance 2 Will get it via interconnect
session1
session2
Instance 1
Instance 2
Wants to modify it
Has modified it 5 5
Wait Events in RAC

6. Checking for Buffers How exactly is this “check” performed? Block
By checking for a lock on the block
The request comes to the Grant Queue of the block
Someone checks that no other instance has any lock
Instance 1 can read from the disk
i.e. Instance 1 is granted the lock
Block
SID1
SID2
SID3
Grant Queue
Convert Queue
SID5
SID6
SID7
Wait Events in RAC

7. Master Instance
Only one instance holds the grant and convert queues of a specific block
This instance is called Master Instance of that block
Master instance varies for each block
The memory structure that shows the master instance of a buffer is called Global Resource Directory (GRD)
That is replicated across all instances
The requesting instance must check the GRD to find the master instance
Then make a request to the master instance for the lock
Block
SID1
SID2
SID3
Grant Queue
Convert Queue
SID5
SID6
SID7
Wait Events in RAC

8. Scenario 1
Session
Instance 1
Instance 2 DB
Session connected to Instance 1 wants to select a block from the table
Activities by Instance 1
Check its own buffer cache to see if the block exists
If it is found, can it just use it?
If it not found, can it select from the disk?
If not, then check the other instances
How will it know which copy of the block is the best source?
Wait Events in RAC

9. Cache Fusion
Node 1
Node 2
Buffer Cache
Buffer Cache
SMON
SMON
LMS
LMS
message
buffer
When node 2 wants a buffer, it sends a message to the other instance. The message is sent to the LMS (Lock Management Server) of the other instance. LMS then sends the buffer to the other instance. LMS is also called Global Cache Server (GCS).
Wait Events in RAC

10. Grant Scenario 2 Check its buffer cache to see if the block exists
And the buffer is found. Can Instance1 use it?
Not really. The buffer may be old; it may have been changed
LMS of node1 sends message to master of the buffer
Master checks the GES and doesn’t sees any lock
Instance 1 is granted the global block lock
No buffer actually gets transferred
Wait Events in RAC

11. Grant Scenario 3 Instance 1 is the master
Then it doesn’t have to make a request for the grant
In summary, here are the possible scenarios when Instance1 requests a buffer
Instance1 is the master; so no more processing is required
No one has the lock on the buffer, the grant is made by the master immediately
Another instance has the buffer in an incompatible mode. It has to be changed.
Wait Events in RAC

12. Buffer States and Locks
Buffers can be gotten in two states
Current – when the intention is to modify
Shared Current – most recent copy. One copy per instance. Same as disk
Exclusive Current – only one copy in the entire cluster. No shared current present
CR – when the intention is to only select
Locks facilitate the state enforcement
XCUR for Exclusive Current
SCUR for Shared Current
No locking for CR
Wait Events in RAC

13. Placeholder Event
When the buffer is first requested, the session does not know which of the three paths it will go on to
Therefore it is assigned a placeholder event
This event is known as gc cr block request (for Consistent Read requests)
If the request is made for the buffer in Current mode, the event is gc current block request
When one of the three options is chosen, the appropriate event replaces the placeholder event
Wait Events in RAC

14. Grant Event If the session merely requests a grant from the master:
It waits with the gc cr|current grant 2-way event
gc cr grant 2-way, for requests for buffers in Consistent Read mode
gc current grant 2-way, for requests for buffers in Current mode
Wait Events in RAC

15. gc current|cr grant 2-way
Instance 1
Instance 2
Block
SID1
SID2
SID3
Grant Queue
Convert Queue
SID5
SID6
SID7
db file scattered read
gc current block request
gc current grant 2-way
Session
request
granted
LMS
LMS
GRD
GES
Database
Wait Events in RAC
Performance Tuning in RAC

16. Block Event After the request is made
and assuming that the buffer is not in the local cache
the buffer may be found in another instance
Requestor requests the buffer from the holding instance
Possibilities:
Holder is also the master of that buffer
Holder is not the master
If the holder is the master, the requesting session waits with the event gc cr|current block 2-way
Wait Events in RAC

17. gc current|cr block 2-way
Instance 1
Instance 2
Session
Database
LMS
gc current block request
gc current block 2-way
GRD
GES
Block
SID1
SID2
SID3
Grant Queue
Convert Queue
SID5
SID6
SID7
Wait Events in RAC
Performance Tuning in RAC

18. gc current|cr block 3-way
Instance 1
Instance 2
Block
SID1
SID2
SID3
Grant Queue
Convert Queue
SID5
SID6
SID7
gc current block 3-way
Session
Master
Instance 3
Holder
Requestor
Wait Events in RAC
Performance Tuning in RAC

19. Grant –vs- Get Events
Action
Wait Event is
Block is requested by a session
Placeholder event – gc cr|current block request
Buffer lock is requested from the master
gc cr|current grant 2-way
Buffer lock is granted, get from the disk
db file scattered|sequential read
Buffer is requested from the holder which is the same as the master
gc cr|current block 2-way
Buffer is requested from the holder which is not the master
gc cr|current block 3-way
There is no 3-way grant event, since the request is made to the master
There is no 4-way block event; since there will a maximum of 2 hops: requestor  master  holder
Wait Events in RAC

20. Interpretation Scenario Interpretation Tuning Implication
The grant waits are very high compared to block waits
The requested blocks are in the current instance; but the master is another instance
Manually remaster the object?
The 2-way block waits are very high compared to 3-way
The cache fusion is taking place as expected, and most of the blocks are held and mastered at the same instance. But most of the blocks are not found in the local instance.
Faster interconnect
Shorten the run queue
The 3-way block waits are high compared to 2-way
Cache fusion is taking place; but the master and the holder are usually different.
Manually remaster the object
Application partitioning
Wait Events in RAC

21. gc current/cr block lost
Lost blocks while being transferred to the remote instance [in the interconnect]
Cause: may not be the network itself
Cause #1 Network
Network traffic drops the packets
Confirm from the ifconfig -a output
If packets are dropping, this could be a cause
Why?
Bad network configuration
CPU used for network processing
Cause #2 CPU
LMS process is CPU starved
Instance 1
Instance 2
Session
Database
LMS
gc current block request
gc current block 2-way
GRD
GES
Block
SID1
SID2
SID3
Grant Queue
Convert Queue
SID5
SID6
SID7
Wait Events in RAC

22. gc current/cr block busy
What it Means
Session wants a block from the remote instance
Remote instance delays preparing the block to fulfill the request [CR or current] read
Cause:
Local delay on the remote instance
Most likely: an I/O bottleneck on the remote instance
block is being accessed by some session which is CPU starved
LGWR has not written a buffer to redo yet
Less likely: CPU starvation
Wait Events in RAC

23. gc current/cr block congested
Meaning
The instance has requested block from remote instance.
Remote instance has prepared the block and shipped it
but it has not reached the requesting instance within 1 ms.
What could be cause – network bottleneck?
Not necessarily. Causes:
Long run queues, causing the LMS process to be delayed in processing the incoming block
Heavy paging due to memory deficiency. This causes the blocks to be paged in before being processed.
Wait Events in RAC

24. Putting it All Together
Every Oracle process is either
Doing some productive work
Waiting for some work to be given (idle)
Waiting for some resource
Understand the reason for the wait
Devise a plan accordingly
RAC related wait events are manifestations of these issues mostly
Caused by
Network issues
LMS being overloaded
Blocks busy
Wait Events in RAC

25. Thank You! Session 362 More Information: Blog: arup.blogspot.com
100 Things You Probably Didn't Know About Oracle Database
Twitter: arupnanda
Wait Events in RAC