Presentation is loading. Please wait.

Presentation is loading. Please wait.

Wait Events in RAC Session 362 Arup Nanda Longtime Oracle DBA arup.blogspot.com.

Similar presentations


Presentation on theme: "Wait Events in RAC Session 362 Arup Nanda Longtime Oracle DBA arup.blogspot.com."— Presentation transcript:

1 Wait Events in RAC Session 362 Arup Nanda Longtime Oracle DBA arup.blogspot.com

2 What is This? RAC Performance Tuning –I teach a course – Performance Tuning in RAC Wait events are useful for understanding the bottlenecks All single instance wait events are applicable to RAC RAC has some special cases This session talks about those RAC-specific wait events This is just a subset of the events; not a comprehensive list 2Wait Events in RAC

3 Whats a Wait? A process in Oracle can only be in three states –Doing something Useful (consuming CPU) ….. U –Idle, waiting for some work to be assigned ….. I –Waiting for something, e.g. ….. W a block from disk a lock a latch (could be waiting on CPU) Response time = U + I + W We must accurately measure each component time before we decide what and how to tune 3 Wait Events in RAC

4 Node 1 Cluster Coordination RAC for Beginners 4 Buffer Cache DBWR LMS SCN1 DBWR must get a lock on the database block before writing to the disk. This is called a Block Lock. Node 2 Database SCN2 Checkpoint!

5 Cache Fusion 5Wait Events in RAC Instance 1Instance 2 5 session1 session2 5 Has modified it Wants to modify it Will get it via interconnect

6 Checking for Buffers How exactly is this check performed? By checking for a lock on the block The request comes to the Grant Queue of the block Someone checks that no other instance has any lock Instance 1 can read from the disk i.e. Instance 1 is granted the lock 6 Block SID1 SID2 SID3 Grant Queue Convert Queue SID5 SID6 SID7 Wait Events in RAC

7 Master Instance Only one instance holds the grant and convert queues of a specific block This instance is called Master Instance of that block Master instance varies for each block The memory structure that shows the master instance of a buffer is called Global Resource Directory (GRD) That is replicated across all instances The requesting instance must check the GRD to find the master instance Then make a request to the master instance for the lock 7Wait Events in RAC Block SID1 SID2 SID3 Grant Queue Convert Queue SID5 SID6 SID7

8 Scenario 1 Session connected to Instance 1 wants to select a block from the table Activities by Instance 1 1.Check its own buffer cache to see if the block exists 1.If it is found, can it just use it? 2.If it not found, can it select from the disk? 2.If not, then check the other instances How will it know which copy of the block is the best source? 8Wait Events in RAC Instance 1 Instance 2 Session

9 Node 2Node 1 Cache Fusion 9Wait Events in RAC Buffer Cache SMON LMS When node 2 wants a buffer, it sends a message to the other instance. The message is sent to the LMS (Lock Management Server) of the other instance. LMS then sends the buffer to the other instance. LMS is also called Global Cache Server (GCS). message buffer

10 Grant Scenario 2 1.Check its buffer cache to see if the block exists 2.And the buffer is found. Can Instance1 use it? Not really. The buffer may be old; it may have been changed 3.LMS of node1 sends message to master of the buffer 3.Master checks the GES and doesnt sees any lock 4.Instance 1 is granted the global block lock 5.No buffer actually gets transferred 10Wait Events in RAC

11 Grant Scenario 3 Instance 1 is the master –Then it doesnt have to make a request for the grant In summary, here are the possible scenarios when Instance1 requests a buffer –Instance1 is the master; so no more processing is required –No one has the lock on the buffer, the grant is made by the master immediately –Another instance has the buffer in an incompatible mode. It has to be changed. 11Wait Events in RAC

12 Buffer States and Locks Buffers can be gotten in two states –Current – when the intention is to modify Shared Current – most recent copy. One copy per instance. Same as disk Exclusive Current – only one copy in the entire cluster. No shared current present –CR – when the intention is to only select Locks facilitate the state enforcement –XCUR for Exclusive Current –SCUR for Shared Current –No locking for CR 12Wait Events in RAC

13 Placeholder Event When the buffer is first requested, the session does not know which of the three paths it will go on to Therefore it is assigned a placeholder event This event is known as gc cr block request (for Consistent Read requests) If the request is made for the buffer in Current mode, the event is gc current block request When one of the three options is chosen, the appropriate event replaces the placeholder event 13Wait Events in RAC

14 Grant Event If the session merely requests a grant from the master: –It waits with the gc cr|current grant 2-way event –gc cr grant 2-way, for requests for buffers in Consistent Read mode –gc current grant 2-way, for requests for buffers in Current mode 14Wait Events in RAC

15 15 gc current|cr grant 2-way Instance 1 Instance 2 Session Database LMS gc current block requestgc current grant 2-way GRD GES db file scattered read Wait Events in RAC Block SID1 SID2 SID3 Grant Queue Conver t Queue SID5 SID6 SID7 request granted

16 Block Event After the request is made –and assuming that the buffer is not in the local cache –the buffer may be found in another instance –Requestor requests the buffer from the holding instance Possibilities: –Holder is also the master of that buffer –Holder is not the master If the holder is the master, the requesting session waits with the event gc cr|current block 2-way 16Wait Events in RAC

17 17 gc current|cr block 2-way Wait Events in RAC Instance 1 Instance 2 Session Database LMS gc current block requestgc current block 2-way GRD GES Block SID1 SID2 SID3 Grant Queue Conver t Queue SID5 SID6 SID7

18 18 gc current|cr block 3-way Instance 1 Instance 2 Instance 3 Session Requestor Master Holder gc current block 3-way Wait Events in RAC Block SID1 SID2 SID3 Grant Queue Conver t Queue SID5 SID6 SID7

19 Grant –vs- Get Events ActionWait Event is Block is requested by a sessionPlaceholder event – gc cr|current block request Buffer lock is requested from the master gc cr|current grant 2-way Buffer lock is granted, get from the disk db file scattered|sequential read Buffer is requested from the holder which is the same as the master gc cr|current block 2-way Buffer is requested from the holder which is not the master gc cr|current block 3-way 19 1.There is no 3-way grant event, since the request is made to the master 2.There is no 4-way block event; since there will a maximum of 2 hops: requestor master holder Wait Events in RAC

20 Interpretation ScenarioInterpretationTuning Implication The grant waits are very high compared to block waits The requested blocks are in the current instance; but the master is another instance Manually remaster the object? The 2-way block waits are very high compared to 3-way The cache fusion is taking place as expected, and most of the blocks are held and mastered at the same instance. But most of the blocks are not found in the local instance. Faster interconnect Shorten the run queue The 3-way block waits are high compared to 2-way Cache fusion is taking place; but the master and the holder are usually different. Manually remaster the object Application partitioning 20Wait Events in RAC

21 gc current/cr block lost Lost blocks while being transferred to the remote instance [in the interconnect] Cause: may not be the network itself Cause #1 Network –Network traffic drops the packets –Confirm from the ifconfig -a output –If packets are dropping, this could be a cause –Why? Bad network configuration CPU used for network processing Cause #2 CPU –LMS process is CPU starved 21Wait Events in RAC Instance 1 Instance 2 Session Database LMS gc current block requestgc current block 2-way GRD GES Block SID1SID1 SID2SID2 SID3SID3 GrantQueueGrantQueue ConvertQueueConvertQueue SID5SID5 SID6SID6 SID7SID7

22 gc current/cr block busy What it Means –Session wants a block from the remote instance –Remote instance delays preparing the block to fulfill the request [CR or current] read Cause: –Local delay on the remote instance –Most likely: an I/O bottleneck on the remote instance block is being accessed by some session which is CPU starved LGWR has not written a buffer to redo yet –Less likely: CPU starvation 22Wait Events in RAC

23 gc current/cr block congested Meaning –The instance has requested block from remote instance. –Remote instance has prepared the block and shipped it but it has not reached the requesting instance within 1 ms. What could be cause – network bottleneck? Not necessarily. Causes: –Long run queues, causing the LMS process to be delayed in processing the incoming block –Heavy paging due to memory deficiency. This causes the blocks to be paged in before being processed. 23Wait Events in RAC

24 Putting it All Together Every Oracle process is either –Doing some productive work –Waiting for some work to be given (idle) –Waiting for some resource Understand the reason for the wait –Devise a plan accordingly RAC related wait events are manifestations of these issues mostly Caused by –Network issues –LMS being overloaded –Blocks busy 24Wait Events in RAC

25 Thank You! Session 362 More Information: –Blog: arup.blogspot.com 100 Things You Probably Didn't Know About Oracle Database –Twitter: arupnanda 25Wait Events in RAC


Download ppt "Wait Events in RAC Session 362 Arup Nanda Longtime Oracle DBA arup.blogspot.com."

Similar presentations


Ads by Google