Presentation is loading. Please wait.

Presentation is loading. Please wait.

Real Life RAC Performance Tuning Arup Nanda Lead DBA Starwood Hotels.

Similar presentations


Presentation on theme: "Real Life RAC Performance Tuning Arup Nanda Lead DBA Starwood Hotels."— Presentation transcript:

1 Real Life RAC Performance Tuning Arup Nanda Lead DBA Starwood Hotels

2 © Arup Nanda Who am I Oracle DBA for 13 years and counting Oracle DBA for 13 years and counting Working on OPS from 1999 Working on OPS from 1999 Speak at conferences, write papers, books Speak at conferences, write papers, books

3 © Arup Nanda Why This Session I get s like this: I get s like this: We are facing performance issues in RAC. What should I do next? We are facing performance issues in RAC. What should I do next? Real Life Advice Real Life Advice Common Issues (with Wait Events) Common Issues (with Wait Events) Dispelling Myths Dispelling Myths Formulate a Plan of Attack Formulate a Plan of Attack Real Life Case Study Real Life Case Study proligence.com/downloads.html proligence.com/downloads.html

4 © Arup Nanda Our RAC Implementation Oracle 10g RAC in March 2004 Oracle 10g RAC in March 2004 Itanium Platform running HP/UX Itanium Platform running HP/UX Oracle Oracle Result: Failed Result: Failed Second Attempt: Dec 2004 – Second Attempt: Dec 2004 – Result: Failed Again Result: Failed Again Third Attempt: March 2005 – Third Attempt: March 2005 – Result: Success! Result: Success!

5 © Arup Nanda Challenges Technology Technology Lone ranger Lone ranger A lot of mystery and disconnected facts! A lot of mystery and disconnected facts! People People Building a team that could not only deliver; but also sustain the delivered parts Building a team that could not only deliver; but also sustain the delivered parts Each day we learned something new Each day we learned something new In todays session: real performance issues we faced and how we resolved them, along with wait events. In todays session: real performance issues we faced and how we resolved them, along with wait events.

6 © Arup Nanda Why RAC Performance? All tuning concepts in single instance applied to RAC as well All tuning concepts in single instance applied to RAC as well RAC has other complexities RAC has other complexities More than 1 buffer cache More than 1 buffer cache Multiple caches – library cache, row cache Multiple caches – library cache, row cache Interconnect Interconnect Pinging Pinging Global Locking Global Locking

7 © Arup Nanda Why RAC Perf Tuning We want to make sure we identify the right problem and go after it We want to make sure we identify the right problem and go after it …. not just a problem …. not just a problem

8 © Arup Nanda Switch VIP Service Listener Instance ASM Clusterware Op Sys VIP Service Listener Instance ASM Clusterware Op Sys Public Interface Cache Cache Fusion OCRVoting Switch Interconnect Storage Node1Node2 Lock Manager

9 © Arup Nanda Areas of Concern in RAC More than 1 buffer cache More than 1 buffer cache Multiple caches – library cache, row cache Multiple caches – library cache, row cache Interconnect Interconnect Global Locking Global Locking

10 © Arup Nanda Cache Issues Two Caches, requires synchronization Two Caches, requires synchronization What that means: What that means: A changed block in one instance, when requested by another, should be sent across via a bridge A changed block in one instance, when requested by another, should be sent across via a bridge This bridge is the Interconnect This bridge is the Interconnect

11 © Arup Nanda Interconnect Performance Interconnect must be on a private LAN Interconnect must be on a private LAN Port aggregation to increase throughput Port aggregation to increase throughput APA on HPUX APA on HPUX If using Gigabit over Ethernet, use Jumbo Frames If using Gigabit over Ethernet, use Jumbo Frames

12 © Arup Nanda Checking Interconnect Used Identify the interconnect used Identify the interconnect used $ oifcfg getif lan global cluster_interconnect lan global public lan global public Is lan902 the bonded interface? If not, then set it Is lan902 the bonded interface? If not, then set it $ oifcfg setif …

13 © Arup Nanda Pop Quiz If I have a very fast interconnect, I can perform the same work in multiple node RAC as a single server with faster CPUs. True/False? If I have a very fast interconnect, I can perform the same work in multiple node RAC as a single server with faster CPUs. True/False? Since cache fusion is now write-write, a fast interconnect will compensate for a slower IO subsystem. True/False? Since cache fusion is now write-write, a fast interconnect will compensate for a slower IO subsystem. True/False?

14 © Arup Nanda Cache Coherence Times The time is a sum of time for: The time is a sum of time for: Finding the block in the cache Finding the block in the cache Identifying the master Identifying the master Get the block in the interconnect Get the block in the interconnect Transfer speed of the interconnect Transfer speed of the interconnect Latency of the interconnect Latency of the interconnect Receive the block by the remote instance Receive the block by the remote instance Create the consistent image for the user Create the consistent image for the user CPUCPU CPUCPU Interconnect

15 © Arup Nanda So it all boils down to: Block Access Cost Block Access Cost more blocks -> more the time more blocks -> more the time Parallel Query Parallel Query Lock Management Cost Lock Management Cost More coordination -> more time More coordination -> more time Implicit Cache Checks – Sequence Numbers Implicit Cache Checks – Sequence Numbers Interconnect Cost Interconnect Cost Latency Latency Speed Speed more data to transfer -> more the time more data to transfer -> more the time

16 © Arup Nanda Hard Lessons In RAC, problem symptoms may not indicate the correct problem! In RAC, problem symptoms may not indicate the correct problem! Example: Example: When the CPU is too busy to receive or send packets via UDP, the packets fails and the Clusterware thinks the node is down and evicts it. When the CPU is too busy to receive or send packets via UDP, the packets fails and the Clusterware thinks the node is down and evicts it.

17 © Arup Nanda OS Troubleshooting OS utilities to troubleshoot CPU issues OS utilities to troubleshoot CPU issues top top glance glance OS Utilities to troubleshoot process issues: OS Utilities to troubleshoot process issues: truss truss strace strace dbx dbx pstack pstack

18 © Arup Nanda Reducing Latency A factor of technology A factor of technology TCP is the most latent TCP is the most latent UDP is better (over Ethernet) UDP is better (over Ethernet) Proprietary protocols are usually better Proprietary protocols are usually better HyperFabric by HP HyperFabric by HP Reliable Datagram (RDP) Reliable Datagram (RDP) Direct Memory Channel Direct Memory Channel Infiniband Infiniband UDP over Infiniband UDP over Infiniband RDP over Infiniband RDP over Infiniband

19 © Arup Nanda Start with AWR

20 © Arup Nanda gc current|cr grant 2-way Instance 1 Instance 2 Session Database LMSLGWR Log Buffer LMS gc current block requestgc current grant 2-way

21 © Arup Nanda gc current|cr block 2-way Instance 1 Instance 2 Session Database LMSLGWR Log Buffer LMS gc current block requestlog file syncgc current block 2-way

22 © Arup Nanda gc current|cr block 3-way Instance 1 Instance 2 Instance 3 Session Requestor Master Holder gc current block 3-way

23 © Arup Nanda RAC related Stats

24 © Arup Nanda RAC Stats contd.

25 © Arup Nanda

26 Other GC Block Waits gc current/cr block lost gc current/cr block lost Lost blocks due to Interconnect or CPU Lost blocks due to Interconnect or CPU gc curent/cr block busy gc curent/cr block busy The consistent read request was delayed, most likely an I/O bottleneck The consistent read request was delayed, most likely an I/O bottleneck gc current/cr block congested gc current/cr block congested Long run queues and/or paging due to memory deficiency. Long run queues and/or paging due to memory deficiency.

27 © Arup Nanda Hung or Slow? Check V$SESSION for WAIT_TIME Check V$SESSION for WAIT_TIME If 0, then its not waiting; its hung If 0, then its not waiting; its hung When hung: When hung: Take a systemstate dump from all nodes Take a systemstate dump from all nodes Wait some time Wait some time Take another systemstate dump Take another systemstate dump Check change in values. If unchanged, then system is hung Check change in values. If unchanged, then system is hung

28 © Arup Nanda Chart a Plan Rule out the obvious Rule out the obvious Start with AWR Report Start with AWR Report Start with Top-5 Waits Start with Top-5 Waits See if they have any significant waits See if they have any significant waits … especially RAC related … especially RAC related Go on to RAC Statistics Go on to RAC Statistics Base your solution based on the wait event Base your solution based on the wait event

29 © Arup Nanda Rule out the obvious Is interconnect private? Is interconnect private? Is interconnect on UDP? Is interconnect on UDP? Do you see high CPU? Do you see high CPU? Do you see a lot of IO bottleneck? Do you see a lot of IO bottleneck? How about memory? How about memory? Are the apps spread over evenly? Are the apps spread over evenly? Do you see lost blocks? Do you see lost blocks?

30 © Arup Nanda Make Simple Fixes Strongly consider RAID 0+1 Strongly consider RAID 0+1 Highest possible number of I/O paths Highest possible number of I/O paths Use fastest interconnect possible Use fastest interconnect possible Use private collision free domain for I/C Use private collision free domain for I/C Cache and NOORDER sequences Cache and NOORDER sequences

31 © Arup Nanda Enterprise Manager

32 © Arup Nanda Buffer Busy Cause Cause Instance wants to bring something from disk to the buffer cache Instance wants to bring something from disk to the buffer cache Delay, due to space not available Delay, due to space not available Delay, bcoz the source buffer is not ready Delay, bcoz the source buffer is not ready Delay, I/O is slow Delay, I/O is slow Delay, bcoz redo log is being flushed Delay, bcoz redo log is being flushed In summary In summary Log buffer flush -> gc buffer busy Log buffer flush -> gc buffer busy

33 © Arup Nanda Parallel Query One major issue in RAC is parallel query that goes across many nodes One major issue in RAC is parallel query that goes across many nodes Instance 1Instance 2 QC Slave Via Interconnect

34 © Arup Nanda Restricting PQ Define Instance Groups Define Instance Groups Specify in init.ora prodb1.instance_groups='pqgroup1'prodb2.instance_groups='pqgroup2' Specify Instance Groups in Session Specify Instance Groups in Session SQL> alter session set parallel_instance_group = 'pqgroup1';

35 © Arup Nanda Forcing PQ on both Nodes Define a common Instance Group Define a common Instance Groupprodb1.instance_groups='pqgroup1prodb1.instance_groups=pq2nodes'prodb2.instance_groups='pqgroup2'prodb2.instance_groups='pq2nodes' Specify Instance Groups in Session Specify Instance Groups in Session SQL> alter session set parallel_instance_group = 'pq2nodes';

36 © Arup Nanda Vital Cache Fusion Views gv$cache_transfer: Monitor blocks transferred by object gv$cache_transfer: Monitor blocks transferred by object gv$class_cache_transfer: Monitor block transfer by class gv$class_cache_transfer: Monitor block transfer by class gv$file_cache_transfer: Monitor the blocks transferred per file gv$file_cache_transfer: Monitor the blocks transferred per file gv$temp_cache_transfer: Monitor the transfer of temporary tablespace blocks gv$temp_cache_transfer: Monitor the transfer of temporary tablespace blocks

37 © Arup Nanda Hot Tables Tables, e.g. Rate Plans Tables, e.g. Rate Plans Small Small Compact blocks Compact blocks High updates High updates High reads High reads Symptoms Symptoms gc buffer busy waits gc buffer busy waits Solution Solution Less rows per block Less rows per block High PCTFREE, INITRANS, High PCTFREE, INITRANS, ALTER TABLE … MINIMIZE RECORDS_PER_BLOCK ALTER TABLE … MINIMIZE RECORDS_PER_BLOCK

38 © Arup Nanda Hot Sequences Symptoms: Symptoms: High waits on Sequence Number latch High waits on Sequence Number latch High waits on SEQ$ table High waits on SEQ$ table Solution: Solution: Increase the cache Increase the cache Make it NOORDER Make it NOORDER Especially AUDSESS$ sequence in SYS, used in Auditing Especially AUDSESS$ sequence in SYS, used in Auditing

39 © Arup Nanda Read Only? Say So. Reading table data from other instances create gc * contentions Reading table data from other instances create gc * contentions Suggestion: Suggestion: Move Read Only tables to a single tablespace Move Read Only tables to a single tablespace Make this tablespace Read Only Make this tablespace Read Only SQL> alter tablespace ROD read only;

40 © Arup Nanda Partitioning Partitioning creates several segments for the same table (or index) Partitioning creates several segments for the same table (or index) => more resources => more resources => less contention => less contention

41 © Arup Nanda Monotonically Increasing Index Problem: Problem: Reservation ID, a sequence generated key Reservation ID, a sequence generated key Index is heavy on one side Index is heavy on one side Symptoms Symptoms Buffer busy waits Buffer busy waits Index block spilts Index block spilts Solutions: Solutions: Reverse key indexes Reverse key indexes Hash partitioned index (even if the table is not partitioned) 10gR2 Hash partitioned index (even if the table is not partitioned) 10gR2

42 © Arup Nanda Library Cache In RAC, Library Cache is global In RAC, Library Cache is global So, parsing cost is worse than non-RAC So, parsing cost is worse than non-RAC Solutions: Solutions: Minimize table alters, drops, creates, truncates Minimize table alters, drops, creates, truncates Use PL/SQL stored programs, not unnamed blocks Use PL/SQL stored programs, not unnamed blocks

43 © Arup Nanda Log Files In 10g R2, the log files are in a single location: In 10g R2, the log files are in a single location: $CRS_HOME/log/ /… $CRS_HOME/log/ /… racg racg crsd crsd cssd cssd evmd evmd client client cssd/oclsmon cssd/oclsmon $ORACLE_HOME/racg/dump $ORACLE_HOME/racg/dump

44 © Arup Nanda Case Study

45 © Arup Nanda Diagnosis ifconfig -a shows no congestion or dropped packets ifconfig -a shows no congestion or dropped packets Top shows 1% idle time on node 2 Top shows 1% idle time on node 2 Top processes Top processes LMS and LMD LMS and LMD And, several Netbackup processes And, several Netbackup processes

46 © Arup Nanda Further Diagnosis SQL: SQL: select * from v$instance_cache_transfer where class = 'data block' and instance = 1; Output: Output: INSTANCE CLASS CR_BLOCK CR_BUSY INSTANCE CLASS CR_BLOCK CR_BUSY CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED data block data block After sometime: After sometime: INSTANCE CLASS CR_BLOCK CR_BUSY INSTANCE CLASS CR_BLOCK CR_BUSY CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED data block data block See increases

47 © Arup Nanda Diagnosis: Diagnosis: CPU starvation by LMS/D processes caused GC waits. CPU starvation by LMS/D processes caused GC waits. Solution: Solution: Killed the Netbackup processes Killed the Netbackup processes LMD and LMS got the CPU LMD and LMS got the CPU

48 © Arup Nanda Increasing Interconnect Speed Faster Hardware Faster Hardware Gigabit Ethernet; not Fast Gigabit Ethernet; not Fast Infiniband, even if IP over IB Infiniband, even if IP over IB NIC settings NIC settings Duplex Mode Duplex Mode Highest Top Bit Rate Highest Top Bit Rate TCP Settings TCP Settings Flow Control Settings Flow Control Settings Network Interrupts for CPU Network Interrupts for CPU Socket Receive Buffer Socket Receive Buffer LAN Planning LAN Planning Private LANs Private LANs Collision Domains Collision Domains

49 © Arup Nanda High Speed Interconnects Oracle will support RDS over Infiniband Oracle will support RDS over Infiniband On 10 Gig Ethernet as well On 10 Gig Ethernet as well

50 © Arup Nanda In summary: Planning Adequate CPU, Network, Memory Adequate CPU, Network, Memory Sequences – cache, noorder Sequences – cache, noorder Tablespaces read only Tablespaces read only Un-compact small hot tables Un-compact small hot tables Keep undo and redo on fastest disks Keep undo and redo on fastest disks Avoid full table scans of large tables Avoid full table scans of large tables Avoid DDLs and unnamed PL/SQL blocks Avoid DDLs and unnamed PL/SQL blocks

51 © Arup Nanda In summary: Diagnosis Start with AWR Start with AWR Identify symptoms and assign causes Identify symptoms and assign causes Dont get fooled by gc waits as interconnect issues Dont get fooled by gc waits as interconnect issues Find the correlation between dropped packets in network, CPU issues from sar and gc buffer lost in sysstat reports. Find the correlation between dropped packets in network, CPU issues from sar and gc buffer lost in sysstat reports.

52 © Arup Nanda Thank You! Download from: proligence.com/downloads.html


Download ppt "Real Life RAC Performance Tuning Arup Nanda Lead DBA Starwood Hotels."

Similar presentations


Ads by Google