2. Oracle Database Performance Secrets Finally Revealed Greg Rahn & Michael Hallas Oracle Real-World Performance Group Server Technologies

3. 3 About The Real-World Performance Group

4. 4 Part of Server Technologies Competitive customer benchmarks Performance escalations

5. 5 The secret to a chef’s masterpiece is all in the recipe

6. 6 Agenda Troubleshooting approach Intro to the optimizer Demonstration Summary of secrets and lessons learned

7. 7 What is your performance troubleshooting approach?

8. 8 Do you... Use Google as your first resource? Use the Oracle Documentation as your second resource? Believe in tuning databases by changing block size? Frequently use database parameters to tune queries? Believe in “Silver Bullet Tuning”? Blindly apply previously successful solutions? Practice the “Change and Test” troubleshooting approach?

9. 9 The Change and Test Troubleshooting Approach

10. 10 What really matters for troubleshooting performance?

11. 11 Quite simply... it's all in the approach.

12. 12 Stack Visualization

13. 13 The Systematic Troubleshooting Approach 1. Define the problem, the scope and identify symptoms 2. Collect and analyze data/metrics 3. Ask “Why?” five times to identify root cause 4. Understand and devise change 5. Make a single change 6. Observe and measure results 7. If necessary, back out change; repeat process

14. 14 Systematic Troubleshooting Guidelines Don’t just look inside the database, look outside as well Make exactly one change at a time Scope of solution matches scope of problem Choose the right tool for the job Carefully document change and impact Suppressing the problem is not the same as root cause Realize that you may not be able to get to the root cause in just one step

15. 15 Fix on Failure vs. Fix it Forever The benefits of root cause analysis Fix on Failure – Finger pointing and the blame game – Stressful for everyone – Never time to fix it right the first time, but always plenty of time to keep fixing it time and time again Fix it Forever – Identify root causes of problems, so permanent solutions can be implemented – Develop a logical, systematic and data driven approach to problem solving

16. 16 Example of Applying the “5 Whys”

17. 17 Applying the “5 Whys” to My batch job ran long last night 1. Why? - A specific query took 5x as long 2. Why? - Execution plan changed from HJ to NLJ 3. Why? - Query optimizer costed the NLJ to be cheaper 4. Why? - Variables involved in the costing have changed 5. Why? - Statistics were gathered with wrong options

18. 18 Choosing Different Levels of Scope System level – database parameters – alter system – object statistics Session level – alter session Statement level – hints – SQL profiles & outlines & baselines

19. 19 Performance Troubleshooting Toolbox ADDM, AWR, ASH reports and raw data SQL Monitoring Active Report (11g) DBMS_XPLAN SQL Trace V$SESSTAT V$SESSION Stack dumps (pstack) OS metrics tools (collectl, iostat, vmstat, mpstat, etc.)

20. 20 Quick Introduction To The Optimizer

21. 21 Good cardinality estimates generally result in a good plan, however, bad cardinality estimates do not always result in a bad plan An Important Note About Cardinality Estimates

22. 22 Introducing the Cost-Based Optimizer Cost and Cardinality Cardinality – Estimated number of rows returned from a join, a table or an index – Factors influencing cardinality Query predicates and query variables Object statistics

23. 23 Introducing the Optimizer Cost and Cardinality Cost – Representation of resource consumption CPU Disk I/O Memory Network I/O – Factors influencing cost Cardinality and selectivity Cost model Parameters System statistics

24. 24 Good SQL and Bad SQL Good SQL – SQL that makes it possible for the optimizer to produce a good cardinality estimate – select * from emp where ename != ‘KING’ Bad SQL – SQL that makes it difficult for the optimizer to produce a good cardinality estimate – select * from emp where replace(ename, ‘KING’) is not null

25. 25 Good Plans and Bad Plans Good Plan – Efficient retrieval or modification of the desired rows – Highly selective index to retrieve few rows from a large table – Scan to retrieve many rows from a large table Bad Plan – Inefficient retrieval or modification of the desired rows – Scan to retrieve few rows from a large table – Non-selective index to retrieve many rows from a large table

26. 26 What is a Query Plan? Access Path – Table scan – Index { fast full | full | range | skip | unique } scan Join Method – Hash – Merge – Nested loops Join Order Distribution Method – Broadcast – Hash

27. 27 Challenges in Cardinality Estimation Complex predicates Correlation Non-representative bind values Out of range predicates Skew Statistics Quality – Frequency – Histograms – Sample Size

28. 28 What Is Dynamic Sampling? Improves quality of cardinality estimates Objects with no statistics – Avoids the use of heuristics – Less complete than statistics stored in the dictionary Objects with existing statistics – Predicates with complex expressions

29. 29 Demonstration

30. 30 SQL> select count(*) from emp; COUNT(*) ---------- 14 Execution Plan ---------------------------------------------------------- Plan hash value: 2083865914 --------------------------------------------------------------------------- | Id | Operation | Name | Rows | Cost (%CPU)| Time | --------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 3 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | | | | 2 | TABLE ACCESS STORAGE FULL| EMP | 14 | 3 (0)| 00:00:01 | --------------------------------------------------------------------------- Note ----- - dynamic sampling used for this statement (level=2) Dynamic Sampling Counting Rows

31. 31 SQL> select ename from emp; ENAME ------------------------------------------------------------------------- SMITH … MILLER 14 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 3956160932 ---------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ---------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 14 | 28028 | 3 (0)| 00:00:01 | | 1 | TABLE ACCESS STORAGE FULL| EMP | 14 | 28028 | 3 (0)| 00:00:01 | ---------------------------------------------------------------------------------- Note ----- - dynamic sampling used for this statement (level=2) Dynamic Sampling Projection

32. 32 SQL> select deptno, count(*) from emp group by deptno; DEPTNO COUNT(*) ---------- 30 6 20 5 10 3 Execution Plan ---------------------------------------------------------- Plan hash value: 4067220884 ----------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ----------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 14 | 182 | 4 (25)| 00:00:01 | | 1 | HASH GROUP BY | | 14 | 182 | 4 (25)| 00:00:01 | | 2 | TABLE ACCESS STORAGE FULL| EMP | 14 | 182 | 3 (0)| 00:00:01 | ----------------------------------------------------------------------------------- Note ----- - dynamic sampling used for this statement (level=2) Dynamic Sampling Aggregation

33. 33 SQL> select dbms_flashback.get_system_change_number scn from dual; 1158999991 SQL> select ename from emp as of scn &get_scn.; ENAME --------------------------------------------------------------------------- SMITH … MILLER 14 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 3956160932 ---------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ---------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 409 | 799K| 6 (0)| 00:00:01 | | 1 | TABLE ACCESS STORAGE FULL| EMP | 409 | 799K| 6 (0)| 00:00:01 | ---------------------------------------------------------------------------------- Dynamic Sampling Flashback

34. 34 SQL> create table emp_ext organization external 2 (type ORACLE_DATAPUMP default directory DATA_PUMP_DIR location ('emp_ext.dmp')) 3 as select * from emp; SQL> select ename from emp_ext; ENAME ------------------------------------------------------------------------------------- SMITH … MILLER 14 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 2795778448 -------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | -------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 8168 | 15M| 29 (0)| 00:00:01 | | 1 | EXTERNAL TABLE ACCESS FULL| EMP_EXT | 8168 | 15M| 29 (0)| 00:00:01 | -------------------------------------------------------------------------------------- Dynamic Sampling External Table

35. 35 SQL> create table emp_hash partition by hash(empno) (partitions 1024) as select * from emp; SQL> select ename from emp_hash; ENAME -------------------------------------------------------------------------------- ALLEN … JONES 14 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 3965505926 ------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | ------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 87398 | 166M| 291 (1)| 00:00:04 | | | | 1 | PARTITION HASH ALL | | 87398 | 166M| 291 (1)| 00:00:04 | 1 | 1024 | | 2 | TABLE ACCESS STORAGE FULL| EMP_HASH | 87398 | 166M| 291 (1)| 00:00:04 | 1 | 1024 | ------------------------------------------------------------------------------------------------------- Dynamic Sampling Empty Partitions

36. 36 SQL> exec dbms_stats.gather_table_stats(USER,'EMP_EXT',estimate_percent=>NULL); SQL> exec dbms_stats.gather_table_stats(USER,'EMP'); SQL> select ename from emp_ext; ENAME ------------------------------------------------------------------------------------- SMITH … MILLER 14 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 2795778448 -------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | -------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 14 | 84 | 2 (0)| 00:00:01 | | 1 | EXTERNAL TABLE ACCESS FULL| EMP_EXT | 14 | 84 | 2 (0)| 00:00:01 | -------------------------------------------------------------------------------------- Statistics External Table

37. 37 SQL> select ename from emp as of scn &get_scn.; ENAME ------------------------------------------------------------------------------------- SMITH … MILLER 14 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 3956160932 ---------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ---------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 14 | 84 | 6 (0)| 00:00:01 | | 1 | TABLE ACCESS STORAGE FULL| EMP | 14 | 84 | 6 (0)| 00:00:01 | ---------------------------------------------------------------------------------- Statistics Flashback

38. 38 SQL> select deptno, count(*) from emp group by deptno; DEPTNO COUNT(*) ---------- 30 6 20 5 10 3 Execution Plan ---------------------------------------------------------- Plan hash value: 4067220884 ----------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ----------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 3 | 9 | 4 (25)| 00:00:01 | | 1 | HASH GROUP BY | | 3 | 9 | 4 (25)| 00:00:01 | | 2 | TABLE ACCESS STORAGE FULL| EMP | 14 | 42 | 3 (0)| 00:00:01 | ----------------------------------------------------------------------------------- Statistics Aggregation

39. 39 SQL> select ename from emp; ENAME ------------------------------------------------------------------------------------- SMITH … MILLER 14 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 3956160932 ---------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ---------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 14 | 84 | 3 (0)| 00:00:01 | | 1 | TABLE ACCESS STORAGE FULL| EMP | 14 | 84 | 3 (0)| 00:00:01 | ---------------------------------------------------------------------------------- Statistics Projection

40. 40 Parallel Query Example Good SQL SQL> select /*+ MONITOR */ 2 count(p1.text) "in may" 3,count(p2.text) "under gemini" 4,sum(decode(p2.text,null, 1, 0)) "in may AND under gemini" 5 from 6 (select person_id, 'birthdays in may' as text from persons where month = 'may') p1 7,(select person_id, 'birthdays under gemini' as text from persons where zodiac = 'gemini') p2 8 where p1.person_id = p2.person_id(+); in may under gemini in may AND under gemini --------------- --------------- ----------------------- 31000000 10000000 21000000 Elapsed: 00:00:05.28

41. 41 ---------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 42 | 123K (1)| | | | | 1 | SORT AGGREGATE | | 1 | 42 | | | | | | 2 | PX COORDINATOR | | | | | | | | | 3 | PX SEND QC (RANDOM) | :TQ10002 | 1 | 42 | | Q1,02 | P->S | QC (RAND) | | 4 | SORT AGGREGATE | | 1 | 42 | | Q1,02 | PCWP | | |* 5 | HASH JOIN OUTER | | 32M| 1286M| 123K (1)| Q1,02 | PCWP | | | 6 | PX RECEIVE | | 32M| 459M| 61895 (1)| Q1,02 | PCWP | | | 7 | PX SEND HASH | :TQ10000 | 32M| 459M| 61895 (1)| Q1,00 | P->P | HASH | | 8 | PX BLOCK ITERATOR | | 32M| 459M| 61895 (1)| Q1,00 | PCWC | | |* 9 | TABLE ACCESS STORAGE FULL| PERSONS | 32M| 459M| 61895 (1)| Q1,00 | PCWP | | | 10 | PX RECEIVE | | 31M| 798M| 61933 (1)| Q1,02 | PCWP | | | 11 | PX SEND HASH | :TQ10001 | 31M| 798M| 61933 (1)| Q1,01 | P->P | HASH | | 12 | PX BLOCK ITERATOR | | 31M| 798M| 61933 (1)| Q1,01 | PCWC | | |* 13 | TABLE ACCESS STORAGE FULL| PERSONS | 31M| 798M| 61933 (1)| Q1,01 | PCWP | | ---------------------------------------------------------------------------------------------------------------- Parallel Query Example Good SQL

42. 42 SQL> select /*+ MONITOR */ 2 count(p1.text) "in may" 3,count(p2.text) "under gemini" 4,sum(decode(p2.text,null, 1, 0)) "in may AND under gemini" 5 from 6 (select person_id, 'birthdays in may' as text from persons where month = 'may') p1 7,(select person_id, 'birthdays under gemini' as text from persons where month in ('may', 'june') and lower(zodiac) = 'gemini') p2 8 where p1.person_id = p2.person_id(+); in may under gemini in may AND under gemini --------------- --------------- ----------------------- 31000000 10000000 21000000 Elapsed: 00:00:27.34 Parallel Query Example Bad SQL

43. 43 ---------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 42 | 123K (1)| | | | | 1 | SORT AGGREGATE | | 1 | 42 | | | | | | 2 | PX COORDINATOR | | | | | | | | | 3 | PX SEND QC (RANDOM) | :TQ10001 | 1 | 42 | | Q1,01 | P->S | QC (RAND) | | 4 | SORT AGGREGATE | | 1 | 42 | | Q1,01 | PCWP | | |* 5 | HASH JOIN RIGHT OUTER | | 32M| 1286M| 123K (1)| Q1,01 | PCWP | | | 6 | PX RECEIVE | | 618K| 15M| 62068 (2)| Q1,01 | PCWP | | | 7 | PX SEND BROADCAST | :TQ10000 | 618K| 15M| 62068 (2)| Q1,00 | P->P | BROADCAST | | 8 | PX BLOCK ITERATOR | | 618K| 15M| 62068 (2)| Q1,00 | PCWC | | |* 9 | TABLE ACCESS STORAGE FULL| PERSONS | 618K| 15M| 62068 (2)| Q1,00 | PCWP | | | 10 | PX BLOCK ITERATOR | | 32M| 459M| 61895 (1)| Q1,01 | PCWC | | |* 11 | TABLE ACCESS STORAGE FULL | PERSONS | 32M| 459M| 61895 (1)| Q1,01 | PCWP | | ---------------------------------------------------------------------------------------------------------------- Parallel Query Example Bad SQL

44. 44 SQL> alter session set optimizer_dynamic_sampling = 2; Session altered. SQL> select /*+ MONITOR */ 2 count(p1.text) "in may" 3,count(p2.text) "under gemini" 4,sum(decode(p2.text,null, 1, 0)) "in may AND under gemini" 5 from 6 (select person_id, 'birthdays in may' as text from persons where month = 'may') p1 7,(select person_id, 'birthdays under gemini' as text from persons where month in ('may', 'june') and lower(zodiac) = 'gemini') p2 8 where p1.person_id = p2.person_id(+); in may under gemini in may AND under gemini --------------- --------------- ----------------------- 31000000 10000000 21000000 Elapsed: 00:00:07.21 Parallel Query Example Bad SQL with Dynamic Sampling

45. 45 ---------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 42 | 124K (1)| | | | | 1 | SORT AGGREGATE | | 1 | 42 | | | | | | 2 | PX COORDINATOR | | | | | | | | | 3 | PX SEND QC (RANDOM) | :TQ10002 | 1 | 42 | | Q1,02 | P->S | QC (RAND) | | 4 | SORT AGGREGATE | | 1 | 42 | | Q1,02 | PCWP | | |* 5 | HASH JOIN OUTER | | 32M| 1287M| 124K (1)| Q1,02 | PCWP | | | 6 | PX RECEIVE | | 32M| 459M| 61895 (1)| Q1,02 | PCWP | | | 7 | PX SEND HASH | :TQ10000 | 32M| 459M| 61895 (1)| Q1,00 | P->P | HASH | | 8 | PX BLOCK ITERATOR | | 32M| 459M| 61895 (1)| Q1,00 | PCWC | | |* 9 | TABLE ACCESS STORAGE FULL| PERSONS | 32M| 459M| 61895 (1)| Q1,00 | PCWP | | | 10 | PX RECEIVE | | 43M| 1126M| 62068 (2)| Q1,02 | PCWP | | | 11 | PX SEND HASH | :TQ10001 | 43M| 1126M| 62068 (2)| Q1,01 | P->P | HASH | | 12 | PX BLOCK ITERATOR | | 43M| 1126M| 62068 (2)| Q1,01 | PCWC | | |* 13 | TABLE ACCESS STORAGE FULL| PERSONS | 43M| 1126M| 62068 (2)| Q1,01 | PCWP | | ---------------------------------------------------------------------------------------------------------------- … Note ----- - dynamic sampling used for this statement (level=6) Parallel Query Example Bad SQL with Dynamic Sampling

46. 46 What Have We Seen? Cardinality Drives Plan Selection ▲ Broadcast ▼ Hash ▼ Broadcast ▲ Hash

47. 47 What Have We Seen? SQL Monitor Report – Ideal tool to use for statement troubleshooting – Can be used on active statements Dynamic Sampling – Good way of getting better cardinality estimates – Be cautious when using DS without table stats – Parallel execution chooses the level automatically (11.2) – RWP used level 4 or 5 for data warehouses (11.1)

48. 48 Effect of Cardinality Estimates Access Methods and Join Methods ▲ Indexes ▲ Nested Loop joins ▼ Hash joins ▼ Merge joins ▼ Scans ▼ Indexes ▼ Nested Loop joins ▲ Hash joins ▲ Merge joins ▲ Scans

49. 49 Initialization Parameters Access Methods and Join Methods ▲ Indexes ▲ Nested Loop joins ▼ Hash joins ▼ Merge joins ▼ Scans ▼ Indexes ▼ Nested Loop joins ▲ Hash joins ▲ Merge joins ▲ Scans

50. 50 Initialization Parameters Access Methods and Join Methods ▲ Indexes ▲ Nested Loop joins ▼ Hash joins ▼ Merge joins ▼ Scans

51. 51 What Have We Seen? SQL Tuning Advisor – Helps identify better plans SQL Profile – “Patch” a plan Generated by SQL Tuning Advisor Identified by the user – Force matching can match literals SQLT/SQLTXPLAIN – coe_xfr_sql_profile.sql – See MOS note 215187.1

52. 52 What People Think Are Performance Secrets Undocumented parameters Documented parameters Undocumented events Index rebuilds and table reorgs Fiddling with block size Silver Bullets

53. 53 What Are The Real-World Performance Secrets Use a systematic approach, always Let data (metrics, etc.) guide your troubleshooting Match the scope of the problem and solution Realize that you may not be able to get to the root cause in just one step

54. 54 What Are The Real-World Performance Secrets Use a systematic approach, always Isolate as much as possible Let data (metrics, etc.) guide your troubleshooting Use the right tool for the job Match the scope of the problem and solution Make exactly one change at a time Realize that you may not be able to get to the root cause in just one step

55. 55 The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

56. 56