1. Waits Events and the Geeks who love them Kyle Hailey http://perfvision.com

2. #.2 Copyright 2006 Kyle Hailey Wait Events

3. #.3 Copyright 2006 Kyle Hailey And the Geeks Who Love Them

4. #.4 Copyright 2006 Kyle Hailey In this Presentation:  Tuning Methodology  Plan of Action  Statspacks for Collection Data  Based on Waits  Waits Cheat Sheet  top 36 Waits  Details  4 hardest Waits

5. #.5 Copyright 2006 Kyle Hailey Word of Wisdom  Half of the game is knowing when to act and how much effort to put in

6. #.6 Copyright 2006 Kyle Hailey Database is Hung!  Everybody blames the database  Poor Database  Yet 9 out of 10 dba’s agree it’s not the database  How do you prove it to management?  On the off chance it’s the database, now we are in some serious trouble!

7. #.7 Copyright 2006 Kyle Hailey Database: Guilty until proven innocent *$%@!!

8. #.8 Copyright 2006 Kyle Hailey Oracle’s Defense  After years of false accusations  Oracle took action and created a defense system: WAIT EVENTS To the rescue

9. #.9 Copyright 2006 Kyle Hailey Oracle InstrumentationRedo Lib Cache Buffer Cache IO Locks Network CPU

10. #.10 Copyright 2006 Kyle Hailey Waits  Instrumented code to indicate bottlenecks  Number of times waited  Amount of time waited  Examples  IO  Locks  SQL*Net  Wouldn’t it be great if Applications took some responsibility and instrumented their code?

11. #.11 Copyright 2006 Kyle Hailey Use Statspack to Find Waits Statspack Cheat Sheet:  Install  Connect as SYSDBA  @?/rdbms/admin/spcreate.sql  Run  Exec statspack.snap;  Generate Reports  @?/rdbms/admin/spreport.sql

12. #.12 Copyright 2006 Kyle Hailey Statspack  Look at Top 5 Timed Events (~50 lines down) Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51

13. #.13 Copyright 2006 Kyle Hailey Method  Relax if CPU + WAIT << Available CPU  Get to work if CPU + WAIT >= Available CPU

14. #.14 Copyright 2006 Kyle Hailey Available CPU Available CPU = # CPUs * Elapsed time # of CPUs SQLPLUS> show parameters cpu_count NAME TYPE VALUE ------------------ -------- ---------- cpu_count integer 2 SQLPLUS> show parameters cpu_count NAME TYPE VALUE ------------------ -------- ---------- cpu_count integer 2

15. #.15 Copyright 2006 Kyle Hailey Elapsed Time STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host ------- ----------- -------- -------- ---------- --- ------- LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess --------- ------------------ -------- --------- Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins) STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host ------- ----------- -------- -------- ---------- --- ------- LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess --------- ------------------ -------- --------- Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins)

16. #.16 Copyright 2006 Kyle Hailey Available CPU # CPUs * Elapsed Time Example: 2 CPU * ( 1.0 min * (60 sec/min) ) = 120 secs CPU time available Elapsed: 1.00 (mins)

17. #.17 Copyright 2006 Kyle Hailey Used CPU Time and Wait Time  Look at Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51

18. #.18 Copyright 2006 Kyle Hailey Example  CPU + WAITS = 250 + 32 + 15 + 8 + 5 = 310 secs  Available CPU was 120 secs  310 >> 120  Get to work tuning ! Event Time (s) --------------------------- ----------- buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5 Event Time (s) --------------------------- ----------- buffer busy waits 250 CPU time 32 free buffer waits 15 write complete waits 8 log buffer space 5

19. #.19 Copyright 2006 Kyle Hailey Dependable Tuning Strategy  Run Statspack  Top 5 Timed Events  ~50 lines down from top  Need Available CPU  Elapsed Time  CPU_COUNT  OEM 10g  Performance Page does everything

20. #.20 Copyright 2006 Kyle Hailey Data Sources  Statspack  Top 5 Timed Events  10g ASH  OEM  ASH Report : ashrpt.sql  Custom  V$session, v$session_wait  v$active_session_history

21. #.21 Copyright 2006 Kyle Hailey Statspack STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host ------- ----------- -------- -------- ---------- --- ------- LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess --------- ------------------ -------- --------- Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins) STATSPACK report for DB Name DB Id Instance Inst Num Release RAC Host ------- ----------- -------- -------- ---------- --- ------- LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess --------- ------------------ -------- --------- Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins)

22. #.22 Copyright 2006 Kyle Hailey Statspack Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51

23. #.23 Copyright 2006 Kyle Hailey OEM 10g  9i &10g  Licensed but worth it  Does all the work  Aggregates Wait time  Sums with CPU time  Displays available CPU  Groups Waits  Filters out unusable waits

24. #.24 Copyright 2006 Kyle Hailey OEM 10g Performance Tab

25. #.25 Copyright 2006 Kyle Hailey OEM 10g Zoom-In Available CPU CPU + WAIT

26. #.26 Copyright 2006 Kyle Hailey OEM 10g Zoom-In Relax Get to Work!

27. #.27 Copyright 2006 Kyle Hailey OEM 10g Zoom-In

28. #.28 Copyright 2006 Kyle Hailey Top Activity Page

29. #.29 Copyright 2006 Kyle Hailey Tuning Methodology  Machine  Run queue (CPU)  reduce CPU usage or add CPUs  Paging  Reduce memory usage or add memory  Oracle  Waits + CPU > Available CPU  Tune waits  CPU 100%  Tune SQL We are going to concentrate here on WAITS

30. #.30 Copyright 2006 Kyle Hailey Waits  360 waits in 9i  36 waits represent 99% of all Bottlenecks  From Anjo Kolks site www.oraperf.com  Based on total wait times  12 Wait events represent 80%

31. #.31 Copyright 2006 Kyle Hailey Top 12 Waits NAME Count % Total 1.db file sequential read 23,850.00 11.67% 2.log file sync 20,594.00 10.08% 3.db file scattered read 15,505.00 7.59% 4.latch free 11,078.00 5.42% 5.enqueue 7,732.00 3.78% 6.SQL*Net more data from client 7,510.00 3.67% 7.direct path read 5,840.00 2.86% 8.direct path write 4,868.00 2.38% 9.buffer busy waits 4,589.00 2.25% 10.SQL*Net more data to client 3,805.00 1.86% 11.log buffer space 2,990.00 1.46% 12.log file switch completion 2,878.00 1.41% Above is over 80% of wait times reported Anjo Kolk www.oraperf.com

32. #.32 Copyright 2006 Kyle Hailey Top 36 Waits 19. 19.write complete waits 20. 20.library cache lock 21. 21.SQL*Net more data from dblink 22. 22.log file switch (checkpoint incomplete) 23. 23.library cache load lock 24. 24.row cache lock 25. 25.local write wait 26. 26.sort segment request 27. 27.process startup 28. 28.unread message 29. 29.file identify 30. 30.pipe put 31. 31.switch logfile command 32. 32.SQL*Net break/reset to dblink 33. 33.log file switch (archiving needed) 34. 34.Wait for a undo record 35. 35.direct path write (lob) 36. 36.undo segment extension 1. 1.db file sequential read 2. 2.log file sync 3. 3.db file scattered read 4. 4.latch free 5. 5.enqueue 6. 6.SQL*Net more data from client 7. 7.direct path read 8. 8.direct path write 9. 9.buffer busy waits 10. 10.SQL*Net more data to client 11. 11.log buffer space 12. 12.log file switch completion 13. 13.library cache pin 14. 14.SQL*Net break/reset to client 15. 15.io done 16. 16.file open 17. 17.free buffer waits 18. 18.db file parallel read minus Background, OPS, RAC, PQ, Resource Mgr http://www.oraperf.com

33. #.33 Copyright 2006 Kyle Hailey Waits I/O Library Cache Locks Redo Buffer Cache SQL*Net Wait Areas

34. #.34 Copyright 2006 Kyle Hailey Wait Tree Waits IO Buffer Cache Library Cache Lock Redo SQL Net Buffer Busy Rollback Free lists IO Read Cache Latches Library Cache Shared Pool TX Row Lock TX ITL Lock HW Lock Write IO Read IO Log Buffer Log File Sync Log File

35. #.35 Copyright 2006 Kyle Hailey Redo REDO

36. #.36 Copyright 2006 Kyle Hailey Redo Wait Solutions 2 log file sync Commit less, put redo logs on faster disks 11 log buffer space Increase log buffer 12 log file switch completion Increase log file sizes 22 log file switch (checkpoint incomplete) Add log files (or increase log file size) 30 switch logfile command Avoid switching log files 32 log file switch (archiving needed) *** Archive log running out of space

37. #.37 Copyright 2006 Kyle Hailey Library Cache REDO Lib Cache

38. #.38 Copyright 2006 Kyle Hailey Library Cache 4 latch free – need p1,p2,p3 library cache shared pool 13 library cache pin Reduce parsing the same cursor concurrently 20 library cache lock Avoid hard parsing same cursor at same time 23 library cache load lock Avoid hard parsing same cursor at same time 24 row cache lock need p1

39. #.39 Copyright 2006 Kyle Hailey Buffer Cache REDO Lib Cache Buffer Cache

40. #.40 Copyright 2006 Kyle Hailey Buffer Cache Details 4 latch free – need p1,p2,p3 cache buffers chains cache buffers chains LRU 9 buffer busy waits – need p1,p2,p3 and SQL 11 log buffer space Increase log buffer 17 free buffer waits Increase buffer cache size 19 write complete waits Increase buffer cache (or tune DBWR) 25 local write wait truncates / reduce cache size

41. #.41 Copyright 2006 Kyle Hailey IO REDO Lib Cache Buffer Cache IO

42. #.42 Copyright 2006 Kyle Hailey IO 1 db file sequential read Tune SQL, speed up disks (5-15ms), increase buffer cache 3 db file scattered read FTS - Tune SQL, add indexes, speed up disks (5-15ms) sorts or PQO - tune IO, sort less 7 direct path read sorts or PQO - tune IO, sort less 8 direct path write direct path load or temp io, improve disk speed 15 io done Io done by IO slaves, oracleIxxx could also be an idle wait – sort of worthless 16 file open Reduce logons/logoffs or keep log files open 18 db file parallel read Tune sql, tune io subsystem, increase buffer cache 26 sort segment request SMON busy, process rollback problem 28 file identify Keep log files open, reduce checkpoints 34 direct path write (lob) Improve IO, reduce lob write size

43. #.43 Copyright 2006 Kyle Hailey Locks REDO Lib Cache Buffer Cache IO Locks

44. #.44 Copyright 2006 Kyle Hailey Waits 1-9 5 enqueue – need p1,p2,p3 and SQL HW TM TX 4 TX 6 ST SQ TS

45. #.45 Copyright 2006 Kyle Hailey Network Redo Lib Cache Buffer Cache IO Locks Network

46. #.46 Copyright 2006 Kyle Hailey SQL*Net 6 SQL*Net more data from client Usually OK, reduce data transferred, possible Network problems 10 SQL*Net more data to client Usually OK, reduce amount of data transferred, possible Network tuning needed 14 SQL*Net break/reset to client Check for errors in sql statement 21 SQL*Net more data from dblink Reduce data transfer, check net response 31 SQL*Net break/reset to dblink Check for errors in sql statement sent

47. #.47 Copyright 2006 Kyle Hailey CPURedo Lib Cache Buffer Cache IO Locks Network 100% CPU tune highest SQL

48. #.48 Copyright 2006 Kyle Hailey CPU  Different from Waits  Still a Timed Event  High CPU & Low Waits  Tune SQL

49. #.49 Copyright 2006 Kyle Hailey ApplicationRedo Lib Cache Buffer Cache IO Locks Network ? 100% CPU tune highest SQL

50. #.50 Copyright 2006 Kyle Hailey It’s the Application If all these areas show no contention, it’s the application

51. #.51 Copyright 2006 Kyle Hailey Waits we will Ignore  Background  Idle  RAC  Resource Manager

52. #.52 Copyright 2006 Kyle Hailey Background & Foreground  Background Processes  DBWR  LGWR  PMON  SMON  Etc  Foreground Processes  SQL*Plus  Pro*C  SQL*Forms  Oracle applications Only interested in Foreground waits

53. #.53 Copyright 2006 Kyle Hailey REDO Log Files Data Files DBWR LGWR User2 User1 User3 Log Buffer Buffer Cache Log Buffer Buffer Cache SGA Library Cache Background ProcessesPMON SMON

54. #.54 Copyright 2006 Kyle Hailey Background Waits Filter Out Background Waits  Statspack  ASH : SESSION_TYPE='FOREGROUND‘  V$session_wait : type='USER'

55. #.55 Copyright 2006 Kyle Hailey Background Waits  ASH 10g  Avoid Background waits in ASH with  V$session_wait joined to v$session Select …from v$active_session_history where SESSION_TYPE='FOREGROUND' select … from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER' select … from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER'

56. #.56 Copyright 2006 Kyle Hailey Idle Waits  Filter Out  10g  where wait_class != ‘Idle’  Create a list  9i  Create a list with  Documentation  List created from 10g Select name from v$event_name where wait_class=‘Idle’;

57. #.57 Copyright 2006 Kyle Hailey Parallel Query Waits Filter Out  Parallel Query Wait events are unusable  Save waits are both idle and waits  Parallel Query Waits start with ‘PX’ or ‘KX’  PX Deq: Par Recov Reply  PX Deq: Parse Reply

58. #.58 Copyright 2006 Kyle Hailey RAC Waits  You are on your own  Check documentation  If you are not using RAC then no worries  10g  9i  RAC and OPS waits usually contain the word “global” Select event from v$event_name where wait_class=‘Cluster’;

59. #.59 Copyright 2006 Kyle Hailey Resource Manager Waits  Resource manager throttles user  Creates wait  Obfuscates problems  10g select name from v$event_name where wait_class='Scheduler';

60. #.60 Copyright 2006 Kyle Hailey Easy Waits vs Difficult Waits  Easy Waits  Can take action without further investigation  Difficult Waits, need  Real time analysis or historical collections  V$session_wait  V$session  V$sql

61. #.61 Copyright 2006 Kyle Hailey Easy Waits 1.log buffer space Increase log buffer 2.log file switch completion Increase log file sizes 3.log file switch (checkpoint incomplete) Add log files (or increase log file size) 4.log file switch (archiving needed) Archive log running out of space 5.switch logfile command Avoid switching log files 6.write complete waits Increase buffer cache (or tune DBWR) 7.free buffer waits Increase buffer cache size

62. #.62 Copyright 2006 Kyle Hailey Intermediate  Library Cache locks & Pins  Run queries to find lockers  IO  Check average read times per file  Should be between 5-20 ms  Data in Statspack under “File IO Stats”  Check Cache buffer Hit ratio  Check db_cache_advice 9i and higher  Data in Statspack under “Buffer Pool Advisory”  Tune High IO SQL  Need to follow Advanced Wait steps to find SQL

63. #.63 Copyright 2006 Kyle Hailey Intermediate : Lib Cache Handles 1. library cache lock Avoid hard parsing same cursor at same time 2. library cache pin Reduce parsing the same cursor concurrently 3. library cache load lock Avoid hard parsing same cursor at same time

64. #.64 Copyright 2006 Kyle Hailey Intermediate - IO 1.db file sequential read Tune SQL, speed up disks (5-15ms), increase buffer cache 2.db file parallel read - Tune sql, tune io subsystem, increase buffer cache 3.db file scattered read FTS, Tune SQL, add indexes, speed up disks (5-15ms) 4.direct path write (lob) Improve IO, reduce lob write size 5.direct path read sorts or PQO - tune IO, sort less 6.direct path write direct path load or temp io, improve disk speed

65. #.65 Copyright 2006 Kyle Hailey Library cache pin select w.sid, kglob.KGLNAOBJ from x$kglob kglob, v$session_wait w where kglob.KGLHDADR= w.P1RAW and event like '%library%'; select w.sid, kglob.KGLNAOBJ from x$kglob kglob, v$session_wait w where kglob.KGLHDADR= w.P1RAW and event like '%library%';

66. #.66 Copyright 2006 Kyle Hailey Library cache lock Find the waiters and who blocks them column wevent format a20 column bevent format a20 select waiter.sid waiter, waiter.p1raw wlockp1, waiter.event wevent, blocker_event.sid blocker, blocker_event.event bevent from x$kglpn p, gv$session blocker_session, gv$session_wait waiter, gv$session_wait blocker_event where p.kglpnuse=blocker_session.saddr and p.kglpnhdl=waiter.p1raw and (waiter.event like 'library cache lock' ) and blocker_event.sid=blocker_session.sid order by waiter.p1raw,waiter.sid / column wevent format a20 column bevent format a20 select waiter.sid waiter, waiter.p1raw wlockp1, waiter.event wevent, blocker_event.sid blocker, blocker_event.event bevent from x$kglpn p, gv$session blocker_session, gv$session_wait waiter, gv$session_wait blocker_event where p.kglpnuse=blocker_session.saddr and p.kglpnhdl=waiter.p1raw and (waiter.event like 'library cache lock' ) and blocker_event.sid=blocker_session.sid order by waiter.p1raw,waiter.sid /

67. #.67 Copyright 2006 Kyle Hailey IO 1.db file sequential read Tune SQL, speed up disks (10-20ms), increase buffer cache 2.db file parallel read Tune sql, tune io subsystem, increase buffer cache 3.db file scattered read FTS, Tune SQL, add indexes, speed up disks (5-15ms) 4.direct path write (lob) Improve IO, reduce lob write size 5.direct path read sorts or PQO - tune IO, sort less 6.direct path write direct path load or temp io, improve disk spee

68. #.68 Copyright 2006 Kyle Hailey IO Solutions  If  Db scattered Read  Db file sequential Read  Db file parallel Read  Then  Check average read times per file  Should be between 5-15 ms  Data in Statspack under “File IO Stats”  Check Cache buffer Hit ratio  Check db_cache_advice 9i and higher  Data in Statspack under “Buffer Pool Advisory”  Tune High IO SQL

69. #.69 Copyright 2006 Kyle Hailey IO Solutions : Ave Read Time File IO Stats DB/Inst:labsf03 Snaps: 1-2 ->Mx Rd Bkt: Max bucket time for single block read ->ordered by Tablespace, File Tablespace Filename ------------------------ ---------------------------------------------------- Av Mx Av Av Rd Rd Av Av Buffer BufWt Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms) ---------- ---- ------- ----- --- ------- ------------ -------- ---------- -- SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf 445 15 0.4 16 1.0 1,157 39 2,744 93.3 USERS /u01/app/oracle/oradata/labsf03/users01.dbf 223 7 0.5 ### 1.0 9,725 324 4 100.0 File IO Stats DB/Inst:labsf03 Snaps: 1-2 ->Mx Rd Bkt: Max bucket time for single block read ->ordered by Tablespace, File Tablespace Filename ------------------------ ---------------------------------------------------- Av Mx Av Av Rd Rd Av Av Buffer BufWt Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms) ---------- ---- ------- ----- --- ------- ------------ -------- ---------- -- SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf 445 15 0.4 16 1.0 1,157 39 2,744 93.3 USERS /u01/app/oracle/oradata/labsf03/users01.dbf 223 7 0.5 ### 1.0 9,725 324 4 100.0

70. #.70 Copyright 2006 Kyle Hailey IO Solutions : Buffer Pool Advisory Buffer Pool Advisory Size for Size Buffers for Read Estimated P Est (M) Factor Estimate Factor Physical Reads --- -------- ------ ------------ ------ -------------- D 56.1 6,986 2.3 58,928 D 112.2 13,972 1.6 42,043 D 224.4 27,944 1.0 25,772 D 336.6 41,916 1.0 25,715 D 448.8 55,888 1.0 25,715 D 596 1.0 74,351 1.0 25,715 D 728 1.2 90,818 1.0 25,715 D 840 1.4 104,790 1.0 25,715 D 952 1.6 118,762 1.0 25,715 D 1,064 1.8 132,734 1.0 25,715 Buffer Pool Advisory Size for Size Buffers for Read Estimated P Est (M) Factor Estimate Factor Physical Reads --- -------- ------ ------------ ------ -------------- D 56.1 6,986 2.3 58,928 D 112.2 13,972 1.6 42,043 D 224.4 27,944 1.0 25,772 D 336.6 41,916 1.0 25,715 D 448.8 55,888 1.0 25,715 D 596 1.0 74,351 1.0 25,715 D 728 1.2 90,818 1.0 25,715 D 840 1.4 104,790 1.0 25,715 D 952 1.6 118,762 1.0 25,715 D 1,064 1.8 132,734 1.0 25,715

71. #.71 Copyright 2006 Kyle Hailey IO Solutions  After Checking  File IO response times  Buffer Cache Hit Ratio  Then the only choice is to  Tune the SQL  See Advanced Wait on how to find SQL

72. #.72 Copyright 2006 Kyle Hailey Advanced 1.buffer busy waits  “Buffer wait Statistics”  v$waitstats  P1 file#, p2 block#, p3 class 2.row cache lock  “Dictionary Cache Stats“  v$rowcache  P1 – rowcache # 3.latch free  “Latch Sleep breakdown”  V$latch  P1 latch# 4.Enqueue  Statspack doesn’t help  V$lock  P1 lock type and mode

73. #.73 Copyright 2006 Kyle Hailey Advanced Waits Need  Wait Arguments  P1  P2  P3  SQL Statements  Session IDS

74. #.74 Copyright 2006 Kyle Hailey Gathering More Data - Basic select s.sid, /*SESSION */ w.event, /* WAIT */ s.sql_hash_value, /* SQL */ w.p1, w.p2, w.p3 /* P1, P2, P3 */ from v$session s, v$session_wait w where w.sid=s.sid / select s.sid, /*SESSION */ w.event, /* WAIT */ s.sql_hash_value, /* SQL */ w.p1, w.p2, w.p3 /* P1, P2, P3 */ from v$session s, v$session_wait w where w.sid=s.sid /

75. #.75 Copyright 2006 Kyle Hailey Gathering More Data – SQL Text select w.sid, w.event, w.p1, w.p2, w.p3, sql.sql_text from v$session_wait w, v$session s, v$sql sql where s.sid = sw.sid and sql.address(+) = s.sql_address and s.type='USER‘ / select w.sid, w.event, w.p1, w.p2, w.p3, sql.sql_text from v$session_wait w, v$session s, v$sql sql where s.sid = sw.sid and sql.address(+) = s.sql_address and s.type='USER‘ /

76. #.76 Copyright 2006 Kyle Hailey SQL Text col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 sql_text from v$session s, v$session_wait w, v$sql sql where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') and sql.address(+) = s.sql_address ; col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 sql_text from v$session s, v$session_wait w, v$sql sql where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') and sql.address(+) = s.sql_address ;

77. #.77 Copyright 2006 Kyle Hailey Gathering More Data - Filtering I and and lower(w.event) not in ( 'queue monitor wait', 'null event', 'pl/sql lock timer', 'px %', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', ‘% timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait‘ ) and and lower(w.event) not in ( 'queue monitor wait', 'null event', 'pl/sql lock timer', 'px %', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', ‘% timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait‘ )

78. #.78 Copyright 2006 Kyle Hailey Gathering More Data – Filtering II col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply'); col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply');

79. #.79 Copyright 2006 Kyle Hailey Detailed Data Results SID SQL_HASH STATUS P1 P2 P3 ---------- -------------- -------------------------------- ---------- ---------- ---------- 234 82347421 CPU 1431502854 39 0 235 3336613934 enq: US - contention 1431502854 44 0 236 1772152815 enq: US - contention 1431502854 42 0 238 2750335498 enq: US - contention 1431502854 44 0 240 343101472 enq: US - contention 1431502854 44 0 246 1782401401 enq: US - contention 1431502854 44 0 248 3333220954 CPU 1650815232 1 0 252 323960517 enq: US - contention 1431502854 44 0 260 1272059733 CPU 1431502854 44 0 SID SQL_HASH STATUS P1 P2 P3 ---------- -------------- -------------------------------- ---------- ---------- ---------- 234 82347421 CPU 1431502854 39 0 235 3336613934 enq: US - contention 1431502854 44 0 236 1772152815 enq: US - contention 1431502854 42 0 238 2750335498 enq: US - contention 1431502854 44 0 240 343101472 enq: US - contention 1431502854 44 0 246 1782401401 enq: US - contention 1431502854 44 0 248 3333220954 CPU 1650815232 1 0 252 323960517 enq: US - contention 1431502854 44 0 260 1272059733 CPU 1431502854 44 0

80. #.80 Copyright 2006 Kyle Hailey Wait Arguments: P1,P2,P3  Each Wait has a 3 parameters P1,P2,P3  Give detailed information  Meaning different for each wait  Meaning defined in V$event_name

81. #.81 Copyright 2006 Kyle Hailey Wait Arguments Example NAME PARAMETER1 PARAMETER2 PARAMETER3 ----------------------------------- --------------- --------------- --------------- latch: cache buffers chains address number tries free buffer waits file# block# set-id# buffer busy waits file# block# class# latch: redo copy address number tries log buffer space switch logfile command log file sync buffer# db file sequential read file# block# blocks enq: TM - contention name|mode object # table/partition undo segment extension segment# enq: TX - row lock contention name|mode usn<<16 | slot sequence row cache lock cache id mode request library cache pin handle address pin address 100*mode+namesp library cache load lock object address lock address 100*mask+namesp pipe put handle address record length timeout Select name, parameter 1, parameter 2, parameter 3 from v$event_nam e; Select name, parameter 1, parameter 2, parameter 3 from v$event_nam e;

82. #.82 Copyright 2006 Kyle Hailey Difficult Waits Multiple causes and solutions  Latches  Locks  Buffer Busy  Row Cache Lock

83. #.83 Copyright 2006 Kyle Hailey Latches  Protect memory for concurrent use  “protect lines of code”  Light weight locks  Bit in memory  Atomic processor call  Fast and cheap  Gone if memory is lost  Exclusive Generally  Sharing reading has been introduced for some latches

84. #.84 Copyright 2006 Kyle Hailey Finding Latches  “latch free”  Covers many latches, find the problem latch by 1. select name from v$latchname where latch# = p1; OR 2. Find highest sleeps in Statspack latch section  In 10g, important latches have a wait event  latch: cache buffers chains  latch: shared pool  latch: library cache

85. #.85 Copyright 2006 Kyle Hailey Statspack : Latch free Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------- ----------- -------- CPU time 76 62.62 latch free 168,185 41 33.45 wait list latch free 251 5 3.71 library cache pin 2 0.08 log file parallel write 52 0.06 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------- ----------- -------- CPU time 76 62.62 latch free 168,185 41 33.45 wait list latch free 251 5 3.71 library cache pin 2 0.08 log file parallel write 52 0.06

86. #.86 Copyright 2006 Kyle Hailey Latches : Find Highest Sleeps  Use Statspack (or AWR report) Latch Sleep breakdown for DB: ORA9 -> ordered by misses des Sleeps Latch Name Get Requests Misses Sleeps ----------- ------------------- -------------- ----------- ----------- 19,098 shared pool 3,044,462 184,517 19,098 15,198 library cache 2,242,805 79,554 15,198 1,774 kks stats 263,464 2,842 1,774 147 row cache object 434,844 955 147 54 library cache lock 666,668 734 54 Latch Sleep breakdown for DB: ORA9 -> ordered by misses des Sleeps Latch Name Get Requests Misses Sleeps ----------- ------------------- -------------- ----------- ----------- 19,098 shared pool 3,044,462 184,517 19,098 15,198 library cache 2,242,805 79,554 15,198 1,774 kks stats 263,464 2,842 1,774 147 row cache object 434,844 955 147 54 library cache lock 666,668 734 54

87. #.87 Copyright 2006 Kyle Hailey Easy Latches  Cache Buffers LRU Chain  Too much activity, uses multiple buffer caches and/or increase LRU latches  Shared Pool  Too much hard parsing, too small a shared pool  Library Cache Latch  Hard Parsing  Use bind variables  Uses cursor_sharing=force  Increase shared pool size if reloads are high  Avoid invalidations  Soft Parsing  Hold cursors open  Use session_cached_cursors  Use cursor_space_for_time

88. #.88 Copyright 2006 Kyle Hailey Advanced Latches  Cache Buffers Chains  Hot blocks  Need to find the SQL to solve  Root of Index  Partition  X$dual  Select for update

89. #.89 Copyright 2006 Kyle Hailey Buffer Busy Waits  First need to see p3  Read (100,110,120,130 )  IO  In 10g becomes “read by other session”  Write (200,210,220,230, 231 )  Depends on Block type

90. #.90 Copyright 2006 Kyle Hailey Buffer Busy Cases  Undo Header - Not enough rollback segments  Undo Block – hot spot in RBS  Data Block  Depends on Object type  Segment Header – add free Lists (with Datablock)  Free List – add free lists groups

91. #.91 Copyright 2006 Kyle Hailey Buffer Busy Wait - Object  Table Block w/ DML : Table lacks Free lists  Index : has hot block, partition or change SQL select segment_name, segment_type from dba_extents where file_id = P1 and P2 between block_id and block_id + blocks – 1; select segment_name, segment_type from dba_extents where file_id = P1 and P2 between block_id and block_id + blocks – 1;

92. #.92 Copyright 2006 Kyle Hailey Buffer Busy Waits  Statspack Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: 2 -3 -> ordered by wait time desc, waits desc Tot Wait Avg Class Waits Time (s) Time (ms) ------------------ ----------- ---------- --------- data block 1,490 2 1 undo header 2 0 0 Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: 2 -3 -> ordered by wait time desc, waits desc Tot Wait Avg Class Waits Time (s) Time (ms) ------------------ ----------- ---------- --------- data block 1,490 2 1 undo header 2 0 0

93. #.93 Copyright 2006 Kyle Hailey Buffer Busy Waits  Statspack Buffer wait Statistics DB/Inst: linux3 Snaps: 39-42 -> ordered by wait time desc, waits desc Class Waits Wait Time(s) Avg Time (ms) ------------------ ------ ------------- ------------- undo header 1,159 18 16 data block 132 2 12 file header block 45 0 2 segment header 4 0 10 undo block 1 0 0 Buffer wait Statistics DB/Inst: linux3 Snaps: 39-42 -> ordered by wait time desc, waits desc Class Waits Wait Time(s) Avg Time (ms) ------------------ ------ ------------- ------------- undo header 1,159 18 16 data block 132 2 12 file header block 45 0 2 segment header 4 0 10 undo block 1 0 0

94. #.94 Copyright 2006 Kyle Hailey Buffer Busy Solutions  IF Buffer Busy Waits high  First look at v$waitstat  Class  Time/Count SQL> select * from v$waitstat; CLASS COUNT TIME ------------------ ---------- ---------- data block 1 1 segment header 0 0 free list 0 0 undo header 0 0 undo block 0 0 SQL> select * from v$waitstat; CLASS COUNT TIME ------------------ ---------- ---------- data block 1 1 segment header 0 0 free list 0 0 undo header 0 0 undo block 0 0

95. #.95 Copyright 2006 Kyle Hailey Row Cache Lock  Statspack ^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2 ->"Pct Misses" should be very low (< 2% in most cases) ->"Cache Usage" is the number of cache entries being used ->"Pct SGA" is the ratio of usage to allocated size for that cache Get Pct Scan Pct Mod Final Cache Requests Miss Reqs Miss Reqs Usage ----------------- --------- ------ ------- ----- -------- ---------- dc_object_ids 45 0.0 0 0 958 dc_objects 89 0.0 0 0 1,129 dc_segments 69 0.0 0 0 807 dc_tablespaces 12 0.0 0 0 13 dc_usernames 22 0.0 0 0 19 dc_sequences 120,003 0.0 0 120,003 5 ^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2 ->"Pct Misses" should be very low (< 2% in most cases) ->"Cache Usage" is the number of cache entries being used ->"Pct SGA" is the ratio of usage to allocated size for that cache Get Pct Scan Pct Mod Final Cache Requests Miss Reqs Miss Reqs Usage ----------------- --------- ------ ------- ----- -------- ---------- dc_object_ids 45 0.0 0 0 958 dc_objects 89 0.0 0 0 1,129 dc_segments 69 0.0 0 0 807 dc_tablespaces 12 0.0 0 0 13 dc_usernames 22 0.0 0 0 19 dc_sequences 120,003 0.0 0 120,003 5

96. #.96 Copyright 2006 Kyle Hailey Row Cache Lock  Need p1 to see the cache type SQL> select cache#, parameter from v$rowcache; CACHE# PARAMETER ---------- -------------------------------- 1 dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints SQL> select cache#, parameter from v$rowcache; CACHE# PARAMETER ---------- -------------------------------- 1 dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints

97. #.97 Copyright 2006 Kyle Hailey Row Cache Lock  Find Row Cache  Find SQL Example row cache : sequence sql : select seq.next_val problem : sequence had cache of 1 solution: increase sequence cache to 20

98. #.98 Copyright 2006 Kyle Hailey Enqueues aka Locks  “Enqueue” wait – covers all locks pre 10  Protect data against concurrent changes  Lock info written into data structures  Block headers  Data blocks  Written in cache structures  Shareable in compatible modes

99. #.99 Copyright 2006 Kyle Hailey Finding Locks  Statspack no help  V$session_wait needs lots of decoding  P1 tells Lock Type and Mode  P2,P3 give more data  Usually Need SQL to solve

100. #.100 Copyright 2006 Kyle Hailey Finding Locks select sid, event, chr(bitand(P1,-16777216)/16777215)|| chr(bitand(P1,16711680)/65535) as "Type", mod(p1,16) as "mode" from v$session_wait where event = 'enqueue‘; select sid, event, chr(bitand(P1,-16777216)/16777215)|| chr(bitand(P1,16711680)/65535) as "Type", mod(p1,16) as "mode" from v$session_wait where event = 'enqueue‘; SID EVENT Ty mode P2 P3 --- ------- -- ---- ------- ---- 240 enqueue TX 6 2686995 433 SID EVENT Ty mode P2 P3 --- ------- -- ---- ------- ---- 240 enqueue TX 6 2686995 433

101. #.101 Copyright 2006 Kyle Hailey Locks 10g  10g breaks Enqueues out  enq: HW - contention Configuration  enq: TM - contention Application  enq: TX - allocate ITL entry Configuration  enq: TX - index contention Concurrency  enq: TX - row lock contention Application  enq: UL - contention Application

102. #.102 Copyright 2006 Kyle Hailey Locks : TM & TX select * from v$lock where type in ('TX', 'TM'); SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK --- -- ---------- ---------- ----- ------- ----- ----- 151 TX 589855 1592 0 6 4049 0 135 TM 53737 0 3 0 4058 0 151 TM 53737 0 3 0 4049 0 135 TX 589855 1592 6 0 4058 1 select * from v$lock where type in ('TX', 'TM'); SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK --- -- ---------- ---------- ----- ------- ----- ----- 151 TX 589855 1592 0 6 4049 0 135 TM 53737 0 3 0 4058 0 151 TM 53737 0 3 0 4049 0 135 TX 589855 1592 6 0 4058 1

103. #.103 Copyright 2006 Kyle Hailey Utllockt.sql  @?/rdbms/admin/catlock.sql  @?/rdbms/admin/utllockt.sql WAIT_SES LOCK_TYPE MODE_REQ MODE_HELD LOCK_ID1 LOCK_ID2 --------- ----------- --------- --------- -------- -------- 13 None 12 Transaction Exclusive Exclusive 65537 14960 WAIT_SES LOCK_TYPE MODE_REQ MODE_HELD LOCK_ID1 LOCK_ID2 --------- ----------- --------- --------- -------- -------- 13 None 12 Transaction Exclusive Exclusive 65537 14960

104. #.104 Copyright 2006 Kyle Hailey Summary  Waits make Tuning Easy  Check Machine Health  Tune Waits  Tune CPU  Tune SQL  Use Statspack or OEM 10g  Ignore Background, Idle, PQO (Rac, Resmgr)  Cheat Sheet on Top 36 Waits  Details on the 4 hardest Waits  See http://perfvision.com for more info

105. #.105 Copyright 2006 Kyle Hailey 10g v$active_session_history  Best source  10g only  Data exists since v7  Can simulate v$active_session_history

106. #.106 Copyright 2006 Kyle Hailey Querying ASHselect event as "TOP 5 Timed Events", event as "TOP 5 Timed Events", cnt as "Count" from ( cnt as "Count" from ( select decode(ash.session_state, select decode(ash.session_state, 'ON CPU','CPU',en.name) event, 'ON CPU','CPU',en.name) event, count(decode(ash.session_state, count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt 'ON CPU','CPU',en.name)) cntfrom v$active_session_history ash, v$active_session_history ash, v$event_name en v$event_name en where en.event# (+) = ash.event# and SAMPLE_TIME > sysdate - (5/(24*60)) and SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name) order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc 'ON CPU','CPU',en.name)) desc ) where rownum < 6; select event as "TOP 5 Timed Events", event as "TOP 5 Timed Events", cnt as "Count" from ( cnt as "Count" from ( select decode(ash.session_state, select decode(ash.session_state, 'ON CPU','CPU',en.name) event, 'ON CPU','CPU',en.name) event, count(decode(ash.session_state, count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt 'ON CPU','CPU',en.name)) cntfrom v$active_session_history ash, v$active_session_history ash, v$event_name en v$event_name en where en.event# (+) = ash.event# and SAMPLE_TIME > sysdate - (5/(24*60)) and SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name) order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc 'ON CPU','CPU',en.name)) desc ) where rownum < 6; Top 5 Timed Events Count ------------------------------------ --------- log file parallel write 8 CPU 7 control file parallel write 3 null event 3 log file sync 2 Top 5 Timed Events Count ------------------------------------ --------- log file parallel write 8 CPU 7 control file parallel write 3 null event 3 log file sync 2

107. #.107 Copyright 2006 Kyle Hailey Querying ASH for P1,P2,P3 SID EVENT P1 P2 P3 --- ----------------------- --- --- --- 236 db file scattered read 3 35291 16 245 db file scattered read 1 13646 2 248 read by other session 1 13646 1 245 db file scattered read 1 8457 7 248 db file scattered read 1 42226 8 245 db file sequential read 1 13591 1 SID EVENT P1 P2 P3 --- ----------------------- --- --- --- 236 db file scattered read 3 35291 16 245 db file scattered read 1 13646 2 248 read by other session 1 13646 1 245 db file scattered read 1 8457 7 248 db file scattered read 1 42226 8 245 db file sequential read 1 13591 1 selectsession_id,event, p1, p2, p3 from v$ash selectsession_id,event, p1, p2, p3 from v$ash

108. #.108 Copyright 2006 Kyle Hailey ASH Report  ASH report  @?/rdbms/admin/ashrpt.sql  Pick interval over last 7 days ! 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***

109. #.109 Copyright 2006 Kyle Hailey V$session_wait select decode(w.wait_time, 0, w.event, 'CPU') as "TOP 5 Timed Events”, count(*) from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') group by decode(w.wait_time, 0, w.event, 'CPU') order by count(*) desc; select decode(w.wait_time, 0, w.event, 'CPU') as "TOP 5 Timed Events”, count(*) from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') group by decode(w.wait_time, 0, w.event, 'CPU') order by count(*) desc; Moment in Time data

110. #.110 Copyright 2006 Kyle Hailey V$session_wait Top 5 Timed Events Top 5 Timed Events COUNT ----------------------------------- ---------- enq: US – contention 4 CPU 3 buffer busy waits 1 latch: row cache objects 1 Top 5 Timed Events Top 5 Timed Events COUNT ----------------------------------- ---------- enq: US – contention 4 CPU 3 buffer busy waits 1 latch: row cache objects 1

111. #.111 Copyright 2006 Kyle Hailey V$session_wait col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply'); col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event, 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply');

112. #.112 Copyright 2006 Kyle Hailey V$session_wait SID SQL_HASH STATUS P1 P2 P3 ---------- -------------- -------------------------------- ---------- ---------- ---------- 234 82347421 CPU 1431502854 39 0 235 3336613934 enq: US - contention 1431502854 44 0 236 1772152815 enq: US - contention 1431502854 42 0 238 2750335498 enq: US - contention 1431502854 44 0 240 343101472 enq: US - contention 1431502854 44 0 246 1782401401 enq: US - contention 1431502854 44 0 248 3333220954 CPU 1650815232 1 0 252 323960517 enq: US - contention 1431502854 44 0 260 1272059733 CPU 1431502854 44 0 SID SQL_HASH STATUS P1 P2 P3 ---------- -------------- -------------------------------- ---------- ---------- ---------- 234 82347421 CPU 1431502854 39 0 235 3336613934 enq: US - contention 1431502854 44 0 236 1772152815 enq: US - contention 1431502854 42 0 238 2750335498 enq: US - contention 1431502854 44 0 240 343101472 enq: US - contention 1431502854 44 0 246 1782401401 enq: US - contention 1431502854 44 0 248 3333220954 CPU 1650815232 1 0 252 323960517 enq: US - contention 1431502854 44 0 260 1272059733 CPU 1431502854 44 0 Moment in Time data

113. #.113 Copyright 2006 Kyle Hailey Waits 1-9 1.db file sequential read Tune SQL, speed up disks (5-15ms), increase buffer cache 2.log file sync Commit less, put redo logs on faster disks 3.db file scattered read FTS - Tune SQL, add indexes, speed up disks (5-15ms) 4.latch free – need p1,p2,p3 and sometimes SQL 5.enqueue – need p1,p2,p3 and SQL 6.SQL*Net more data from client Usually OK, reduce data transferred, possible Network problems 7.direct path read sorts or PQO - tune IO, sort less 8.direct path write direct path load or temp io, improve disk speed 9.buffer busy waits – need p1,p2,p3

114. #.114 Copyright 2006 Kyle Hailey Waits 10-18 10.SQL*Net more data to client Usually OK, reduce amount of data transferred, possible Network tuning needed 11.log buffer space Increase log buffer 12.log file switch completion Increase log file sizes 13.library cache pin Reduce parsing the same cursor concurently 14.SQL*Net break/reset to client Check for errors in sql statement 15.io done Io done by IO slaves, oracleIxxx could also be an idle wait – sort of worthless 16.file open Reduce logons/logoffs or keep log files open 17.free buffer waits Increase buffer cache size 18.db file parallel read Tune sql, tune io subsystem, increase buffer cache

115. #.115 Copyright 2006 Kyle Hailey Waits 19-27 19.write complete waits Increase buffer cache (or tune DBWR) 20.library cache lock Avoid hard parsing same cursor at same time 21.SQL*Net more data from dblink Reduce data transfer, check net response 22.log file switch (checkpoint incomplete) Add log files (or increase log file size) 23.library cache load lock Avoid hard parsing same cursor at same time 24.row cache lock need p1 & SQL 25.local write wait truncates / reduce cache size 26.sort segment request SMON busy, process rollback problem 27.process startup If MTS, increase min servers and dispatchers in init.ora

116. #.116 Copyright 2006 Kyle Hailey Waits 28-36 28.file identify Keep log files open, reduce checkpoints 29.pipe put Speed up pipe readers 30.switch logfile command Avoid switching log files 31.SQL*Net break/reset to dblink Check for errors in sql statement sent 32.log file switch (archiving needed) Archive log running out of space 33.Wait for a undo record ?? 34.direct path write (lob) Improve IO, reduce lob write size 35.undo segment extension Use UNDO or with RBS, increase RBS size, avoid OPTIMAL 36.undo segment tx slot Use UNDO, increae # of RBS segs

117. #.117 Copyright 2006 Kyle Hailey Custom Collecting  collecting: ash.collect(sleep,loops) save data in "ash_data" table  sleep = wait time between loops  loops = # of loops  Debug or testing: ash.print(sleep,loops) prints with dbms_output to see output, run  set serveroutput on  execute dbms_output.enable(1000000)

118. #.118 Copyright 2006 Kyle Hailey How much data to keep create view ash_data_v as select * from ash_data_1 union all select * from ash_data_2 union all select * from ash_data_3; create view ash_data_v as select * from ash_data_1 union all select * from ash_data_2 union all select * from ash_data_3; internally max_rows number := 10000; sets max rows saved in each of 3 tables in view ash_data_v

119. #.119 Copyright 2006 Kyle Hailey Sampling ASH Data  Foreground and Background  Waiting and CPU  Mimics main Fields of ASH

120. #.120 Copyright 2006 Kyle Hailey Sampling Select to_char(sysdate,'SSSSS')+ trunc(sysdate-to_date('JAN-01-1970 00:00:00','MON-DD-YYYY HH24:MI:SS'))*86400, sysdate, s.indx "SID", decode(w.ksusstim, 0,decode(n.kslednam, 'db file sequential read', 'I/O', 'db file scattered read','I/O', 'WAITING'), 'CPU') "STATE", s.ksuseser "SERIAL#", s.ksuudlui "USER#", s.ksusesql "SQL_ADDRESS", s.ksusesqh "SQL_HASH_VALUE", s.ksuudoct "COMMAND" /* aka SQL_OPCODE */, s.ksuseflg "SESSION_TYPE", w.ksussopc "EVENT# ", w.ksussseq "SEQ#" /* xksuse.ksuseseq */, w.ksussp1 "P1" /* xksuse.ksusep1 */, w.ksussp2 "P2" /* xksuse.ksusep2 */, w.ksussp3 "P3" /* xksuse.ksusep3 */, w.ksusstim "WAIT_TIME" /* xksuse.ksusetim */, s.ksuseobj "ROW_WAIT_OBJ#", s.ksusefil "ROW_WAIT_FILE#", s.ksuseblk "ROW_WAIT_BLOCK#", s.ksusepnm "PROGRAM", s.ksuseaph "MODULE_HASH", /* ASH collects string */ s.ksuseach "ACTION_HASH" /* ASH collects string */ Select to_char(sysdate,'SSSSS')+ trunc(sysdate-to_date('JAN-01-1970 00:00:00','MON-DD-YYYY HH24:MI:SS'))*86400, sysdate, s.indx "SID", decode(w.ksusstim, 0,decode(n.kslednam, 'db file sequential read', 'I/O', 'db file scattered read','I/O', 'WAITING'), 'CPU') "STATE", s.ksuseser "SERIAL#", s.ksuudlui "USER#", s.ksusesql "SQL_ADDRESS", s.ksusesqh "SQL_HASH_VALUE", s.ksuudoct "COMMAND" /* aka SQL_OPCODE */, s.ksuseflg "SESSION_TYPE", w.ksussopc "EVENT# ", w.ksussseq "SEQ#" /* xksuse.ksuseseq */, w.ksussp1 "P1" /* xksuse.ksusep1 */, w.ksussp2 "P2" /* xksuse.ksusep2 */, w.ksussp3 "P3" /* xksuse.ksusep3 */, w.ksusstim "WAIT_TIME" /* xksuse.ksusetim */, s.ksuseobj "ROW_WAIT_OBJ#", s.ksusefil "ROW_WAIT_FILE#", s.ksuseblk "ROW_WAIT_BLOCK#", s.ksusepnm "PROGRAM", s.ksuseaph "MODULE_HASH", /* ASH collects string */ s.ksuseach "ACTION_HASH" /* ASH collects string */

121. #.121 Copyright 2006 Kyle Hailey Sampling from x$ksuse s, x$ksusecst w, x$ksled n where s.indx != ( select distinct sid from v$mystat ) and bitand(s.ksspaflg,1)!=0 and bitand(s.ksuseflg,1)!=0 and n.indx=w.ksussopc and s.indx = w.indx and ( ( /* status Active - seems inactive & "on cpu"=> not on CPU */ w.ksusstim != 0 and /* on CPU */ bitand(s.ksuseidl,11)=1 /* ACTIVE */ ) or w.ksussopc not in /* waiting and the wait event is not idle */ from x$ksuse s, x$ksusecst w, x$ksled n where s.indx != ( select distinct sid from v$mystat ) and bitand(s.ksspaflg,1)!=0 and bitand(s.ksuseflg,1)!=0 and n.indx=w.ksussopc and s.indx = w.indx and ( ( /* status Active - seems inactive & "on cpu"=> not on CPU */ w.ksusstim != 0 and /* on CPU */ bitand(s.ksuseidl,11)=1 /* ACTIVE */ ) or w.ksussopc not in /* waiting and the wait event is not idle */

122. #.122 Copyright 2006 Kyle Hailey Sampling Select event# from v$event_name where lower(name) in ( 'queue monitor wait', 'null event', 'pl/sql lock timer', 'px deq: execution msg', 'px deq: table q normal', 'px idle wait', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', 'pmon timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'smon timer', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait', 'SQL*Net message from client' ) Select event# from v$event_name where lower(name) in ( 'queue monitor wait', 'null event', 'pl/sql lock timer', 'px deq: execution msg', 'px deq: table q normal', 'px idle wait', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', 'pmon timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'smon timer', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait', 'SQL*Net message from client' )

123. #.123 Copyright 2006 Kyle Hailey