1. 1 06/05/08 Statspack Kyle Hailey http://oraclemonitor.com

2. Copyright @ 2006 Kyle Hailey Brief History  1989 6 Weakness of Ratios  Ratio based guess work – no waits  Utlbstat/Utlestat  Lists of rules – no global picture/perspective  1993 7 Power of Waits  Time gains and losses clear  Undocumented  “Expert Only”  1999 8.1.6 Statspack  Top 5 Wait Events  Filtered out Idle Events  Compulsive Tuning Disorder  2001 9iR2 CPU  Top 5 Timed Events (includes CPU)  Response Time = Service Time (CPU) + Wait Time  2003 10g Simplicity of AAS  Average Active Session (AAS) = DB Time / Elapsed Time  2007 11g SQL Row Source Counters Timings Sampling

3. V6 Problem  How do you shorten your daily commute?  Analyze Statistics ?  RPM  Number of rotations of wheels?  Average amount of gas in tank? No ! Solution :  Analyze Time  Where do you get blocked?  Are there alternatives ? Oracle 6 didn’t have wait times, but you do !

4. Ratios, Example v6  Wait timings unavailable for IO  Db file sequential read  Db file scattered read  Direct path read  Ratio : Cache Buffer Hit  proxy for IO wait time  Unreliable  High ratio  Bad – untuned SQL doing mega LIOs  Low ratio  Good - I/Os are coming from UNIX file cache or disk cache Reliability: IO Wait Times 06/05/08 Wait Events replaced Ratios !

5. Oracle 7  Waits introduced in version 7  Quantum Step  1992 slipped in for a benchmark  by kernel architect (now Senior VP) Juan Loaiza in 1992  Benchmark couldn’t find where time was being spent  Wait instrumentation solved the mystery  Originally Undocumented  Still badly documented course should help change that  Experts only Thus, ratio based tuning dominated 06/05/08

6. Copyright 2006 Kyle Hailey Oracle Instrumentation *$%@!! Database: Guilty until proven innocent Redo Lib Cache Buffer Cache IO Locks Network CPU Database

7. Statspack Introduced in 8.1.6 by Connie Dialeris  Snapshots  Selects from 55 tables and views  x$ structures  v$ views  DBA views  Number of tables has expanded with each version  30 plus tables in Oracle8i  40 plus tables in Oracle9i  55 plus tables in Oracle10g  Calculates Deltas (and ratios)  Reports 1000-2000 lines output, about 30 pages Copyright @ 2006 Kyle Hailey

8. Copyright 2006 Kyle Hailey Statspack: Cheat Sheet  Install  Connect as SYSDBA  @?/rdbms/admin/spcreate.sql  Run  Exec statspack.snap;  Generate Reports  @?/rdbms/admin/spreport.sql  (AWR : @?/rdbms/admin/awrrpt.sql )

9. Statspack Sections (10g) Instance description Host hardware Snapshot headline Cache information Load profile Instance efficiency Shared pool stats Top timed events Host CPU load Host / Instance CPU warning VM activity Memory usage Time model stats RAC statistics All wait events Background wait events Event histogram SQL x 9 Instance activity Log switches OS stats Latch parent and child Mutex Segment stats x 8 Dictionary cache Library cache RAC (GES) RAC (CR and CUR served) RAC (cache xfer x 2) RAC (Remastering) Streams x 7 Shared pool advisor Java pool advisor SGA resizing SGA target advisor SGA summary SGA detail SQL memory summary Resource limits Parameters Session details x 3 Tablespace I/O File I/O File I/O histogram Buffer pool Instance Recovery Buffer pool advisory Buffer busy waits Various PGA summaries PGA histogram PGA advisory PGA allocation summary PGA allocation top N Enqueue (lock) activity Undo stats x 2 Latch activity Latch miss details

10. Statspack Method Statspack Method 06/05/08 Load profile good for having a feel for the application and comparing two periods for changes Efficiency ratios misleading carry over from version 6 days Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time ---------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ---------------------------------------- Summary Waits Who/When Big Picture

11. Oracle 8 06/05/08 Top 5 Wait Events ~~~~~~~~~~~~~~~~~ Wait % Total Event Waits Time (cs) Wt Time -------------------------------------------- ------------ ------------ ------- direct path read 4,232 10,827 52.01 db file scattered read 6,105 6,264 30.09 direct path write 1,992 3,268 15.70 control file parallel write 893 198.95 db file parallel write 40 131.63 Statistic Total per Second per Trans --------------------------------- ---------------- ------------ ------------ CPU used by this session 358,806 130.5 12,372.6 Missing the CPU CPU time = 95% Eliminating all IO gains 3 % ! IO Looks horrible, but wait … Total time in secs = 108+62+32+1+1+3588=3792 3588/3792=95% 108/3792=3%

12. Oracle 9 06/05/08  Statspack adds CPU  Method  % wait time = potential performance gain Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------------------- ------------ ----------- -------- CPU time 8,026 93.78 direct path write 1,930 421 4.92 db file scattered read 4,746 69.80 direct path read 77 18.21 db file parallel write 23 11.13 ------------------------------------------------------------ Max gain is 6.22% by tuning all waits In other words go tune the top SQL % gain = Savings/Service Time Service Time = CPU time + Wait Time Service Time =DB Time in 10g

13. Problems in 9 1.Lack of Data to solve issues Examples 1.Buffer busy wait 2.latch free 3.db file scattered read (IO) 4.Enqueue (lock)

14. Buffer Busy Wait 06/05/08 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------ Buffer Busy Wait – trying to modify a block a buffer. Who blocks, what kind of buffer, what is the SQL? Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3 Class Waits Wait Time (s) Avg Time (ms) ------------------ ----- ------------- ------------- file header block 264 203 769 data block 6,070 162 27 undo header 355 0 1 Statspack fails for analysis

15. Latch Free 06/05/08 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------------------- ------------ ----------- -------- latch free 9,652 760 66.10 CPU time 248 21.62 PL/SQL lock timer 41 123 10.72 SQL*Net message from dblink 681 14 1.22 log file parallel write 128 1.13 ------------------------------------------------------------- What Latch? There are 100s Latch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4 -> ordered by misses desc Sleeps Latch Name Requests Misses Sleeps Sleeps 1->4 -------------------------- -------------- ----------- ----------- ------------ cache buffers chains 8,448,787 649,484 6,930 0/0/0/0/0 library cache pin 8,405,896 82,915 1,427 81537/1330/4 7/1/0 library cache 8,435,488 55,645 1,294 54375/1247/2 2/1/0 shared pool 58,626 7 1 6/1/0/0/0

16. IO 06/05/08 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time ------------------------ ------------ ----------- -------- db file scattered read 7,549 5 52.26 CPU time 2 23.29 db file sequential read 884 2 19.11 control file sequential read 154 0 2.68 control file parallel write 255 0 1.87 Are the disks slow? What files? Is the buffer cache too small? IO – what SQL? 5/7549=0.6ms Very fast Problem is SQL

17. IO 06/05/08 Is the buffer cache too small? IO – what SQL? File IO Stats for DB: ORA9 Instance: ora9 Snaps: 21 -22 Tablespace Filename ------------------------ ---------------------------------------------------- Av Av Av Av Av Buffer Av Buf Rd(ms) Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms) -------------- ------- ------ ------- ------------ -------- ---------- ------ EXAMPLE D:\ORACLE\ORADATA\ORA9\EXAMPLE01.DBF 3.1 49 0 3.1 3.0 0 0 0 SYSTEM D:\ORACLE\ORADATA\ORA9\SYSTEM01.DBF 0.7 8,331 11 0.7 7.3 3 0 0 UNDOTBS1 D:\ORACLE\ORADATA\ORA9\UNDOTBS01.DBF 8.0 5 0 8.0 1.0 0 0 0 Are the disks slow? What files? < 10ms

18. IO 06/05/08 IO – what SQL? Buffer Pool Advisory for DB: ORA9 Instance: ora9 End Snap: 22 Size for Size Buffers for Est Physical Estimated P Estimate (M) Factr Estimate Read Factor Physical Reads --- ------------ ----- ---------------- ------------- ------------------ D 4.1 979 1.05 67,973 D 16.5 3,916 1.00 64,668 D 32 1.0 7,832 1.00 64,612 D 48 1.5 11,748 1.00 64,612 D 64 2.0 15,664 1.00 64,612 D 68 2.1 16,643 1.00 64,612 D 72 2.3 17,622 0.30 19,132 D 76 2.4 18,601 0.30 19,132 D 80 2.5 19,580 0.30 19,132 Is the buffer cache too small?

19. IO 06/05/08 Sorted by Reads, not Time (reads proxy for IO wait time) SQL ordered by Reads for DB: ORA9 Instance: ora9 Snaps: 21 -22 CPU Elapsd Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value --------------- ------------ -------------- ------ -------- --------- ---------- 60,197 4 15,049.3 98.4 0.48 5.04 1432830824 Module: SQL*Plus select count(*) from sys.source$ 666 1 666.0 1.1 0.70 1.46 2893531678 Module: SQL*Plus select count(*) from all_objects 232 24 9.7 0.4 0.53 0.97 238087931 select t.schema, t.name, t.flags, q.name from system.aq$_queue_t ables t, sys.aq$_queue_table_affinities aft, system.aq$_que What SQL?

20. Row Locks 10g+ 06/05/08 op 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time ----------------------------------------- ------------ ----------- ------ ------ enq: TX - row lock contention 59 160 2714 41.8 PL/SQL lock timer 4 117 29291 30.6 CPU time 28 7.2 buffer busy waits 1,217 18 15 4.7 log file parallel write 422 11 27 3.0 Who is waiting Who is blocking What is the SQL What is the row?

21. 2110/26/2015 Current Method  1000 lines of data / 30 pages  What do you look at ?  Top 5 Timed Events? Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time ------------------------ ------------ ----------- -------- CPU time 11,330 81.95 log file sync 163,725 1,012 7.32 enqueue 4,486 615 4.45 latch free 28,786 236 1.71 db file sequential read 1,420,355 149 1.08 ----------------------------------------- Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html 188 minutes CPU Elapsed 15 minutes 48 Processors or 74% idle

22. Statspack Top 5 Timed Events missing  CPU Count  Elapsed Time  As well as drill down diagnostic data Take a step back, take deep breath, and start over  Start with Typical Problems  Identify Solutions  Design a Visual Representation  Put intelligence into the interface 06/05/08

23. OEM 10g Simple, Powerful Database Load Load Source: SQL Session Wait