1 06/05/08 Statspack Kyle Hailey

Kyle Hailey Brief History  Weakness of Ratios  Ratio based guess work – no waits  Utlbstat/Utlestat  Lists of rules – no global picture/perspective  Power of Waits  Time gains and losses clear  Undocumented  “Expert Only”  Statspack  Top 5 Wait Events  Filtered out Idle Events  Compulsive Tuning Disorder  iR2 CPU  Top 5 Timed Events (includes CPU)  Response Time = Service Time (CPU) + Wait Time  g Simplicity of AAS  Average Active Session (AAS) = DB Time / Elapsed Time  g SQL Row Source Counters Timings Sampling

V6 Problem  How do you shorten your daily commute?  Analyze Statistics ?  RPM  Number of rotations of wheels?  Average amount of gas in tank? No ! Solution :  Analyze Time  Where do you get blocked?  Are there alternatives ? Oracle 6 didn’t have wait times, but you do !

Ratios, Example v6  Wait timings unavailable for IO  Db file sequential read  Db file scattered read  Direct path read  Ratio : Cache Buffer Hit  proxy for IO wait time  Unreliable  High ratio  Bad – untuned SQL doing mega LIOs  Low ratio  Good - I/Os are coming from UNIX file cache or disk cache Reliability: IO Wait Times 06/05/08 Wait Events replaced Ratios !

Oracle 7  Waits introduced in version 7  Quantum Step  1992 slipped in for a benchmark  by kernel architect (now Senior VP) Juan Loaiza in 1992  Benchmark couldn’t find where time was being spent  Wait instrumentation solved the mystery  Originally Undocumented  Still badly documented course should help change that  Experts only Thus, ratio based tuning dominated 06/05/08

Copyright 2006 Kyle Hailey Oracle Instrumentation Database: Guilty until proven innocent Redo Lib Cache Buffer Cache IO Locks Network CPU Database

Statspack Introduced in by Connie Dialeris  Snapshots  Selects from 55 tables and views  x$ structures  v$ views  DBA views  Number of tables has expanded with each version  30 plus tables in Oracle8i  40 plus tables in Oracle9i  55 plus tables in Oracle10g  Calculates Deltas (and ratios)  Reports lines output, about 30 pages 2006 Kyle Hailey

Copyright 2006 Kyle Hailey Statspack: Cheat Sheet  Install  Connect as SYSDBA  Run  Exec statspack.snap;  Generate Reports  (AWR )

Statspack Sections (10g) Instance description Host hardware Snapshot headline Cache information Load profile Instance efficiency Shared pool stats Top timed events Host CPU load Host / Instance CPU warning VM activity Memory usage Time model stats RAC statistics All wait events Background wait events Event histogram SQL x 9 Instance activity Log switches OS stats Latch parent and child Mutex Segment stats x 8 Dictionary cache Library cache RAC (GES) RAC (CR and CUR served) RAC (cache xfer x 2) RAC (Remastering) Streams x 7 Shared pool advisor Java pool advisor SGA resizing SGA target advisor SGA summary SGA detail SQL memory summary Resource limits Parameters Session details x 3 Tablespace I/O File I/O File I/O histogram Buffer pool Instance Recovery Buffer pool advisory Buffer busy waits Various PGA summaries PGA histogram PGA advisory PGA allocation summary PGA allocation top N Enqueue (lock) activity Undo stats x 2 Latch activity Latch miss details

Statspack Method Statspack Method 06/05/08 Load profile good for having a feel for the application and comparing two periods for changes Efficiency ratios misleading carry over from version 6 days Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space Summary Waits Who/When Big Picture

Oracle 8 06/05/08 Top 5 Wait Events ~~~~~~~~~~~~~~~~~ Wait % Total Event Waits Time (cs) Wt Time direct path read 4,232 10, db file scattered read 6,105 6, direct path write 1,992 3, control file parallel write db file parallel write Statistic Total per Second per Trans CPU used by this session 358, ,372.6 Missing the CPU CPU time = 95% Eliminating all IO gains 3 % ! IO Looks horrible, but wait … Total time in secs = = /3792=95% 108/3792=3%

Oracle 9 06/05/08  Statspack adds CPU  Method  % wait time = potential performance gain Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time CPU time 8, direct path write 1, db file scattered read 4, direct path read db file parallel write Max gain is 6.22% by tuning all waits In other words go tune the top SQL % gain = Savings/Service Time Service Time = CPU time + Wait Time Service Time =DB Time in 10g

Problems in 9 1.Lack of Data to solve issues Examples 1.Buffer busy wait 2.latch free 3.db file scattered read (IO) 4.Enqueue (lock)

Buffer Busy Wait 06/05/08 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time buffer busy waits 2, CPU time free buffer waits 1, write complete waits log buffer space Buffer Busy Wait – trying to modify a block a buffer. Who blocks, what kind of buffer, what is the SQL? Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3 Class Waits Wait Time (s) Avg Time (ms) file header block data block 6, undo header Statspack fails for analysis

Latch Free 06/05/08 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time latch free 9, CPU time PL/SQL lock timer SQL*Net message from dblink log file parallel write What Latch? There are 100s Latch Sleep breakdown for DB: CDB Instance: cdb Snaps: > ordered by misses desc Sleeps Latch Name Requests Misses Sleeps Sleeps 1-> cache buffers chains 8,448, ,484 6,930 0/0/0/0/0 library cache pin 8,405,896 82,915 1, /1330/4 7/1/0 library cache 8,435,488 55,645 1, /1247/2 2/1/0 shared pool 58, /1/0/0/0

IO 06/05/08 Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time db file scattered read 7, CPU time db file sequential read control file sequential read control file parallel write Are the disks slow? What files? Is the buffer cache too small? IO – what SQL? 5/7549=0.6ms Very fast Problem is SQL

IO 06/05/08 Is the buffer cache too small? IO – what SQL? File IO Stats for DB: ORA9 Instance: ora9 Snaps: Tablespace Filename Av Av Av Av Av Buffer Av Buf Rd(ms) Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms) EXAMPLE D:\ORACLE\ORADATA\ORA9\EXAMPLE01.DBF SYSTEM D:\ORACLE\ORADATA\ORA9\SYSTEM01.DBF 0.7 8, UNDOTBS1 D:\ORACLE\ORADATA\ORA9\UNDOTBS01.DBF Are the disks slow? What files? < 10ms

IO 06/05/08 IO – what SQL? Buffer Pool Advisory for DB: ORA9 Instance: ora9 End Snap: 22 Size for Size Buffers for Est Physical Estimated P Estimate (M) Factr Estimate Read Factor Physical Reads D ,973 D , ,668 D , ,612 D , ,612 D , ,612 D , ,612 D , ,132 D , ,132 D , ,132 Is the buffer cache too small?

IO 06/05/08 Sorted by Reads, not Time (reads proxy for IO wait time) SQL ordered by Reads for DB: ORA9 Instance: ora9 Snaps: CPU Elapsd Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value , , Module: SQL*Plus select count(*) from sys.source$ Module: SQL*Plus select count(*) from all_objects select t.schema, t.name, t.flags, q.name from system.aq$_queue_t ables t, sys.aq$_queue_table_affinities aft, system.aq$_que What SQL?

Row Locks 10g+ 06/05/08 op 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time enq: TX - row lock contention PL/SQL lock timer CPU time buffer busy waits 1, log file parallel write Who is waiting Who is blocking What is the SQL What is the row?

2110/26/2015 Current Method  1000 lines of data / 30 pages  What do you look at ?  Top 5 Timed Events? Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time CPU time 11, log file sync 163,725 1, enqueue 4, latch free 28, db file sequential read 1,420, Jonathan Lewis minutes CPU Elapsed 15 minutes 48 Processors or 74% idle

Statspack Top 5 Timed Events missing  CPU Count  Elapsed Time  As well as drill down diagnostic data Take a step back, take deep breath, and start over  Start with Typical Problems  Identify Solutions  Design a Visual Representation  Put intelligence into the interface 06/05/08

OEM 10g Simple, Powerful Database Load Load Source: SQL Session Wait