1. IS ORACLE 12C MAKING YOU WAIT? DISCOVER HOW TO IDENTIFY, PRIORITIZE AND SOLVE THE MOST IMPORTANT WAIT EVENTS Janis Griffin Senior DBA / Performance Evangelist

2. Who am I? Senior DBA / Performance Evangelist for Solarwinds o Janis.Griffin@solarwinds.com Janis.Griffin@solarwinds.com o Twitter - @DoBoutAnything o Current – 25+ Years in Oracle, SQL Server, ASE, now MySQL o DBA and Developer Specialize in Performance Tuning Review Database Performance for Customers and Prospects Common Question – How do I fix this wait event? 2

3. AGENDA 3 What Are Wait Events Wait Time Tables Top 10 List of Wait Events Several Solutions For Each o To Reduce Wait Time

4. Oracle has been instrumented to give clues about what it is doing when processing SQL statements Understand the total time a SQL statement spends in Database Measure time / capture wait events while SQL statement executes What are Wait Events Focus on Wait Time 4

5. Events have 0-3 parameters that provide more information o Example: db file sequential read – P1=file#, P2=block#, P3=blocks Knowing what a query waits on - gives a starting place o Locking issues may lead to a different solution o Than if it were waiting on disk reads Oracle 10g – 800+ wait events Oracle 11g – 1100+ wait events Oracle 12c – 1500+ wait events Good news: only need to know small (very small) subset of them o If you know the ones in this presentation, it will take you a long way What are Wait Events 5

6. V$EVENT_NAME o List of all events with wait classes o Includes definitions of P1, P2 and P3 V$SESSION o Real-time view into what each session/SQL is waiting for o STATE column tells you if the SQL is waiting or has waited WAITING – currently waiting on wait in EVENT column WAITED…TIME - meaning past tense, the session is no longer waiting - now on CPU o Example query: Useful Views for Wait Events SELECT s.sql_id, sql.sql_text, s.sid, s.username, s.machine, s.program, s.module, s.action, s.last_call_et, s.blocking_session, DECODE (s.state, 'WAITING', s.event, 'ON CPU') event, s.p1, s.p2, s.p3, s.wait_class FROM v$session s LEFT OUTER JOIN v$sql sql ON s.sql_id = sql.sql_id AND s.sql_address = sql.address WHERE s.status = 'ACTIVE‘ AND s.type <> 'BACKGROUND‘; 6

7. V$ACTIVE_SESSION_HISTORY o Retains data from V$SESSION by polling once per second o Active (in name) means: on CPU or waiting for non-idle wait event o Resides in memory & size is lesser of # CPU Cores * 2 5% of shared pool o Based on size and busyness, may not contain needed data o Can help answer the questions like: “What happened in the database between 8:12 and 8:23 am this morning when problems were occurring?” o Example query next slide Another Useful View 7

8. Details of what happened between 12:07 and 12:18 am V$ACTIVE_SESSION_HISTORY SELECT --s.sql_id, sql.sql_text, s.session_id, s.user_id, s.machine, s.program, s.module, s.action, s.blocking_session, s.event, s.p1, s.p2, s.p3, s.wait_class FROM v$active_session_history s LEFT OUTER JOIN v$sql sql ON s.sql_id = sql.sql_id AND s.sql_child_number = sql.child_number WHERE s.session_type <> 'BACKGROUND' AND s.sample_time BETWEEN TO_DATE(‘03/31/15 12:07', 'mm/dd/yy hh24:mi') AND TO_DATE(‘03/31/15 12:18','mm/dd/yy hh24:mi') 8

9. Historical views into wait information o Populated once an hour with summary data o Data is retained for 8 days for 11g and higher o Can change defaults with DBMS_WORKLOAD_REPOSITORY EXECUTE DBMS_WORKLOAD_REPOSITORY.MODIFY_SNAPSHOT_SETTINGS - (interval =>60, retention =>20160); (calculated in minutes) (14 days) o DBA_HIST_WR_CONTROL holds current settings DBA_HIST_SNAPSHOT - snapshot times DBA_HIST_ACTIVE_SESS_HISTORY – hourly summary DBA_HIST_SYS_TIME_MODEL – CPU time information DBA_HIST_SQLTEXT – SQL text for most SQLs DBA_HIST_SQLSTAT – SQL stats for most SQLs o Top X Queries during the hour only DBA_HIST Views 9

10. Historical Detailed Wait Data o Provides a historical view into what has happened o Use a BeginTime & EndTime that match problematic timeframe SELECT ash.sample_time, ash.session_id, u.username, ash.program, DECODE(ash.session_state,'WAITING', ash.event, 'ON CPU') event, p1, p2, p3, sql.sql_id, sql.sql_text FROM dba_hist_active_sess_history ash, v$sql sql, dba_users u WHERE ash.sql_id = sql.sql_id AND ash.user_id = u.user_id AND ash.session_type <> 'BACKGROUND‘ AND ash.sample_time BETWEEN TO_DATE('&BeginTime','mm/dd/yy hh24:mi') AND TO_DATE('&EndTime','mm/dd/yy hh24:mi') DBA_HIST Views 10

11. Historical Summary Wait Data o Provides a ranked view of SQL statements o Wait events with the most associated wait time o Use a BeginTime & EndTime that match the problematic timeframe SELECT sql_id, event, SUM(time_waited) seconds_waited FROM ( SELECT sql_id, DECODE(session_state,'WAITING', event, 'ON CPU') event, time_waited FROM dba_hist_active_sess_history WHERE session_type <> 'BACKGROUND' AND sample_time BETWEEN TO_DATE('&BeginTime','mm/dd/yy hh24:mi') AND TO_DATE('&EndTime','mm/dd/yy hh24:mi')) GROUP BY sql_id, event ORDER BY 3 desc; DBA_HIST Views 11

12. Wait Time & Events 12 1 hr of time in database. 78% is spent on db file sequential read Over half of the time is spent on buffer waits Top SQL Statements | SPRD2_PROD2 | January 20, 2015 8:00AM to 9 :00AM

13. Response Time Tuning Data 13 Top SQL Statements | KPSPROD| February 18, 2014 Top Wait Time for SQL Statement Update Quantity February 18, 2014

14. Response Time Benefits 14 Before After Executions for SQL Statement Update Quantity February 18, 2014 Total Wait Time for SQL Statement Update Quantity February 18, 2014 Total Wait Time for SQL Statement Update Quantity February 28, 2014 Daily Time Range: 12:00AM - 11:00AM Executions for SQL Statement Update Quantity February 28, 2014 Daily Time Range: 12:00AM - 11:00AM

15. IDENTIFY END-TO-END TIME 15 Accurate End-to-End Response Time Analysis

16. From my experience: o There is a small list of wait events you need to know well The other 1500+ you can Google or ask Oracle Need to know: o Causes of these waits o How to reduce / fix these waits Let’s review common wait events that I’ve personally seen the most o While working with hundreds of customers Top 10 List of Wait Events 16

17. 1.db file sequential read 2.db file scattered read 3.CPU 4.log file sync 5.read by other session 6.db file parallel read 7.direct path read / write 8.direct path read / write temp 9.enq: TX - row lock contention 10.cursor: pin S wait on X My Top 10 Wait Events 17

18. Description o Occurs when session needs to read a block of data, and can’t find it in buffer cache If not in memory, the data will be read from disk into the buffer cache o Session waits on this event while reading the data in single block reads o Single block reads are most often associated with reading index blocks except when it is a fast full scan also occurs when Oracle has to go back to the table to get required data Parameters o P1 – file# - used to join to dba_data_files table on file_id o P2 – block# - used to join to dba_extents using block_id (P2) & blocks (P3) o P3 – blocks – number of blocks requested by the read Getting File and Object Details - run the “file & object” query: SELECT df.tablespace_name, df.file_name, e.segment_name, e.segment_type, e.partition_name FROM dba_data_files df, dba_extents e WHERE df.file_id = e.file_id AND df.file_id = &file AND &block BETWEEN e.block_id AND e.block_id + e.blocks - 1 1. db file sequential read 18

19. Most solutions will fall into 2 categories o Make the SQL statement perform less I/O - e.g. tune it o Increase the performance of the underlying disks If indexes are being returned by the file & object query o Ensure the index being used is the most efficient one for the table o Consider adding another column to the index from the WHERE clause If tables are being returned by the file & object query o Review the joins closely Ensure indexes are used on the join columns If no indexes, costly trips back to table are needed to join the data. o Inspect the columns in the SELECT clause If not indexed, a visit to the table will occur Review adding that column to an existing index to reduce I/O Covering Index Typical Solutions – db file sequential read 19

20. If insert statement has this wait & the objects are indexes o Inserts have to maintain underlying indexes Look for inefficient/unused indexes & drop them Insure foreign keys are indexed Investigate size of the buffer cache o Keep more blocks in memory to avoid disk reads for the SQL Analyze Oracle File I/O for storage latencies & throughput o Slow disks mean Oracle spends time reading the data into buffer cache o Review the 'DB Single Block Disk Read Time' If the time to read data > 15ms, could indicate slow disks Review % of this wait event to % of total wait for query / application Solutions cont. – db file sequential read 20

21. Description o Occurs when session needs to read a block of data not in the buffer cache If not in memory, the data will be read from disk into the buffer cache o Sessions wait on this event while reading data from disk f or multi-block reads o Multi-block reads are most often associated with full table scans and index fast full scans A fast full scan is similar to a full table scan except the index can be used to satisfy the query Parameters o P1 – file# - used to join to dba_data_files table on file_id o P2 – block# - used to join to dba_extents using block_id (P2) & blocks (P3) o P3 – blocks – number of blocks requested by the read Get information about what being read, use “file & object” query o See Slide #17 for actual 2. db file scattered read 21

22. If full table scan or index fast full scan o Review adding a proper index based on the where clause and/or join criteria o Avoid indexing small tables where a full scan may be more efficient o Make sure the columns in the index have good selectivity If proper indexes exist but are ignored, review the following items: o Do the columns in the WHERE clause contain functions around them The optimizer won’t be able to use the index Consider adding a function-based index if code can’t be changed o Equality criteria? Using “=” or a non-equality criteria with a <>, !=, NOT, etc… Non-equality criteria will most likely not use existing indexes o Does the criteria in WHERE clause match leading edge (1st column) of index? If not, a skip scan could be used, - better than no index but not as efficient Try to create index with a leading edge matching the criteria in the SQL statement o Does the data type passed (via bind variable) match data type of the column? Most times Oracle will convert literal values to proper data types automatically If bind values are used, the calling program could define them incorrectly Example – next slide Typical Solutions – db file scattered read 22

23. SELECT company, attribute FROM data_out WHERE segment = :B1 Wait Time – 100% on “db file scattered read” Plan from EXPLAIN PLAN Plan from V$SQL_PLAN using DBMS_XPLAN Data Type Issues - db file scattered read 23

24. If proper indexes exist but are ignored – cont. o Is the index invisible? If so, the index won’t be used Unless OPTIMIZER_USE_INVISIBLE_INDEXES parameter is true Check the VISIBILITY column in DBA_INDEXES for this. o Are optimizer statistics up to date and correct for the table / columns? Check the LAST_ANALYZED column in the DBA_INDEXES & DBA_TABLES Review SAMPLE_SIZE column in tables to ensure proper number of rows sampled Sometimes a full scan is necessary o Due to the amount of data returned by the query If lots of data needs to be read, reduce wait times by: o If the query is summarizing data from a detailed table: Consider creating a materialized view Note: to get accurate data, adjust the frequency of refreshes o If many sessions running same SQL statement & data hardly changes Review if a RESULTS CACHE can be used Can be turned on with a hint or database parameter Consider caching at the application layer More Solutions – db file scattered read 24

25. Ensure the underlying files are performing well Queries below provide latency information o MS_PER_READ column per file. o Respectable latencies for single block reads < 15 ms Respectable latencies for multi block reads < 30 ms Probably better (lower) for high end production database environments o Higher latencies may signify a potential storage issue o Keep in mind - many layers between a database & storage (O/S, HBA, network) Increasing the size of the buffer cache could be beneficial o Larger buffer cache will keep more blocks in memory & avoid disk reads More Solutions – db file scattered read -- Since instance startup select fs.file#, name, singleblkrds, singleblkrdtim, singleblkrdtim / singleblkrds * 10 ms_per_read from v$filestat fs, v$datafile f where fs.file# = f.file# and singleblkrds > 0 -- For the problematic timeframe / Note that this table has hourly data. select file#, filename, total_reads, total_read_time, total_read_time / total_reads * 10 ms_per_read from ( select file#, filename, sum(singleblkrds) total_reads, sum(singleblkrdtim) total_read_time from dba_hist_filestatxs fs, dba_hist_snapshot s where s.snap_id = fs.snap_id and s.begin_interval_time >= to_date('08/20/14 16:00','mm/dd/yy hh24:mi') and s.end_interval_time <= to_date('08/20/14 20:00','mm/dd/yy hh24:mi') group by file#,filename) 25

26. Not a Wait Event - identifies time on CPU Query Response Time o = Wait Time (wait events) + Service Time (CPU) Waiting on CPU when v$session has: o state = to anything else but ‘waiting’ CPU time is often spent doing logical I/O (i.e. buffer gets) Could also come from compiles/recompiles Other CPU intensive operations on server 3. CPU SELECT sid, serial#, username, program, module, action, machine, osuser, sql_id, blocking_session, decode(state, ’WAITING’, event, ’CPU’) event, p1, p1text, p2, p2text, p3, p3text, SYSDATE FROM V$SESSION s WHERE s.status = ’ACTIVE’ AND wait_class != ’Idle’ AND username != USER; 26

27. Tune queries with high amount of CPU time o Look for high ‘buffer gets’ as CPU is consumed with reading memory Look for Full Table Scans - consider adding an index o Review Index Reads - may be reading more index blocks than necessary Does it make sense to have fewer columns in index if it still needs read a table block anyway It may make sense to put more columns in the index if it will prevent the need to hit any table blocks Review & rebuild fragmented indexes (fill factor, etc…) o Can rows be ordered in the table to get fewer table blocks All similar rows would be in the same or fewer blocks. Reduce high execution count queries o Do more batch processing Check O/S for other activity putting pressure on CPU o Review CPU queue lengths o Is the hardware undersized May need to purchase larger/faster servers Typical Solutions - CPU 27

28. Description o This wait event is triggered when a user session issues a commit or rollback o Session will signal or post the LGWR to write the log buffer to the redo log file When the LGWR has finished writing, it will post back to the user session o Sometimes called ‘commit latency’ Parameters o P1 – Buffer # All changes up to this buffer number (in the log buffer) must be flushed to disk and the writes confirmed to ensure that the transaction is committed and will be kept on an instance crash. The wait is for LGWR to flush up to this ‘buffer#’ o P2 – Sync SCN Wait can be broken down to the following components: 1. Wakeup LGWR if idle 2. LGWR gathers the redo to be written and issues the I/O 3. Wait time for the log write I/O to complete 4. LGWR I/O post processing 5. LGWR posting the foreground/user session that the write has completed 6. Foreground/user session wakeup 4. log file sync 28

29. If the average wait time is low, but the number of waits is high o Check if application is committing after every row Commit less often waiting after "n" rows (batch them) o Each commit is confirmed to make sure the REDO is on disk If the SQL statement is a SELECT statement o Review Oracle Auditing settings Check for Oracle related bugs o If Auditing is enabled for SELECT statements Time is spent writing and committing data to the AUDIT$ table Improved with Unified Auditing in 12C (sys.unified_audit_trail) Reduce other I/O activity on the disks containing the redo logs o Try to use dedicated disks Alternate redo logs on different disks o To maximize the effectiveness of the LGWR More Solutions in breaking down components / tuning LGWR o http://logicalread.solarwinds.com/oracle-log-file-sync-wait-event-dr01 http://logicalread.solarwinds.com/oracle-log-file-sync-wait-event-dr01 Typical Solutions - log file sync 29

30. Description: o When information is needed, Oracle first reads the data from disk to buffer cache o If two or more sessions need the same data, the 1st one reads it into memory while others wait Previous versions (< 10.1) lumped this wait time into the ‘buffer busy waits’ event Excessive waits for this event are typically due to several processes repeatedly reading the same blocks Many sessions scanning the same index or performing full table scans on the same table Parameters o P1 – file# - used to join to dba_data_files table on file_id o P2 – block# - used to join to dba_extents using block_id o P3 – class# – type of block i.e. data block, segment header, undo header,undo block Find the contention 5. read by other session SELECT p1 "file#", p2 "block#", p3 "class#" FROM v$session_wait WHERE event = 'read by other session'; SELECT relative_fno, owner, segment_name, segment_type FROM dba_extents WHERE file_id = &file AND &block BETWEEN block_id AND block_id + blocks - 1; 30

31. Eliminate read contention o Tune or eliminate redundant queries o Redistribute data from frequently read blocks o Reduce block size so the probability of the contention is reduced o Optimize Indexes: Low cardinality indexes causes excessive number of blocks to be read in buffer cache Causes premature aging of good blocks. References: http://logicalread.solarwinds.com/oracle-read-by-other-session-wait-event/#.VRHmMY7F98Ehttp://logicalread.solarwinds.com/oracle-read-by-other-session-wait-event/#.VRHmMY7F98E Typical Solutions - read by other session SELECT data_object_id FROM dba_objects WHERE owner='&owner' AND object_name='&object'; --rowid for the first row in the block SELECT dbms_rowid.rowid_create(1,,,,0) start_rowid FROM dual; --rowid for the 500th row in the block SELECT dbms_rowid.rowid_create(1,,,,500) end_rowid FROM dual; SELECT FROM. WHERE rowid BETWEEN AND 31

32. HOT BLOCKS - read by other session 32

33. Description: o Process issues multiple I/O requests in parallel to read blocks from data files into memory coordinator process waits on this even for all requests to complete o Occurs when session must read multiple blocks which are not adjacent Data is in different random files and locations Session cannot continue processing with the result of a single block i.e. db file sequential read o This wait event also occurs during recovery o This wait event DOES NOT apply to parallel query or parallel DML. Parameters o P1 – files - indicates the number of files the session is reading o P2 – blocks – indicates the total number of blocks to be read o P3 – requests indicates the total number of I/O requests, same as blocks 6. db file parallel read 33

34. Tune SQL to reduce I/O o Tuning SQL can produce large gains in performance Try to get the least amount of reads for the result set returned o Changing the RDBMS itself – Uh not so much! Identify & resolve any SQL using unselective index scan o Review execution plans to see if index scans are high cost with low cardinality o If composite index – make sure most selective column left leading o Increase buffer cache size with DB_BLOCK_BUFFERS if extra memory This should reduce the cost of the I/O More likely to find data in memory - however, it won't reduce the amount of I/O Consider using the operating system's data cache if available o For tables that are frequently accessed via index scans Placing their data files on buffered file systems can reduce the I/O to actual drives Consider using SSDs or Flash technology Evaluate Data Clustering so related data is sorted on disk Look at table partitioning – can it reduce the amount of data read? References: https://fritshoogland.wordpress.com/2013/07/07/the-oracle-db-file-parallel-read-wait-event/ https://fritshoogland.wordpress.com/2013/07/07/the-oracle-db-file-parallel-read-wait-event/ Typical Solutions - db file parallel read 34

35. Description: o Direct path reads (or writes) multiple blocks directly to the Process Global Area (PGA) memory o Bypasses the buffer cache in the Shared Global Area (SGA) o Sometimes related to sorting operations Check to see if occurring primarily in temp tablespaces o If asynchronous I/O is supported (and in use) then Oracle can submit I/O requests and continue processing Picks up results of the I/O requests later Wait on "direct path read" until the required I/O completes. Parameters o P1 – file# If P1 is not in DBA_DATA_FILES or V$DATAFILE and it is greater than the DB_FILES INIT.ORA parameter, then it is a TEMPFILE o P2 – Starting block# o P3 – Number of blocks 7. direct path read / write 35

36. Decision to read via direct path or through cache based on: o Size of the table, buffer cache size and various other stats o tanelpoder-direct_read_decision_statistics_driven/ tanelpoder-direct_read_decision_statistics_driven/ Reading from temp tablespaces o If the database parameter workarea_size_policy = AUTO Increase the database parameter 'pga_aggregate_target‘ o Otherwise: Increase the database parameter 'sort_area_size'. o Another possible solution is to cache the temp datafile At the O/S or Storage system level Reading application data o Ensure database parameter, DISK_ASYNC_IO is TRUE Otherwise direct path read is synchronous (= physical reads) o Ensure no disks are I/O bound. o Ensure the OS asynchronous I/O is configured correctly. Solutions - direct path read 36

37. Is it necessary? Look for: o Insert using the APPEND hint o Data Pump export with ‘DIRECT=y’ o SQL*Loader Direct Path Loads with ‘DIRECT=true’ If the file indicates a temporary tablespace o Check for unexpected disk sort operations Review the OS asynchronous I/O o Ensure database parameter, DISK_ASYNC_IO is TRUE o Is it configured correctly Ensure no disks are I/O bound Review the SQL statement to see if I/O can be reduced Solutions - direct path write 37

38. Description: o Occurs when a session reads/writes buffers from disk directly into the PGA o This event is closely related to the direct path read/write wait o If the I/O subsystem doesn’t support asynchronous I/Os each wait corresponds to a physical read request o This event is usually caused by a sort operation that cannot be completed in memory Requires storage access Also commonly seen during parallel query execution o Similarly, Direct Path Write Temp is an access path Which multiple Oracle blocks are written directly to temporary files by the shadow Oracle process Parameters o P1 – File# o P2 – Starting Block# o P3 – Number of Blocks 8. direct path read / write temp 38

39. Look for high disk sorts o If sorts are too large to fit in memory, they are sent to disk If parallel slaves are used o They may be scanning data o Parallel DML may be used to create and populate objects Can cause direct path read wait and direct path write wait respectively Direct path loads o Direct path API passes data to the load engine in the server Can cause related direct path write wait Is server process ahead of I/O o Can indicate an overloaded I/O system Data Warehouse environment? o Sorts in a data warehouse environment may always go to disk o If High waits on direct path read temp and/or direct path write temp For query plans that call for a hash join Excessive I/O could result from having HASH_AREA_SIZE too small Solutions - direct path read / write temp 39

40. To check for high disk sorts SELECT a.instance_number, TO_CHAR(a.begin_interval_time,'dd/mon/yyyy hh24:mi') meas_date, b.value FROM dba_hist_snapshot a, dba_hist_sysstat b, v$statname c WHERE c.name='sorts (disk)' AND a.begin_interval_time>SYSDATE-7 AND c.name=b.stat_name AND b.snap_id=a.snap_id ORDER BY a.instance_number,a.begin_interval_time ; If high disk sorts, use pga_aggregate_target o Total work area cannot exceed 200 MB RAM o Due to default setting of hidden parameter ‘_pga_max_size’ o No RAM sort may use more than 5% of pga_aggregate_target Or ‘_pga_max_size’, whichever is smaller o No task may use more than 200 MB for sorting or hash joins o Note: Oracle doesn’t recommend changing hidden parameters However, performance can be improved in certain environments Solutions - direct path read / write temp 40

41. Find the cause of the sorts o Find the session experiencing the waits o Review the SQL statement currently being run Query V$TEMPSEG_USAGE o To find the SQL statement generating the sort Query V$SESSTAT for the session to determine the size of the sort Is it possible to reduce the sorting by tuning the SQL statement? If tables are defined with high degree of parallelism, the optimizer leans towards full table scans with parallel processes o Locate the object being loaded o Consider using disk striping Or Automatic Storage Management(ASM) which can stripe for you Check the I/O distribution across the disks Make sure I/O is configure for the parallelism being used Verify that the parameter DISK_ASYNCH_IO is set to true Solutions - direct path read / write temp 41

42. Description: o This occurs when a session is waiting on a row level lock held by another session Resulting in a blocked query o Occurs when attempting to access a row that is locked in exclusive mode o TX lock is acquired exclusive (X) when a transaction initiates its first change Lock is held until the transaction does a COMMIT or ROLLBACK o Ensures a read consistent image of the block before the transaction started It is used mainly as a queuing mechanism so that other sessions can wait for the transaction to complete Parameters o P1 – Name | mode o P2 – USN<<16 | slot – maps to xidusn & xidslot in v$transaction o P3 – Sequence – maps to xidsqn column in v$transaction 9. enq: tx - row lock contention 42

43. Find waits due to row locked by an active transaction SELECT substr(event,0,20) lock_name, ash.session_id waiter, mod(ash.p1,16) lmode, o.object_name object, o.object_type otype, current_file# filen, current_block# blockn, ash.sql_id waiting_sql, blocking_session blocker --,ash.xid, ash.p2, ash.p3 FROM v$active_session_history ash, all_objects o WHERE event LIKE 'enq: TX - row lock contention' AND o.object_id (+)= ash.CURRENT_OBJ#; Solutions - enq: tx - row lock contention 43

44. Check for any uncommited transactions SELECT t.start_time, s.sid, s.serial#, s.username, s.status,s.schemaname, s.osuser, s.process, s.machine, s.terminal, s.program, s.module, to_char(s.logon_time,'DD/MON/YY HH24:MI:SS') logon_time FROM v$transaction t, v$session s WHERE s.saddr = t.ses_addr ORDER BY start_time; Check to see if certain jobs can be run at other times Investigate application code problems such as: o Commits occurring later than necessary o Unnecessary repetitive updates Solutions - enq: tx - row lock contention 44

45. Are waits due to constraint enforcement o Make sure foreign keys are indexed o If primary / unique keys taking too long update Is it possible to drop the constraint & check constraints later (perhaps via a batch job) Review indexes on underlying table o Can they be dropped then recreated after DML Check if rows covered by the same BITMAP index fragment o Mode=4 Rebuild index if fragmented Look into compressing index o Fewer index leaf blocks need read / more rows fit into fewer blocks o Insert statements are faster since fewer leaf blocks o Space savings Reference: http://bangela.org/how-to-resolve-a-enq-tx-row-lock-contention-event/http://bangela.org/how-to-resolve-a-enq-tx-row-lock-contention-event/ Solutions - enq: tx - row lock contention 45

46. Description: o Sessions wait when requesting a mutex for sharable operations related to pins Such as executing a cursor o The mutex can’t be granted because it’s held exclusively by another session Which is most likely parsing the cursor o This is a replacement for ‘library cache pin waits’ in prior versions of Oracle Parameters o P1 – IDN – Same definition as v$mutex_sleep_history.mutex_identifier o P2 – Value – top 2 bytes = SID holding mutex in exclusive mode o P3 – Where – An internal code locator OR’d with Mutex sleeps SELECT p2raw, to_number(substr(to_char(rawtohex(p2raw)), 1, 8), 'XXXXXXXX') sid, COUNT(1) sessions_waiting FROM v$session WHERE event = 'cursor: pin S wait on X' GROUP BY p2raw, to_number(substr(to_char(rawtohex(p2raw)), 1, 8), 'XXXXXXXX'); 10. cursor: pin s wait on x 46

47. The three main causes to sessions waiting on this event o High number of hard parses o A high number of versions of the SQL statement o Bugs Unfortunately, there are a number of bugs related to this wait event Most are fixed in 11.2.0.4 or 12.1.0.1 Consider upgrading to one of the more recent Oracle versions o It could be a result of Oracle bug 5184776 in 11g References: o http://www.pythian.com/blog/cursor-pin-s-wait-on-x-in-the-top-5-wait-events/ http://www.pythian.com/blog/cursor-pin-s-wait-on-x-in-the-top-5-wait-events/ o http://blog.tanelpoder.com/2010/04/21/cursor-pin-s-waits-sporadic-cpu-spikes- and-systematic-troubleshooting/ http://blog.tanelpoder.com/2010/04/21/cursor-pin-s-waits-sporadic-cpu-spikes- and-systematic-troubleshooting/ Solutions for mutex wait problems are very similar o See Troubleshooting of Waits on Mutexes for more informationTroubleshooting of Waits on Mutexes Solutions - cursor: pin S wait on X 47

48. Oracle has instrumented itself with WAIT EVENTS o Shows resources being used up or waited on for each query o Great clues on how to tune Thousands of WAIT EVENTS o Only need to know a small percentage of them Use ASH and Historical DBA tables to troubleshoot o Need tuning & diagnostic license o Consider using DPA for continuous monitoring Top 10 WAIT EVENTS are mostly for IO intensive queries o Learn the most common ones – google the rest SUMMARY 48 Q & A

49. Resolve Performance Issues Quickly Try Database Performance Analyzer FREE for 14 days Improve root cause of slow performance o Quickly identify root cause of issues that impact end-user response time o See historical trends over days, months, and years o Understand impact of VMware ® performance o Agentless architecture with no dependence on Oracle Packs, installs in minutes Free trial at: http://www.solarwinds.com/database-performance-analyzer Free Current View: http://www.solarwinds.com/database_free_edition

50. All SolarWinds Products 50

51. 51