Presentation is loading. Please wait.

Presentation is loading. Please wait.

PeopleSoft Instrumentation for Oracle RDBMS

Similar presentations


Presentation on theme: "PeopleSoft Instrumentation for Oracle RDBMS"— Presentation transcript:

1 PeopleSoft Instrumentation for Oracle RDBMS
UKOUG Apps PeopleSoft Instrumentation for Oracle 06/04/2017 PeopleSoft Instrumentation for Oracle RDBMS David Kurtz Go-Faster Consultancy Ltd. ©2013 Go-Faster Consultancy Ltd.

2 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
Row Migration can Aggravate Contention on Cache Buffer Chains Latch 06/04/2017 06/04/2017 Who Am I? Oracle Database Specialist Independent consultant Performance tuning PeopleSoft ERP Oracle RDBMS Book OakTable UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd. ©2011 Go-Faster Consultancy Ltd. 2

3 Instrumentation Data! Data! Data! I can't make bricks without clay.
The Adventures of Sherlock Holmes, Arthur Conan-Doyle Performance Tuning is, despite what many people think, not a black art. It is an experimental science and must be approached and conducted in the same unemotional manner. To do that science properly you have to have data. And as Sherlock Holmes also said – It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts. And that is what we are going to talk about in this presentation. The data that informs us about the performance of our systems. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

4 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017 Instrumentation PeopleTools SQL Trace Batch Timings Performance Monitor Oracle Dynamic Performance Views (V$) SQL Trace Automatic Workload Repository (AWR) Active Session History (ASH) Both PeopleSoft and Oracle have lots of instrumentation. Historically they have been separate worlds which have been slightly tricky to join up. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd.

5 Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017 ASH: A Brief Overview Samples active sessions every second Circular buffer in memory v$active_Session_history It should hold about 1 hour of data 1 in 10 samples stored in database DBA_HIST_ACTIVE_SESS_HISTORY Flushed out during AWR snapshot Active Session History (ASH) was introduced in Oracle 10g. It samples the activity of each active database session every second. The data is held in a buffer in memory in the database. The design goal is to keep about an hour (your mileage will vary). If a session is not active it will not be sampled. The in-memory buffer is exposed via a view called v$active_session_history. You could sort of simulate some of ASH by taking a snapshot of v$session every session, but the overhead would be prohibitive. ASH is built into the Oracle kernel, so its overhead is minimal. When an AWR snapshot is taken, 1 row in 10 from the ASH buffer is copied down into the AWR repository. It can also be flushed to disk between snapshots when the buffer reaches 66% full, so there is no missed data.The data is stored in WRH$_ACTIVE_SESSION_HISTORY and it is exposed via DBA_HIST_ACTIVE_SESS_HISTORY. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2011 Go-Faster Consultancy Ltd.

6 Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017 Background Reading Sifting through the ASHes of (DB) TIme, Graham Wood Video of presentation at MOW2010 Doug Burns’ Oracle Blog Practical ASH Introduction to DBMS_XPLAN Sifting through the ASHes, Graham Wood, Oracle (http://www.oracle.com/technology/products/manageability/database/pdf/twp03/PPT_active_session_history.pdf) The ASHes of (DB) Time, Graham Wood at UKOUG2009 (and other conferences). Introduction to DBMS_XPLAN (http://www.go-faster.co.uk/Intro_DBMS_XPLAN.ppt), UKOUG2008 UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2011 Go-Faster Consultancy Ltd.

7 Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017 What does ASH retain? Most of the columns are on v$session Session Session ID and serial, query coordinator Wait event id, name and parameters SQL SQL_ID, plan hash, opcode, plan line Object object, file, block and row numbers Application module, action … Most of the columns on v$active_session_history are taken directly from column of the same name on v$session, some have different name, and there is some additional information that is not available elsewhere. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2011 Go-Faster Consultancy Ltd.

8 Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017 ASH –v- SQL*Trace ASH Licensed (Diagnostics) Always there Real Time No marginal overhead Who is blocking me? Statistical data SQL ID Plan if captured by AWR Estimate of duration Per wait event Per Plan Operation 11g SQL*Trace Free Enable, File, Profile Reactive Run-time overhead Being Blocked Every SQL & event SQL HASH Actual execution plan Exact duration Operations in Plan ASH and SQL*Trace are not the same thing, but both are valuable tools for finding out about where processes spend time. SQL*Trace (or event as we used to call it) has been my weapon of choice for solving performance issues for a very long time, and it is extremely effective, and there is still a place for it. There are difficulties with using SQL trace, especially in a production environment. Firstly, it does have a run time overhead. You could afford to trace a single process, but you certainly couldn’t trace the entire database. You have to work with trace in a reactive way. You will probably not already be tracing a process when you experience a performance problem, so you need to run the process again and reproduce the poor performance with trace. Trace will tell you if a session is blocked waiting on a lock. However, it will not tell you who is blocking you. ASH will do this (although there are limitations). A trace file records everything that happens in a session, whereas ASH data samples the session every seconds. Short-lived events will be missed, so the data has to be handled statistically (see page 11). There are problems with both approaches if you have the kind of application where you have lots of different SQL statements because the application uses literal values rather than bind variables (and cursor sharing is EXACT). Oracle TKPROF trace file profilers cannot aggregate these statements, but I have found another called ORASRP (www.oracledba.ru/orasrp) that can. With ASH, you will see different SQL_IDs, but it can be effective to group statements with the same execution plan. You may have trouble finding the SQL text in the SGA (or via the DBMS_XPLAN package) because it has already been aged out of the library cache. You may have similar problems with historical ASH data because the statement had been aged out when the AWR snapshot was taken. Through the rest of this document you will see SQL_IDs. However, in a SQL trace the statements are identified by hash_value. Those hash values do not show up if you profile your trace file with tkprof, but they do if you use OraSRP. SQL_ID is just a fancy representation of hash value, so you can convert from a SQL_ID to a hash_value. Oracle supply function DBMS_UTILITY.SQLID_TO_SQLHASH(), but as the comment on the blog says Tanel’s script is much cooler[1]. You can’t get the whole of the SQL_ID back from the hash values (because it is trimmed off), but you can get the last 5 or 6 characters it help you find or match SQL statements[2] [1] See Tanel Poder’s blog: [2] And I could never have written this without seeing Tanel’s code! CREATE OR REPLACE FUNCTION h2i (p_hash_value NUMBER) RETURN VARCHAR2 IS l_output VARCHAR2(10) := ''; BEGIN FOR i IN ( select substr(' abcdfghjkmnpqrstuvwxyz',1+floor(mod(p_hash_value/(POWER(32,LEVEL-1)),32)),1) sqlidchar FROM dual CONNECT BY LEVEL <= LN(p_hash_value)/LN(32) ORDER BY LEVEL DESC ) LOOP l_output := l_output || i.sqlidchar; END LOOP; RETURN l_output; END; / UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2011 Go-Faster Consultancy Ltd.

9 ASH –v- SQL*Trace ASH can be used to resolve many of your performance issues. Sometimes, you will still need SQL*Trace UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

10 Application Instrumentation
Row Migration can Aggravate Contention on Cache Buffer Chains Latch 06/04/2017 Application Instrumentation It is essential to be able to match database sessions application processes DBMS_APPLICATION_INFO set_module, set_action Calls in application Introduced in PeopleTools Appears in SQL*Trace and ASH Application Instrumentation Oracle has provided a package called DBMS_APPLICATION_INFO since at least Oracle 8. This allows you to set two attributes; MODULE and ACTION for a session. That value then appears in v$session, and can be very useful to help you identify what database sessions relate to what part of an application. These values are also captured by ASH, and without sensible values in these columns you may struggle to identify ASH data for the sessions which are of interest. These values are not set by default. Instead DBAs are dependent on developers to include them in their code. For example, Oracle E-Business Suite has built this into the application. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2011 Go-Faster Consultancy Ltd.

11 CLIENT_INFO PeopleTools 7.53 Every application server service calls
DBMS_APPLICATION_INFO.SET_CLIENT_INFO V$SESSION.CLIENT_INFO Operator ID, … Inserts into Audit tables can be done by database trigger. Trigger uses READ_CLIENT_INFO to obtain PeopleSoft OPRID Not copied to ASH UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

12 CLIENT_ID PeopleTools 8.52 Also set to PeopleSoft OPRID.
Client_ID is copied to ASH So you can identify the PS operator UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

13 PIA Instrumentation PeopleTools 8.50 Module = Component Action = Page
Or XYZZY for search dialogue UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

14 IB Instrumentation PeopleTools 8.50 Module = Service Action = Queue
UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

15 Application Engine PeopleTools 8.50 Module = ‘PSAE’ Action = Program
This is not good See UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

16 Application Engine PeopleTools 8.52
Module = ‘PSAE.’||AE program name||session id session id is stored in PSPRCSQUE.SESSIONID_NUM Action = Program.Section.Step.Actions UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

17 Cobol / SQR No instrumentation DIY with trigger
on Process Scheduler request table UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

18 OEM UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013

19 Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017 ASH Report in OEM You can run ASH reports via EM This example is for a specific module Of course, OEM provides a way to run ASH reports, and here you see I have picked a particular time window, and I have specified a module name – in this case the main payroll calculation process. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2011 Go-Faster Consultancy Ltd.

20 Enable Trace by Module/Action
Like setting a watchpoint in a debugger. See Can set SQL trace for a specific component/page. But you can’t predict the value of module for AE so may need to go back to the trigger based solution. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

21 Ability to Profile SQL duration by Process/Process Instance
PROGRAM MODULE ASH_SECS PSQRYSRV PSQRYSRV oracle oracle rman backup archivelog PSAESRV GL_JEDIT rman backup incr datafile FSPCCURR FSPCCURR PSAPPSRV VCHR_EXPRESS sum UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

22 Profile of Whole System
Snapshots for Period of interest WITH x AS ( SELECT /*+leading(x h) use_nl(h)*/ CASE WHEN h.module IS NULL THEN REGEXP_SUBSTR(h.program, WHEN h.module LIKE 'PSAE.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,2) WHEN h.module LIKE '%.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,1) WHEN h.module LIKE THEN REGEXP_SUBSTR(h.module, ELSE h.module END AS module , REGEXP_SUBSTR(h.program, program , h.event FROM dba_hist_snapshot x , dba_hist_active_sess_history h WHERE h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND x.end_interval_time >= TO_DATE(' ', 'YYYYMMDD') AND x.begin_interval_time <= TO_DATE(' ', 'YYYYMMDD') AND h.sample_time BETWEEN TO_DATE(' ','YYYYMMDD') AND TO_DATE(' ','YYYYMMDD') ) SELECT program, module, sum(10) ash_Secs FROM x GROUP BY program, module ORDER BY ash_secs DESC ASH Data for period of interest Logic to trim module name or . UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

23 Batch Timings by Exec Time
UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

24 ASH Profile a Single Process
Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS fpj0dmac7rw AR_POSTING.HSS_PYMT.ITEMS.S qnfa4afzvj AR_POSTING.HS_PYMNT.ITEMS2.S u37a5hzzc2z AR_POSTING.HS_PYMNT.ITEMS.S grhj2f0dcbd AR_POSTING.HSS_DSO.BACK_OUT.S a6nrbg7sbx AR_POSTING.HSS_PYMT.UPD_STAT.S yzw3xa50r1t AR_POSTING.HS_USER.INSERTS.S wcx4syn7tjk AR_POSTING.HS_USER.UPDATES.S sjax92d6kqa AR_POSTING.HS_DSO.ADJ_BAL.S cm5xjajkfjvkg AR_POSTING.HSS_USR1.INSERTS.S djur79tkd6uum AR_POSTING.HS_USER.UPD_ACTV.S cb1cankq47ky AR_POSTING.HSS_USR1.UPDATES.S awa6cr95ug3t AR_POSTING.HS_PYMNT.UPD_STAT.S p8ga7kazdg AR_POSTING.HSS_USER.UPD_ACTV.S yrur2nv3nn2p AR_POSTING.HSS_DSO.NO_ACTIV.S UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

25 How profile a process (i)
UKOUG Apps PeopleSoft Instrumentation for Oracle 06/04/2017 How profile a process (i) Hint to guarantee sensible join order, and partition elimination WITH x AS ( SELECT /*+LEADING(r q x h)*/ h.sql_id, h.sql_plan_hash_value, h.action , SUM(10) ash_secs , 10*count(t.sql_id) awr_secs FROM sysadm.psprcsrqst r , sysadm.psprcsque q , dbA_hist_snapshot x , dba_hist_active_Sess_history h LEFT OUTER JOIN dba_hist_sqltext t ON t.sql_id = h.sql_id WHERE x.END_INTERVAL_TIME >= r.begindttm AND x.begin_interval_time <= r.enddttm AND h.sample_time BETWEEN r.begindttm AND r.enddttm AND h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND q.prcsinstance = r.prcsinstance AND h.module = 'PSAE.'||r.prcsname||'.'||q.sessionidnum AND r.prcsname = 'AR_UPDATE2' AND r.prcsinstance = GROUP BY h.sql_id, h.sql_plan_hash_value, h.action) … Process Request Table One Row per Process Process Request Queue Table One Row per Process ASH data for period for which process was running Filter ASH data by module Specific Process Request UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd.

26 How profile a process (ii)
UKOUG Apps PeopleSoft Instrumentation for Oracle 06/04/2017 How profile a process (ii) Ranking by SQL Plan Hash Value … ), y AS ( SELECT ROW_NUMBER() OVER (PARTITION BY x.sql_plan_hash_value order by x.awr_secs desc) as ranking , x.sql_id, x.sql_plan_hash_value, x.action , SUM(x.ash_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_ash_secs , SUM(x.awr_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_awr_secs , COUNT(distinct sql_id) OVER (PARTITION BY x.sql_plan_hash_value) sql_ids FROM x ) Select * from y where y.ranking = 1 AND tot_ash_secs > 30 ORDER BY tot_ash_secs DESC / DB Time for Execution Plan AWR secs 0 if SQL not captured in ASH repository UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd.

27 Profile a Single Process
Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS fpj0dmac7rw AR_POSTING.HSS_PYMT.ITEMS.S qnfa4afzvj AR_POSTING.HS_PYMNT.ITEMS2.S u37a5hzzc2z AR_POSTING.HS_PYMNT.ITEMS.S grhj2f0dcbd AR_POSTING.HSS_DSO.BACK_OUT.S a6nrbg7sbx AR_POSTING.HSS_PYMT.UPD_STAT.S yzw3xa50r1t AR_POSTING.HS_USER.INSERTS.S wcx4syn7tjk AR_POSTING.HS_USER.UPDATES.S sjax92d6kqa AR_POSTING.HS_DSO.ADJ_BAL.S cm5xjajkfjvkg AR_POSTING.HSS_USR1.INSERTS.S djur79tkd6uum AR_POSTING.HS_USER.UPD_ACTV.S cb1cankq47ky AR_POSTING.HSS_USR1.UPDATES.S awa6cr95ug3t AR_POSTING.HS_PYMNT.UPD_STAT.S p8ga7kazdg AR_POSTING.HSS_USER.UPD_ACTV.S yrur2nv3nn2p AR_POSTING.HSS_DSO.NO_ACTIV.S UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

28 Now investigate the SQL
UKOUG Apps PeopleSoft Instrumentation for Oracle 06/04/2017 Now investigate the SQL Extract SQL from AWR repository with the dbms_xplan package Obtaining and Interpreting Execution Plans using DBMS_XPLAN UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd.

29 SELECT. FROM table( dbms_xplan
SELECT * FROM table( dbms_xplan.display_awr( 'duu7n4uzsthxz', ,NULL,'ADVANCED')); SQL_ID duu7n4uzsthxz INSERT INTO PS_HS_CLSITM_TMP4 (…) SELECT , I.BUSINESS_UNIT ,… FROM PS_ITEM I ,PS_ITEM_ACTIVITY A ,PS_POST_REQ_TMP4 R WHERE R.PROCESS_INSTANCE = AND R.PROCESS_P_HIST ='Y' -- AND I.BUSINESS_UNIT = R.BUSINESS_UNIT AND I.ITEM_STATUS = 'C' AND I.HIST_STATUS = 'N' AND A.BUSINESS_UNIT = I.BUSINESS_UNIT AND A.CUST_ID = I.CUST_ID AND A.ITEM = I.ITEM AND A.ITEM_LINE = I.ITEM_LINE AND A.ENTRY_TYPE = I.ENTRY_TYPE AND A.ENTRY_REASON = I.ENTRY_REASON AND A.ITEM_SEQ_NUM = ( SELECT MIN(ITEM_SEQ_NUM) FROM PS_ITEM_ACTIVITY X WHERE X.BUSINESS_UNIT = I.BUSINESS_UNIT AND X.CUST_ID = I.CUST_ID AND X.ITEM = I.ITEM AND X.ITEM_LINE = I.ITEM_LINE AND X.ENTRY_TYPE = I.ENTRY_TYPE AND X.ENTRY_REASON = I.ENTRY_REASON) AND A.ENTRY_AMT_BASE > 0 UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

30 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017 Execution Plan Plan hash value: | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | | 0 | INSERT STATEMENT | | | | | 222K(100)| | | 1 | FILTER | | | | | | | | 2 | HASH JOIN | | | 3613K| | 111K (1)| 00:22:19 | | 3 | TABLE ACCESS BY INDEX ROWID | PS_POST_REQ_TMP4 | 8 | 112 | | 2 (0)| 00:00:01 | | 4 | INDEX RANGE SCAN | PS_POST_REQ_TMP4 | 8 | | | 1 (0)| 00:00:01 | | 5 | HASH JOIN | | | 3289K| 11M| 111K (1)| 00:22:19 | | 6 | TABLE ACCESS BY INDEX ROWID| PS_ITEM | 118K| 9944K| | (1)| 00:06:04 | | 7 | INDEX RANGE SCAN | PSKITEM | 280K| | | 812 (1)| 00:00:10 | | 8 | TABLE ACCESS FULL | PS_ITEM_ACTIVITY | 3480K| 185M| | (1)| 00:13:33 | | 9 | SORT AGGREGATE | | 1 | 42 | | | | | 10 | TABLE ACCESS BY INDEX ROWID | PS_ITEM_ACTIVITY | 1 | 42 | | 5 (0)| 00:00:01 | | 11 | INDEX RANGE SCAN | PS_ITEM_ACTIVITY | 1 | | | 4 (0)| 00:00:01 | Query Block Name / Object Alias (identified by operation id): SEL$1 3 - SEL$1 / 4 - SEL$1 / 6 - SEL$1 / 7 - SEL$1 / 8 - SEL$1 / 9 - SEL$ SEL$2 / 11 - SEL$2 / Outline Data /*+ BEGIN_OUTLINE_DATA IGNORE_OPTIM_EMBEDDED_HINTS OPTIMIZER_FEATURES_ENABLE(' ') OPT_PARAM('_unnest_subquery' 'false') OPT_PARAM('optimizer_dynamic_sampling' 4) OPT_PARAM('_optimizer_cost_based_transformation' 'off') ALL_ROWS ("PS_ITEM"."HIST_STATUS")) ("PS_POST_REQ_TMP4"."PROCESS_INSTANCE" "PS_POST_REQ_TMP4"."BUSINESS_UNIT")) ("PS_ITEM_ACTIVITY"."BUSINESS_UNIT" "PS_ITEM_ACTIVITY"."CUST_ID" "PS_ITEM_ACTIVITY"."ITEM" "PS_ITEM_ACTIVITY"."ITEM_LINE" "PS_ITEM_ACTIVITY"."ITEM_SEQ_NUM")) END_OUTLINE_DATA */ Note dynamic sampling used for this statement UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd.

31 Limitations SQL may not be captured by AWR
SQL inside a loop With bind variables Without ReUseStatement (binds become literals) Increase AWR snapshot frequency 15 minutes instead of an hour Top N SQL threshold UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

32 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017 Conclusion Instrumentation permits you to profile DB performance with ASH Proactively address performance issues UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd.

33 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017 Nullius in verba Questions? UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013 ©2013 Go-Faster Consultancy Ltd.

34 You know my methods, apply them.
A Study in Scarlet, Arthur Conan-Doyle Last slide - More Sherlock Holmes. UKOUG Apps PeopleSoft Instrumentation for Oracle ©2013

35 PeopleSoft Instrumentation for Oracle RDBMS
UKOUG Apps PeopleSoft Instrumentation for Oracle 06/04/2017 PeopleSoft Instrumentation for Oracle RDBMS David Kurtz Go-Faster Consultancy Ltd. ©2013 Go-Faster Consultancy Ltd.


Download ppt "PeopleSoft Instrumentation for Oracle RDBMS"

Similar presentations


Ads by Google