Presentation is loading. Please wait.

Presentation is loading. Please wait.

PeopleSoft Instrumentation for Oracle RDBMS David Kurtz Go-Faster Consultancy Ltd.

Similar presentations


Presentation on theme: "PeopleSoft Instrumentation for Oracle RDBMS David Kurtz Go-Faster Consultancy Ltd."— Presentation transcript:

1 PeopleSoft Instrumentation for Oracle RDBMS David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk

2 Who Am I? Oracle Database Specialist –Independent consultant Performance tuning –PeopleSoft ERP –Oracle RDBMS Book –www.psftdba.comwww.psftdba.com OakTable UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 2

3 Instrumentation Data! Data! Data! I can't make bricks without clay. The Adventures of Sherlock Holmes, Arthur Conan-Doyle UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 3

4 Instrumentation PeopleTools –SQL Trace –Batch Timings –Performance Monitor Oracle –Dynamic Performance Views (V$) –SQL Trace –Automatic Workload Repository (AWR) –Active Session History (ASH) UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 4

5 5 ASH: A Brief Overview Samples active sessions every second Circular buffer in memory –v$active_Session_history –It should hold about 1 hour of data 1 in 10 samples stored in database –DBA_HIST_ACTIVE_SESS_HISTORY –Flushed out during AWR snapshot UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

6 ©2013 www.go-faster.co.uk 6 Background Reading Sifting through the ASHes of (DB) TIme, Graham Wood http://www.oracle.com/technetwork/database/manageability/ppt-active-session-history- 129612.pdfhttp://www.oracle.com/technetwork/database/manageability/ppt-active-session-history- 129612.pdf –Video of presentation at MOW2010 http://www.oaktable.net/media/mow2010-graham-wood-ashes-time-part1 http://www.oaktable.net/media/mow2010-graham-wood-ashes-time-part-2 Doug Burns’ Oracle Blog –http://oracledoug.com/serendipity/index.php?/plugin/tag/ASHhttp://oracledoug.com/serendipity/index.php?/plugin/tag/ASH Practical ASH –http://www.go-faster.co.uk/ukougpres.htm#Practical_ASH.ppthttp://www.go-faster.co.uk/ukougpres.htm#Practical_ASH.ppt Introduction to DBMS_XPLAN –http://www.go-faster.co.uk/Intro_DBMS_XPLAN.ppthttp://www.go-faster.co.uk/Intro_DBMS_XPLAN.ppt UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

7 ©2013 www.go-faster.co.uk 7 What does ASH retain? Most of the columns are on v$session –Session Session ID and serial, query coordinator –Wait event id, name and parameters –SQL SQL_ID, plan hash, opcode, plan line –Object object, file, block and row numbers –Application module, action … UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

8 ©2013 www.go-faster.co.uk 8 ASH –v- SQL*Trace ASH –Licensed (Diagnostics) –Always there –Real Time –No marginal overhead –Who is blocking me? –Statistical data –SQL ID –Plan if captured by AWR –Estimate of duration Per wait event Per Plan Operation 11g SQL*Trace –Free –Enable, File, Profile –Reactive –Run-time overhead –Being Blocked –Every SQL & event –SQL HASH –Actual execution plan –Exact duration Operations in Plan UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

9 ©2013 www.go-faster.co.uk 9 ASH –v- SQL*Trace ASH can be used to resolve many of your performance issues. Sometimes, you will still need SQL*Trace UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

10 ©2013 www.go-faster.co.uk 10 Application Instrumentation It is essential to be able to match –database sessions –application processes DBMS_APPLICATION_INFO –set_module, set_action –Calls in application Introduced in PeopleTools 8.50-8.52. –Appears in SQL*Trace and ASH UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle

11 CLIENT_INFO –PeopleTools 7.53 –Every application server service calls DBMS_APPLICATION_INFO.SET_CLIENT_INFO V$SESSION.CLIENT_INFO Operator ID, … –Inserts into Audit tables can be done by database trigger. Trigger uses READ_CLIENT_INFO to obtain PeopleSoft OPRID –Not copied to ASH UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 11

12 CLIENT_ID PeopleTools 8.52 Also set to PeopleSoft OPRID. Client_ID is copied to ASH –So you can identify the PS operator UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 12

13 PIA Instrumentation PeopleTools 8.50 Module = Component Action = Page –Or XYZZY for search dialogue UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 13

14 IB Instrumentation PeopleTools 8.50 Module = Service Action = Queue UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 14

15 Application Engine PeopleTools 8.50 Module = ‘PSAE’ Action = Program –This is not good –See http://blog.psftdba.com/2009/03/using- oracle-enterprise-manager-grid.htmlhttp://blog.psftdba.com/2009/03/using- oracle-enterprise-manager-grid.html UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 15

16 Application Engine PeopleTools 8.52 Module = ‘PSAE.’||AE program name||session id –session id is stored in PSPRCSQUE.SESSIONID_NUM Action = Program.Section.Step.Actions UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 16

17 Cobol / SQR No instrumentation DIY with trigger –on Process Scheduler request table –http://www.go- faster.co.uk/scripts.htm#psftapi.sqlhttp://www.go- faster.co.uk/scripts.htm#psftapi.sql UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 17

18 OEM UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 18

19 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 19 ASH Report in OEM You can run ASH reports via EM –This example is for a specific module

20 Enable Trace by Module/Action Like setting a watchpoint in a debugger. See http://blog.psftdba.com/2012/09/oracle-sql- tracing-by-module-action.htmlhttp://blog.psftdba.com/2012/09/oracle-sql- tracing-by-module-action.html Can set SQL trace for a specific component/page. But you can’t predict the value of module for AE so may need to go back to the trigger based solution. UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 20

21 Ability to Profile SQL duration by Process/Process Instance PROGRAM MODULE ASH_SECS -------------------- -------------------------------- ---------- PSQRYSRV PSQRYSRV 633520 oracle oracle 328630 rman backup archivelog 236050 PSAESRV GL_JEDIT 203120 rman backup incr datafile 132840 FSPCCURR FSPCCURR 119350 PSAPPSRV VCHR_EXPRESS 111520... ---------- sum 2982340 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 21

22 Profile of Whole System WITH x AS ( SELECT /*+leading(x h) use_nl(h)*/ CASE WHEN h.module IS NULL THEN REGEXP_SUBSTR(h.program, '[^@]+',1,1) WHEN h.module LIKE 'PSAE.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,2) WHEN h.module LIKE '%.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,1) WHEN h.module LIKE '%@%' THEN REGEXP_SUBSTR(h.module, '[^@]+',1,1) ELSE h.module END AS module, REGEXP_SUBSTR(h.program, '[^.@]+',1,1) program, h.event FROM dba_hist_snapshot x, dba_hist_active_sess_history h WHERE h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND x.end_interval_time >= TO_DATE('20130730', 'YYYYMMDD') AND x.begin_interval_time <= TO_DATE('20130807', 'YYYYMMDD') AND h.sample_time BETWEEN TO_DATE('20130730','YYYYMMDD') AND TO_DATE('20130807','YYYYMMDD') ) SELECT program, module, sum(10) ash_Secs FROM x GROUP BY program, module ORDER BY ash_secs DESC UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 22 ASH Data for period of interest Snapshots for Period of interest Logic to trim module name at @ or.

23 Batch Timings by Exec Time UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 23

24 ASH Profile a Single Process Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS --------- ------------- ---------- -------------------------------- ----- ---------- 51159995 fpj0dmac7rw20 207497280 AR_POSTING.HSS_PYMT.ITEMS.S 2797 410 51159995 2qnfa4afzvj91 416825695 AR_POSTING.HS_PYMNT.ITEMS2.S 2797 320 51159995 1u37a5hzzc2z6 768682620 AR_POSTING.HS_PYMNT.ITEMS.S 2797 270 51159995 grhj2f0dcbd70 2964999827 AR_POSTING.HSS_DSO.BACK_OUT.S 2797 190 51159995 a6nrbg7sbx401 721249526 AR_POSTING.HSS_PYMT.UPD_STAT.S 2797 170 51159995 49yzw3xa50r1t 2665694782 AR_POSTING.HS_USER.INSERTS.S 2797 160 51159995 0wcx4syn7tjk2 1678621177 AR_POSTING.HS_USER.UPDATES.S 2797 150 51159995 44sjax92d6kqa 3752318859 AR_POSTING.HS_DSO.ADJ_BAL.S 2797 130 51159995 cm5xjajkfjvkg 3814124734 AR_POSTING.HSS_USR1.INSERTS.S 2797 130 51159995 djur79tkd6uum 2504040148 AR_POSTING.HS_USER.UPD_ACTV.S 2797 130 51159995 cb1cankq47ky9 362952569 AR_POSTING.HSS_USR1.UPDATES.S 2797 130 51159995 1awa6cr95ug3t 3137593480 AR_POSTING.HS_PYMNT.UPD_STAT.S 2797 110 51159995 03p8ga7kazdg4 2504040148 AR_POSTING.HSS_USER.UPD_ACTV.S 2797 100 51159995 5yrur2nv3nn2p 2663443920 AR_POSTING.HSS_DSO.NO_ACTIV.S 2797 80 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 24

25 How profile a process (i) WITH x AS ( SELECT /*+LEADING(r q x h)*/ h.sql_id, h.sql_plan_hash_value, h.action, SUM(10) ash_secs, 10*count(t.sql_id) awr_secs FROM sysadm.psprcsrqst r, sysadm.psprcsque q, dbA_hist_snapshot x, dba_hist_active_Sess_history h LEFT OUTER JOIN dba_hist_sqltext t ON t.sql_id = h.sql_id WHERE x.END_INTERVAL_TIME >= r.begindttm AND x.begin_interval_time <= r.enddttm AND h.sample_time BETWEEN r.begindttm AND r.enddttm AND h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND q.prcsinstance = r.prcsinstance AND h.module = 'PSAE.'||r.prcsname||'.'||q.sessionidnum AND r.prcsname = 'AR_UPDATE2' AND r.prcsinstance = 51159995 GROUP BY h.sql_id, h.sql_plan_hash_value, h.action) … UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 25 Process Request Table One Row per Process Process Request Queue Table One Row per Process Specific Process Request ASH data for period for which process was running Hint to guarantee sensible join order, and partition elimination Filter ASH data by module

26 How profile a process (ii) … ), y AS ( SELECT ROW_NUMBER() OVER (PARTITION BY x.sql_plan_hash_value order by x.awr_secs desc) as ranking, x.sql_id, x.sql_plan_hash_value, x.action, SUM(x.ash_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_ash_secs, SUM(x.awr_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_awr_secs, COUNT(distinct sql_id) OVER (PARTITION BY x.sql_plan_hash_value) sql_ids FROM x ) Select * from y where y.ranking = 1 AND tot_ash_secs > 30 ORDER BY tot_ash_secs DESC / UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 26 Ranking by SQL Plan Hash Value DB Time for Execution Plan AWR secs 0 if SQL not captured in ASH repository

27 Profile a Single Process Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS --------- ------------- ---------- -------------------------------- ----- ---------- 51159995 fpj0dmac7rw20 207497280 AR_POSTING.HSS_PYMT.ITEMS.S 2797 410 51159995 2qnfa4afzvj91 416825695 AR_POSTING.HS_PYMNT.ITEMS2.S 2797 320 51159995 1u37a5hzzc2z6 768682620 AR_POSTING.HS_PYMNT.ITEMS.S 2797 270 51159995 grhj2f0dcbd70 2964999827 AR_POSTING.HSS_DSO.BACK_OUT.S 2797 190 51159995 a6nrbg7sbx401 721249526 AR_POSTING.HSS_PYMT.UPD_STAT.S 2797 170 51159995 49yzw3xa50r1t 2665694782 AR_POSTING.HS_USER.INSERTS.S 2797 160 51159995 0wcx4syn7tjk2 1678621177 AR_POSTING.HS_USER.UPDATES.S 2797 150 51159995 44sjax92d6kqa 3752318859 AR_POSTING.HS_DSO.ADJ_BAL.S 2797 130 51159995 cm5xjajkfjvkg 3814124734 AR_POSTING.HSS_USR1.INSERTS.S 2797 130 51159995 djur79tkd6uum 2504040148 AR_POSTING.HS_USER.UPD_ACTV.S 2797 130 51159995 cb1cankq47ky9 362952569 AR_POSTING.HSS_USR1.UPDATES.S 2797 130 51159995 1awa6cr95ug3t 3137593480 AR_POSTING.HS_PYMNT.UPD_STAT.S 2797 110 51159995 03p8ga7kazdg4 2504040148 AR_POSTING.HSS_USER.UPD_ACTV.S 2797 100 51159995 5yrur2nv3nn2p 2663443920 AR_POSTING.HSS_DSO.NO_ACTIV.S 2797 80 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 27

28 Now investigate the SQL Extract SQL from AWR repository with the dbms_xplan package –Obtaining and Interpreting Execution Plans using DBMS_XPLANObtaining and Interpreting Execution Plans using DBMS_XPLAN http://www.go- faster.co.uk/ukougpres.htm#Intro_DBMS_XPLANhttp://www.go- faster.co.uk/ukougpres.htm#Intro_DBMS_XPLAN UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 28

29 SELECT * FROM table( dbms_xplan.display_awr( 'duu7n4uzsthxz', 768682620,NULL,'ADVANCED')); SQL_ID duu7n4uzsthxz -------------------- INSERT INTO PS_HS_CLSITM_TMP4 (…) SELECT 51164249, I.BUSINESS_UNIT,… FROM PS_ITEM I,PS_ITEM_ACTIVITY A,PS_POST_REQ_TMP4 R WHERE R.PROCESS_INSTANCE = 51164249 AND R.PROCESS_P_HIST ='Y' -- AND I.BUSINESS_UNIT = R.BUSINESS_UNIT AND I.ITEM_STATUS = 'C' AND I.HIST_STATUS = 'N' -- AND A.BUSINESS_UNIT = I.BUSINESS_UNIT AND A.CUST_ID = I.CUST_ID AND A.ITEM = I.ITEM AND A.ITEM_LINE = I.ITEM_LINE AND A.ENTRY_TYPE = I.ENTRY_TYPE AND A.ENTRY_REASON = I.ENTRY_REASON AND A.ITEM_SEQ_NUM = ( SELECT MIN(ITEM_SEQ_NUM) FROM PS_ITEM_ACTIVITY X WHERE X.BUSINESS_UNIT = I.BUSINESS_UNIT AND X.CUST_ID = I.CUST_ID AND X.ITEM = I.ITEM AND X.ITEM_LINE = I.ITEM_LINE AND X.ENTRY_TYPE = I.ENTRY_TYPE AND X.ENTRY_REASON = I.ENTRY_REASON) AND A.ENTRY_AMT_BASE > 0 UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 29

30 Execution Plan Plan hash value: 768682620 ----------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ----------------------------------------------------------------------------------------------------------- | 0 | INSERT STATEMENT | | | | | 222K(100)| | | 1 | FILTER | | | | | | | | 2 | HASH JOIN | | 23719 | 3613K| | 111K (1)| 00:22:19 | | 3 | TABLE ACCESS BY INDEX ROWID | PS_POST_REQ_TMP4 | 8 | 112 | | 2 (0)| 00:00:01 | | 4 | INDEX RANGE SCAN | PS_POST_REQ_TMP4 | 8 | | | 1 (0)| 00:00:01 | | 5 | HASH JOIN | | 23719 | 3289K| 11M| 111K (1)| 00:22:19 | | 6 | TABLE ACCESS BY INDEX ROWID| PS_ITEM | 118K| 9944K| | 30295 (1)| 00:06:04 | | 7 | INDEX RANGE SCAN | PSKITEM | 280K| | | 812 (1)| 00:00:10 | | 8 | TABLE ACCESS FULL | PS_ITEM_ACTIVITY | 3480K| 185M| | 67704 (1)| 00:13:33 | | 9 | SORT AGGREGATE | | 1 | 42 | | | | | 10 | TABLE ACCESS BY INDEX ROWID | PS_ITEM_ACTIVITY | 1 | 42 | | 5 (0)| 00:00:01 | | 11 | INDEX RANGE SCAN | PS_ITEM_ACTIVITY | 1 | | | 4 (0)| 00:00:01 | ----------------------------------------------------------------------------------------------------------- Query Block Name / Object Alias (identified by operation id): ------------------------------------------------------------- 1 - SEL$1 3 - SEL$1 / R@SEL$1 4 - SEL$1 / R@SEL$1 6 - SEL$1 / I@SEL$1 7 - SEL$1 / I@SEL$1 8 - SEL$1 / A@SEL$1 9 - SEL$2 10 - SEL$2 / X@SEL$2 11 - SEL$2 / X@SEL$2 Outline Data ------------- /*+ BEGIN_OUTLINE_DATA IGNORE_OPTIM_EMBEDDED_HINTS OPTIMIZER_FEATURES_ENABLE('10.2.0.4') OPT_PARAM('_unnest_subquery' 'false') OPT_PARAM('optimizer_dynamic_sampling' 4) OPT_PARAM('_optimizer_cost_based_transformation' 'off') ALL_ROWS OUTLINE_LEAF(@"SEL$2") OUTLINE_LEAF(@"SEL$1") OUTLINE_LEAF(@"INS$1") FULL(@"INS$1" "PS_HS_CLSITM_TMP4"@"INS$1") INDEX_RS_ASC(@"SEL$1" "I"@"SEL$1" ("PS_ITEM"."HIST_STATUS")) FULL(@"SEL$1" "A"@"SEL$1") INDEX_RS_ASC(@"SEL$1" "R"@"SEL$1" ("PS_POST_REQ_TMP4"."PROCESS_INSTANCE" "PS_POST_REQ_TMP4"."BUSINESS_UNIT")) LEADING(@"SEL$1" "I"@"SEL$1" "A"@"SEL$1" "R"@"SEL$1") USE_HASH(@"SEL$1" "A"@"SEL$1") USE_HASH(@"SEL$1" "R"@"SEL$1") SWAP_JOIN_INPUTS(@"SEL$1" "R"@"SEL$1") INDEX_RS_ASC(@"SEL$2" "X"@"SEL$2" ("PS_ITEM_ACTIVITY"."BUSINESS_UNIT" "PS_ITEM_ACTIVITY"."CUST_ID" "PS_ITEM_ACTIVITY"."ITEM" "PS_ITEM_ACTIVITY"."ITEM_LINE" "PS_ITEM_ACTIVITY"."ITEM_SEQ_NUM")) END_OUTLINE_DATA */ Note ----- - dynamic sampling used for this statement UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 30

31 Limitations SQL may not be captured by AWR –SQL inside a loop –With bind variables –Without ReUseStatement (binds become literals) Increase AWR snapshot frequency –15 minutes instead of an hour –Top N SQL threshold UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 31

32 Conclusion Instrumentation permits you to profile DB performance with ASH Proactively address performance issues UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 32

33 QUESTIONS? Nullius in verba UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 33

34 You know my methods, apply them. A Study in Scarlet, Arthur Conan-Doyle UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle ©2013 www.go-faster.co.uk 34

35 PeopleSoft Instrumentation for Oracle RDBMS David Kurtz Go-Faster Consultancy Ltd. david.kurtz@go-faster.co.uk www.go-faster.co.uk


Download ppt "PeopleSoft Instrumentation for Oracle RDBMS David Kurtz Go-Faster Consultancy Ltd."

Similar presentations


Ads by Google