1. PeopleSoft Instrumentation for Oracle RDBMS
UKOUG Apps PeopleSoft Instrumentation for Oracle
06/04/2017
PeopleSoft Instrumentation for Oracle RDBMS
David Kurtz
Go-Faster Consultancy Ltd.
©2013 Go-Faster Consultancy Ltd.

2. UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017
06/04/2017
Who Am I?
Oracle Database Specialist
Independent consultant
Performance tuning
PeopleSoft ERP
Oracle RDBMS
Book
OakTable
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.
©2011 Go-Faster Consultancy Ltd. 2

3. Instrumentation Data! Data! Data! I can't make bricks without clay.
The Adventures of Sherlock Holmes, Arthur Conan-Doyle
Performance Tuning is, despite what many people think, not a black art. It is an experimental science and must be approached and conducted in the same unemotional manner. To do that science properly you have to have data.
And as Sherlock Holmes also said – It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
And that is what we are going to talk about in this presentation. The data that informs us about the performance of our systems.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

4. UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017
Instrumentation
PeopleTools
SQL Trace
Batch Timings
Performance Monitor
Oracle
Dynamic Performance Views (V$)
SQL Trace
Automatic Workload Repository (AWR)
Active Session History (ASH)
Both PeopleSoft and Oracle have lots of instrumentation. Historically they have been separate worlds which have been slightly tricky to join up.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.

5. Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017
ASH: A Brief Overview
Samples active sessions every second
Circular buffer in memory
v$active_Session_history
It should hold about 1 hour of data
1 in 10 samples stored in database
DBA_HIST_ACTIVE_SESS_HISTORY
Flushed out during AWR snapshot
Active Session History (ASH) was introduced in Oracle 10g. It samples the activity of each active database session every second. The data is held in a buffer in memory in the database. The design goal is to keep about an hour (your mileage will vary). If a session is not active it will not be sampled. The in-memory buffer is exposed via a view called v$active_session_history.
You could sort of simulate some of ASH by taking a snapshot of v$session every session, but the overhead would be prohibitive. ASH is built into the Oracle kernel, so its overhead is minimal.
When an AWR snapshot is taken, 1 row in 10 from the ASH buffer is copied down into the AWR repository. It can also be flushed to disk between snapshots when the buffer reaches 66% full, so there is no missed data.The data is stored in WRH$_ACTIVE_SESSION_HISTORY and it is exposed via DBA_HIST_ACTIVE_SESS_HISTORY.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2011 Go-Faster Consultancy Ltd.

6. Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017
Background Reading
Sifting through the ASHes of (DB) TIme, Graham Wood
Video of presentation at MOW2010
Doug Burns’ Oracle Blog
Practical ASH
Introduction to DBMS_XPLAN
Sifting through the ASHes, Graham Wood, Oracle (
The ASHes of (DB) Time, Graham Wood at UKOUG2009 (and other conferences).
Introduction to DBMS_XPLAN ( UKOUG2008
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2011 Go-Faster Consultancy Ltd.

7. Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017
What does ASH retain?
Most of the columns are on v$session
Session
Session ID and serial, query coordinator
Wait
event id, name and parameters
SQL
SQL_ID, plan hash, opcode, plan line
Object
object, file, block and row numbers
Application
module, action …
Most of the columns on v$active_session_history are taken directly from column of the same name on v$session, some have different name, and there is some additional information that is not available elsewhere.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2011 Go-Faster Consultancy Ltd.

8. Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017
ASH –v- SQL*Trace
ASH
Licensed (Diagnostics)
Always there
Real Time
No marginal overhead
Who is blocking me?
Statistical data
SQL ID
Plan if captured by AWR
Estimate of duration
Per wait event
Per Plan Operation 11g
SQL*Trace
Free
Enable, File, Profile
Reactive
Run-time overhead
Being Blocked
Every SQL & event
SQL HASH
Actual execution plan
Exact duration
Operations in Plan
ASH and SQL*Trace are not the same thing, but both are valuable tools for finding out about where processes spend time.
SQL*Trace (or event as we used to call it) has been my weapon of choice for solving performance issues for a very long time, and it is extremely effective, and there is still a place for it.
There are difficulties with using SQL trace, especially in a production environment.
Firstly, it does have a run time overhead. You could afford to trace a single process, but you certainly couldn’t trace the entire database.
You have to work with trace in a reactive way. You will probably not already be tracing a process when you experience a performance problem, so you need to run the process again and reproduce the poor performance with trace.
Trace will tell you if a session is blocked waiting on a lock. However, it will not tell you who is blocking you. ASH will do this (although there are limitations).
A trace file records everything that happens in a session, whereas ASH data samples the session every seconds. Short-lived events will be missed, so the data has to be handled statistically (see page 11).
There are problems with both approaches if you have the kind of application where you have lots of different SQL statements because the application uses literal values rather than bind variables (and cursor sharing is EXACT).
Oracle TKPROF trace file profilers cannot aggregate these statements, but I have found another called ORASRP ( that can. With ASH, you will see different SQL_IDs, but it can be effective to group statements with the same execution plan.
You may have trouble finding the SQL text in the SGA (or via the DBMS_XPLAN package) because it has already been aged out of the library cache. You may have similar problems with historical ASH data because the statement had been aged out when the AWR snapshot was taken.
Through the rest of this document you will see SQL_IDs. However, in a SQL trace the statements are identified by hash_value. Those hash values do not show up if you profile your trace file with tkprof, but they do if you use OraSRP. SQL_ID is just a fancy representation of hash value, so you can convert from a SQL_ID to a hash_value. Oracle supply function DBMS_UTILITY.SQLID_TO_SQLHASH(), but as the comment on the blog says Tanel’s script is much cooler[1].
You can’t get the whole of the SQL_ID back from the hash values (because it is trimmed off), but you can get the last 5 or 6 characters it help you find or match SQL statements[2]
[1] See Tanel Poder’s blog:
[2] And I could never have written this without seeing Tanel’s code!
CREATE OR REPLACE FUNCTION h2i (p_hash_value NUMBER) RETURN VARCHAR2 IS
l_output VARCHAR2(10) := '';
BEGIN
FOR i IN (
select substr(' abcdfghjkmnpqrstuvwxyz',1+floor(mod(p_hash_value/(POWER(32,LEVEL-1)),32)),1) sqlidchar
FROM dual CONNECT BY LEVEL <= LN(p_hash_value)/LN(32) ORDER BY LEVEL DESC
) LOOP
l_output := l_output || i.sqlidchar;
END LOOP;
RETURN l_output;
END; /
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2011 Go-Faster Consultancy Ltd.

9. ASH –v- SQL*Trace
ASH can be used to resolve many of your performance issues.
Sometimes, you will still need SQL*Trace
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

10. Application Instrumentation
Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017
Application Instrumentation
It is essential to be able to match
database sessions
application processes
DBMS_APPLICATION_INFO
set_module, set_action
Calls in application
Introduced in PeopleTools
Appears in SQL*Trace and ASH
Application Instrumentation
Oracle has provided a package called DBMS_APPLICATION_INFO since at least Oracle 8. This allows you to set two attributes; MODULE and ACTION for a session. That value then appears in v$session, and can be very useful to help you identify what database sessions relate to what part of an application. These values are also captured by ASH, and without sensible values in these columns you may struggle to identify ASH data for the sessions which are of interest.
These values are not set by default. Instead DBAs are dependent on developers to include them in their code. For example, Oracle E-Business Suite has built this into the application.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2011 Go-Faster Consultancy Ltd.

11. CLIENT_INFO PeopleTools 7.53 Every application server service calls
DBMS_APPLICATION_INFO.SET_CLIENT_INFO
V$SESSION.CLIENT_INFO
Operator ID, …
Inserts into Audit tables can be done by database trigger.
Trigger uses READ_CLIENT_INFO to obtain PeopleSoft OPRID
Not copied to ASH
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

12. CLIENT_ID PeopleTools 8.52 Also set to PeopleSoft OPRID.
Client_ID is copied to ASH
So you can identify the PS operator
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

13. PIA Instrumentation PeopleTools 8.50 Module = Component Action = Page
Or XYZZY for search dialogue
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

14. IB Instrumentation PeopleTools 8.50 Module = Service Action = Queue
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

15. Application Engine PeopleTools 8.50 Module = ‘PSAE’ Action = Program
This is not good
See
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

16. Application Engine PeopleTools 8.52
Module = ‘PSAE.’||AE program name||session id
session id is stored in PSPRCSQUE.SESSIONID_NUM
Action = Program.Section.Step.Actions
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

17. Cobol / SQR No instrumentation DIY with trigger
on Process Scheduler request table
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

18. OEM UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013

19. Row Migration can Aggravate Contention on Cache Buffer Chains Latch
06/04/2017
ASH Report in OEM
You can run ASH reports via EM
This example is for a specific module
Of course, OEM provides a way to run ASH reports, and here you see I have picked a particular time window, and I have specified a module name – in this case the main payroll calculation process.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2011 Go-Faster Consultancy Ltd.

20. Enable Trace by Module/Action
Like setting a watchpoint in a debugger.
See
Can set SQL trace for a specific component/page.
But you can’t predict the value of module for AE so may need to go back to the trigger based solution.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

21. Ability to Profile SQL duration by Process/Process Instance
PROGRAM MODULE ASH_SECS PSQRYSRV PSQRYSRV oracle oracle rman backup archivelog PSAESRV GL_JEDIT rman backup incr datafile FSPCCURR FSPCCURR PSAPPSRV VCHR_EXPRESS sum
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

22. Profile of Whole System
Snapshots for Period of interest
WITH x AS ( SELECT /*+leading(x h) use_nl(h)*/ CASE WHEN h.module IS NULL THEN REGEXP_SUBSTR(h.program, WHEN h.module LIKE 'PSAE.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,2) WHEN h.module LIKE '%.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,1) WHEN h.module LIKE THEN REGEXP_SUBSTR(h.module, ELSE h.module END AS module , REGEXP_SUBSTR(h.program, program , h.event FROM dba_hist_snapshot x , dba_hist_active_sess_history h WHERE h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND x.end_interval_time >= TO_DATE(' ', 'YYYYMMDD') AND x.begin_interval_time <= TO_DATE(' ', 'YYYYMMDD') AND h.sample_time BETWEEN TO_DATE(' ','YYYYMMDD') AND TO_DATE(' ','YYYYMMDD') ) SELECT program, module, sum(10) ash_Secs FROM x GROUP BY program, module ORDER BY ash_secs DESC
ASH Data for period of interest
Logic to trim module name or .
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

23. Batch Timings by Exec Time
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

24. ASH Profile a Single Process
Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS fpj0dmac7rw AR_POSTING.HSS_PYMT.ITEMS.S qnfa4afzvj AR_POSTING.HS_PYMNT.ITEMS2.S u37a5hzzc2z AR_POSTING.HS_PYMNT.ITEMS.S grhj2f0dcbd AR_POSTING.HSS_DSO.BACK_OUT.S a6nrbg7sbx AR_POSTING.HSS_PYMT.UPD_STAT.S yzw3xa50r1t AR_POSTING.HS_USER.INSERTS.S wcx4syn7tjk AR_POSTING.HS_USER.UPDATES.S sjax92d6kqa AR_POSTING.HS_DSO.ADJ_BAL.S cm5xjajkfjvkg AR_POSTING.HSS_USR1.INSERTS.S djur79tkd6uum AR_POSTING.HS_USER.UPD_ACTV.S cb1cankq47ky AR_POSTING.HSS_USR1.UPDATES.S awa6cr95ug3t AR_POSTING.HS_PYMNT.UPD_STAT.S p8ga7kazdg AR_POSTING.HSS_USER.UPD_ACTV.S yrur2nv3nn2p AR_POSTING.HSS_DSO.NO_ACTIV.S
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

25. How profile a process (i)
UKOUG Apps PeopleSoft Instrumentation for Oracle
06/04/2017
How profile a process (i)
Hint to guarantee sensible join order, and partition elimination
WITH x AS ( SELECT /*+LEADING(r q x h)*/ h.sql_id, h.sql_plan_hash_value, h.action , SUM(10) ash_secs , 10*count(t.sql_id) awr_secs FROM sysadm.psprcsrqst r , sysadm.psprcsque q , dbA_hist_snapshot x , dba_hist_active_Sess_history h LEFT OUTER JOIN dba_hist_sqltext t ON t.sql_id = h.sql_id WHERE x.END_INTERVAL_TIME >= r.begindttm AND x.begin_interval_time <= r.enddttm AND h.sample_time BETWEEN r.begindttm AND r.enddttm AND h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND q.prcsinstance = r.prcsinstance AND h.module = 'PSAE.'||r.prcsname||'.'||q.sessionidnum AND r.prcsname = 'AR_UPDATE2' AND r.prcsinstance = GROUP BY h.sql_id, h.sql_plan_hash_value, h.action) …
Process Request Table
One Row per Process
Process Request Queue Table
One Row per Process
ASH data for period for which process was running
Filter ASH data by module
Specific Process Request
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.

26. How profile a process (ii)
UKOUG Apps PeopleSoft Instrumentation for Oracle
06/04/2017
How profile a process (ii)
Ranking by SQL Plan Hash Value
… ), y AS ( SELECT ROW_NUMBER() OVER (PARTITION BY x.sql_plan_hash_value order by x.awr_secs desc) as ranking , x.sql_id, x.sql_plan_hash_value, x.action , SUM(x.ash_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_ash_secs , SUM(x.awr_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_awr_secs , COUNT(distinct sql_id) OVER (PARTITION BY x.sql_plan_hash_value) sql_ids FROM x ) Select * from y where y.ranking = 1 AND tot_ash_secs > 30 ORDER BY tot_ash_secs DESC /
DB Time for Execution Plan
AWR secs 0 if SQL not captured in ASH repository
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.

27. Profile a Single Process
Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS fpj0dmac7rw AR_POSTING.HSS_PYMT.ITEMS.S qnfa4afzvj AR_POSTING.HS_PYMNT.ITEMS2.S u37a5hzzc2z AR_POSTING.HS_PYMNT.ITEMS.S grhj2f0dcbd AR_POSTING.HSS_DSO.BACK_OUT.S a6nrbg7sbx AR_POSTING.HSS_PYMT.UPD_STAT.S yzw3xa50r1t AR_POSTING.HS_USER.INSERTS.S wcx4syn7tjk AR_POSTING.HS_USER.UPDATES.S sjax92d6kqa AR_POSTING.HS_DSO.ADJ_BAL.S cm5xjajkfjvkg AR_POSTING.HSS_USR1.INSERTS.S djur79tkd6uum AR_POSTING.HS_USER.UPD_ACTV.S cb1cankq47ky AR_POSTING.HSS_USR1.UPDATES.S awa6cr95ug3t AR_POSTING.HS_PYMNT.UPD_STAT.S p8ga7kazdg AR_POSTING.HSS_USER.UPD_ACTV.S yrur2nv3nn2p AR_POSTING.HSS_DSO.NO_ACTIV.S
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

28. Now investigate the SQL
UKOUG Apps PeopleSoft Instrumentation for Oracle
06/04/2017
Now investigate the SQL
Extract SQL from AWR repository with the dbms_xplan package
Obtaining and Interpreting Execution Plans using DBMS_XPLAN
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.

29. SELECT. FROM table( dbms_xplan
SELECT * FROM table( dbms_xplan.display_awr( 'duu7n4uzsthxz', ,NULL,'ADVANCED'));
SQL_ID duu7n4uzsthxz INSERT INTO PS_HS_CLSITM_TMP4 (…) SELECT , I.BUSINESS_UNIT ,… FROM PS_ITEM I ,PS_ITEM_ACTIVITY A ,PS_POST_REQ_TMP4 R WHERE R.PROCESS_INSTANCE = AND R.PROCESS_P_HIST ='Y' -- AND I.BUSINESS_UNIT = R.BUSINESS_UNIT AND I.ITEM_STATUS = 'C' AND I.HIST_STATUS = 'N' AND A.BUSINESS_UNIT = I.BUSINESS_UNIT AND A.CUST_ID = I.CUST_ID AND A.ITEM = I.ITEM AND A.ITEM_LINE = I.ITEM_LINE AND A.ENTRY_TYPE = I.ENTRY_TYPE AND A.ENTRY_REASON = I.ENTRY_REASON
AND A.ITEM_SEQ_NUM = (
SELECT MIN(ITEM_SEQ_NUM)
FROM PS_ITEM_ACTIVITY X
WHERE X.BUSINESS_UNIT = I.BUSINESS_UNIT
AND X.CUST_ID = I.CUST_ID
AND X.ITEM = I.ITEM
AND X.ITEM_LINE = I.ITEM_LINE
AND X.ENTRY_TYPE = I.ENTRY_TYPE
AND X.ENTRY_REASON = I.ENTRY_REASON)
AND A.ENTRY_AMT_BASE > 0
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

30. UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017
Execution Plan
Plan hash value: | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | | 0 | INSERT STATEMENT | | | | | 222K(100)| | | 1 | FILTER | | | | | | | | 2 | HASH JOIN | | | 3613K| | 111K (1)| 00:22:19 | | 3 | TABLE ACCESS BY INDEX ROWID | PS_POST_REQ_TMP4 | 8 | 112 | | 2 (0)| 00:00:01 | | 4 | INDEX RANGE SCAN | PS_POST_REQ_TMP4 | 8 | | | 1 (0)| 00:00:01 | | 5 | HASH JOIN | | | 3289K| 11M| 111K (1)| 00:22:19 | | 6 | TABLE ACCESS BY INDEX ROWID| PS_ITEM | 118K| 9944K| | (1)| 00:06:04 | | 7 | INDEX RANGE SCAN | PSKITEM | 280K| | | 812 (1)| 00:00:10 | | 8 | TABLE ACCESS FULL | PS_ITEM_ACTIVITY | 3480K| 185M| | (1)| 00:13:33 | | 9 | SORT AGGREGATE | | 1 | 42 | | | | | 10 | TABLE ACCESS BY INDEX ROWID | PS_ITEM_ACTIVITY | 1 | 42 | | 5 (0)| 00:00:01 | | 11 | INDEX RANGE SCAN | PS_ITEM_ACTIVITY | 1 | | | 4 (0)| 00:00:01 | Query Block Name / Object Alias (identified by operation id): SEL$1 3 - SEL$1 / 4 - SEL$1 / 6 - SEL$1 / 7 - SEL$1 / 8 - SEL$1 / 9 - SEL$ SEL$2 / 11 - SEL$2 / Outline Data /*+ BEGIN_OUTLINE_DATA IGNORE_OPTIM_EMBEDDED_HINTS OPTIMIZER_FEATURES_ENABLE(' ') OPT_PARAM('_unnest_subquery' 'false') OPT_PARAM('optimizer_dynamic_sampling' 4) OPT_PARAM('_optimizer_cost_based_transformation' 'off') ALL_ROWS ("PS_ITEM"."HIST_STATUS")) ("PS_POST_REQ_TMP4"."PROCESS_INSTANCE" "PS_POST_REQ_TMP4"."BUSINESS_UNIT")) ("PS_ITEM_ACTIVITY"."BUSINESS_UNIT" "PS_ITEM_ACTIVITY"."CUST_ID" "PS_ITEM_ACTIVITY"."ITEM" "PS_ITEM_ACTIVITY"."ITEM_LINE" "PS_ITEM_ACTIVITY"."ITEM_SEQ_NUM")) END_OUTLINE_DATA */ Note dynamic sampling used for this statement
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.

31. Limitations SQL may not be captured by AWR
SQL inside a loop
With bind variables
Without ReUseStatement
(binds become literals)
Increase AWR snapshot frequency
15 minutes instead of an hour
Top N SQL threshold
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

32. UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017
Conclusion
Instrumentation permits you to profile DB performance with ASH
Proactively address performance issues
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.

33. UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
06/04/2017
Nullius in verba
Questions?
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013
©2013 Go-Faster Consultancy Ltd.

34. You know my methods, apply them.
A Study in Scarlet, Arthur Conan-Doyle
Last slide - More Sherlock Holmes.
UKOUG Apps PeopleSoft Instrumentation for Oracle
©2013

35. PeopleSoft Instrumentation for Oracle RDBMS
UKOUG Apps PeopleSoft Instrumentation for Oracle
06/04/2017
PeopleSoft Instrumentation for Oracle RDBMS
David Kurtz
Go-Faster Consultancy Ltd.
©2013 Go-Faster Consultancy Ltd.