Presentation is loading. Please wait.

Presentation is loading. Please wait.

Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk1 Practical Guidance on the Use of PeopleSoft Performance Monitor David Kurtz Go-Faster Consultancy Ltd.

Similar presentations


Presentation on theme: "Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk1 Practical Guidance on the Use of PeopleSoft Performance Monitor David Kurtz Go-Faster Consultancy Ltd."— Presentation transcript:

1 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk1 Practical Guidance on the Use of PeopleSoft Performance Monitor David Kurtz Go-Faster Consultancy Ltd.

2 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk2 Oracle Database Specialist –Independent consultant System Performance tuning –PeopleSoft ERP –Oracle RDBMS Book –www.psftdba.com Who Am I?

3 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk3 Agenda Instrumentation –Oracle RDBMS –PeopleSoft PeopleTools Using PeopleSoft Performance Monitor –Problems –Performance Trace –Digging in the data

4 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk4 Resources If you can’t hear me say so now. Please feel free to ask questions as we go along. The presentation will be available from Conference website Article in UKOUG Oracle Scene magazine

5 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk5 Performance Tuning We do not use ratio based tuning any more. –Especially the buffer cache hit ratio We do use timed event based tuning. Further reading: –YAPP – Kolk, Yamaguchi, and Viscusi –Optimising Oracle Performance - Millsap & Holt –The Goal – Eli Goldratt

6 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk6 Performance Instrumentation Oracle does understand instrumentation. –Instrumentation is built in throughout the database kernel. PeopleTools 8.44 upwards includes Performance Monitor. –Instrumentation built into PeopleSoft technology.

7 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk7 Oracle RDBMS Instrumentation Dynamic Performance Views –Statspack –AWR / ADDM / ASH (licensed extras) Trace/Dump to operating system files –Enhanced by setting events –Profilers

8 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk8 Event Level 8 a.k.a. SQL*Trace with timed event information Includes –Every SQL Statement How long it took to parse/execute/fetch –Row source operation list How long each operation took –STATISTICS_LEVEL = ALL –Every event for which the database waits Can then profile with TKPROF or other profilers

9 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk9 What does this instrumentation do for us? If you have a performance problem, then you can determine exactly what the database is doing, and how long that is taking. –It can also prove that the problem is not located in the database! Hence, you can work out what to do about it.

10 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk10 Nørgaard’s Law Every 18 months one of your vendors will add another tier or layer of software somewhere in your stack. Below the database –SAN, Cache, Raid Controller Above the database –Application Server, Web Server, Java, E-WAN (Ever Wider Area Network), Encryption, Browser, JavaScript

11 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk11 You probably don’t need event 10046! SQL*Net Message from Client –Idle Event? –Database is Idle, but is the user idle? –Or is something else active in the technology stack (so the user is still waiting)? This event cannot distinguish between these conditions. You probably won’t get all you need from event 10046!

12 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk12 Timed Event Information Oracle timed events tell us about the database. We need similar information for every element in the technology chain. PeopleSoft realised this and instrumented their entire technology stack. –And they got it right!

13 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk13 PeopleTools Performance Utilities Instrumentation in PeopleTools 8.4 –Query Statistics 8.44, usable from 8.45 –PeopleSoft Ping 8.42, back-ported to 8.19 –Performance Monitor 8.44

14 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk14 Performance Monitor PeopleTools 8.44 –Fully instrumented –Including a timed-event interface –Event for the application –Useful PeopleBook –Additional analytics in PT8.45

15 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk15 Performance Monitor Architecture Based upon existing PeopleSoft technology Monitored System –Send information to servlet in monitoring system Monitoring System –Monitor servlet writes results to database via PSPPMSRV process in application server Ideally PeopleTools only system database This minimises measurement intrusion effect

16 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk16 Web Server Web Server Application Server (application logic) Application Server (application logic) APPQ PSAPPSRV DBMS (application data & meta- data DBMS (application data & meta- data SQL Tuxedo Message http / https Browser (presentation & JavaScript) Browser (presentation & JavaScript) Monitoring System PIA Servlet PSPPMSRV Monitored System Web Server (presentation logic) Web Server (presentation logic) PIA Servlet DBMS (application data & meta- data DBMS (application data & meta- data Tuxedo Message http / https Browser (presentation & JavaScript) Browser (presentation & JavaScript) Screen Paint Java Script Application Server (application logic) Application Server (application logic) APPQ PSAPPSRV PSMONITORSRV SQL PPMI Servlet 0345f1003x Monitor Servlet

17 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk17 Performance Monitor Architecture Instrumentation in –Application Server processes PSMONITORSRV collects host resource statistics –Memory –CPU –Process Scheduler –PIA servlet

18 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk18 Performance Monitor Metrics Transactions –User activities in PIA that cause communications with application server –Sampled –Enabled to form a trace Events –Periodic samples –Usually initiated by monitoring agents –eg. CPU, Tuxedo counters

19 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk19 Performance Monitor Transactions User activity in PIA Performance Monitoring Unit –Hierarchy of transactions Similar to Oracle event trace –recursive actions

20 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk20 Transactions Stored to PSPMTRANSCURR table –As PMUs are closed moved to PSPMTRANSHIST –Later deleted or archived to PSPMTRANSARCH ERD downloadable from Customer Connection

21 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk21 ERD of Transaction PSTRANSHIST (C) PSPMSYSDEFN (B) PSPMAGENT (A) PSPMCONTEXTDEFN (C1) PSPMMETRICDEFN (M1) PSPMCONTEXTDEFN (C2) PSPMCONTEXTDEFN (C3) PSPMMETRICDEFN (M2) PSPMMETRICDEFN (M3) PSPMMETRICDEFN (M4) PSPMMETRICDEFN (M5) PSPMMETRICDEFN (M6) PSPMMETRICDEFN (M7) PM_CONTEXTID_n (n=1-3) PM_SYSTEMID PM_TRANS_DEFN_SET, PM_TRANS_DEFN_ID PM_AGENTID PM_METRICIDn (n=1-7) search criteria PSPMTRANSDEFN (E)

22 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk22 Metrics Metric IDs specified on transaction definition PSPMTRANSDEFN –Metrics Types defined on PSPMMETRICDEFN Type 1: Counters (including timers) –Metric 4: Total Servlet Request time (ms) Type 2: Gauges –Metric 102: %CPU Used Type 3: Numeric Identifier –Metric 20: HTTP response code Type 4: String Identifier –Metric 27: File Name

23 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk23 Transaction 101 Reported at entry and exit of PIA servlet –Context 1 Action=View Page –Context 2 IP Address= –Context 3 Session ID=AN7tpzSwpZc4kt9k8... –Additional Description psc/ps/EMPLOYEE/HRMS/c/UTILITIES.PTPERF_TES T.GBL

24 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk24 Transaction metrics –Metric 19: Response Size (bytes) =17613 –Metric 20: Response Code =200 –Metric 22: Static Content Count =0 –Metric 23: Is this a Pagelet? =0

25 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk25 Transaction Query Results PM_TOP_INST_ID PM_INSTANCE_ID PM_PARENT_INST_ID DBNAME PM_HOST_PORT PM_DOMAIN_NAME PM_AGENT_TYPE PM_INSTANCE PM_AGENT_STRT_DTTM PM_MON_STRT_DTTM OPRID PM_PERF_TRACE PM_PROCESS_ID PM_TRANS_DEFN_ID DESCR60 'CONTEXT1:'||C.PM_CONTEXTID_1||'-'||C1.PM_CONTEXT_LABEL||'='||C.PM_CONTEXT_VALUE … PM_TRANS_DURATION 'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1 … PM_ADDTNL_DESCR HR88 go-faster-3:7201:7202 ps WEBSERVER -1 16:12: :12: PS PS: :01: Reported at entry and exit of PIA servlet Context1:3-Session ID=AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt! ! Context2:2-IP Address= Context3:1-Action=View Page 1322 Metric1:Response Size (bytes)=17613 Metric2:Response Code=200 Metric3:Static Content Count=0 Metric4:Is this a Pagelet?=0 Metric5:=0 Metric6:=0 Metric7:=

26 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk26 Events Certain events are defined for which the monitor agents collect metrics –Regular cycle Host resources –In response to user action PeopleSoft Ping –On an exception Jolt Exception or Query Timeout

27 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk27 Events Do not have an explicit context –Collecting agent provide context Stored in PSPMEVENTHIST –Later deleted or archived to PSPMEVETARCH

28 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk28 ERD of Events PSEVENTHIST (C) PSPMEVENTDEFN (E) PSPMSYSDEFN (B) PSPMAGENT (A) PSPMMETRICDEFN (M1) PSPMMETRICDEFN (M2) PSPMMETRICDEFN (M3) PSPMMETRICDEFN (M4) PSPMMETRICDEFN (M5) PSPMMETRICDEFN (M6) PSPMMETRICDEFN (M7) PM_AGENTIDPM_EVENT_DEFN_SET, PM_EVENT_DEFN_ID PM_SYSTEMIDsearch criteria PM_METRICIDn (n=1-7)

29 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk29 Event Query Results DBNAME PM_HOST_PORT PM_AGENT_TYPE PM_DOMAIN_NAME PM_INSTANCE PM_AGENT_DTTM PM_INSTANCE_ID PM_EVENT_DEFN_ID DESCR60 'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1 'METRIC2:'||M2.PM_METRICLABEL||'='||C.PM_METRIC_VALUE2 'METRIC3:'||M3.PM_METRICLABEL||'='||C.PM_METRIC_VALUE3 'METRIC4:'||M4.PM_METRICLABEL||'='||C.PM_METRIC_VALUE4 'METRIC5:'||M5.PM_METRICLABEL||'='||C.PM_METRIC_VALUE5 'METRIC6:'||M6.PM_METRICLABEL||'='||C.PM_METRIC_VALUE6 'METRIC7:'||M7.PM_METRICLABEL||'='||C.PM_METRIC_VALUE7 PM_ADDTNL_DESCR HR88 go-faster-3:7201:7202 WEBSERVER ps -1 16:12: PSPING metrics fowarded from browser Metric1:Network Latency (ms)=435 Metric2:WebServer Latency (ms)=100 Metric3:AppServer Latency (ms)=561 Metric4:DB Latency (millisecs)=451 Metric5:=0 Metric6:=0 Metric7:IP Address= PS;AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt! !

30 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk30 Using PeopleSoft Performance Monitor Bugs & Fixes Performance Tuning the Performance Monitor Purge Process Performance Trace Demo Network Latency

31 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk31 Bugs and Fixes Ports left in CLOSE_WAIT on Monitored System –Until Unix/Linux runs out of ports Windows doesn’t have a limit on number of ports –‘Application Server is Down’ message Affects PT8.49 –From at least –Fixed in

32 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk32 Tuning Performance Monitor Some of the delivered analytics do not perform well with even moderate data volumes –Set up the monitoring system to self monitor –Then you can generate PPM traces on the analytics –You will need additional indexes tuning-performance-monitor.html (YMMV)http://blog.psftdba.com/2006/04/performance- tuning-performance-monitor.html

33 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk33 Purge Process Data normally held in history tables –PSPMTRANSHIST, PSPMEVENTHIST Clone tables –PSPMTRANSHISTCL, PSPMEVENTHISTCL PPM writes to tables specified in PSPMTABLEMAP –Archive process switches this to clone tables select * from pspmtablemap; PM_TRANS_TBL_NAME PM_EVENT_TBL_NAME PSPMTRANSHIST PSPMEVENTHIST

34 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk34 PSPMTABLEMAP Archive/Purge switches PPM destination –Prevents concurrent INSERT and DELETE/Query operations –Saves read consistency problems on Oracle –Saves page locks on other databases PPM appears not to collect data during this process –But it is written to clone tables –Archive process moves it to main hist tables after purge

35 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk35

36 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk36 Purge Process Can’t Keep Up Platform generic query leads to full scan –Even if data has been deleted (manually) high water marks (HWM) on tables not reset Customisation –Oracle specific statement –May need to rebuild HIST tables to reset HWM In which case manually set PSPMTABLEMAP to clones, rebuild history tables, run archive process

37 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk37 Performance Fix for Purge Vanilla Code &TransHistSQL.Open("SELECT … AND %DateTimeDiff(X.PM_MON_STRT_DTTM, %CurrentDateTimeIn) >= (PM_MAX_HIST_AGE * 24 * 60)"); Expands to … AND ROUND((CAST(( SYSDATE) AS DATE) - CAST((X.PM_MON_STRT_DTTM) AS DATE)) * 1440, 0) >= (PM_MAX_HIST_AGE * 24 * 60) My Suggestions … AND X.PM_MON_STRT_DTTM < SYSDATE - Z.PM_MAX_HIST_AGE See blog entry performance-monitor.html performance-monitor.html

38 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk38 How much data? Control sampling –Proportion of transactions collected Depends upon activity on system On busy self-service system as little as 1in 5000 –Event sampling frequency For each agent 5 minutes – 15 minutes Depends on whether you want to be able to see shortlived behaviours.

39 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk39

40 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk40 Agent Filter Levels Controls amount of detail recorded in PMUs –4- standard –5 – verbose Includes SQL fetch operations Performance penalty –6 - debug

41 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk41 Agent Filter Levels

42 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk42 Analytics: System Performance

43 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk43 Analytics: Top Components

44 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk44 Performance Trace Generates a group of PMUs for activity in a user session –Choose an ID to identify records later

45 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk45 Performance Trace

46 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk46 Performance Monitoring Unit Look at PMU Tree Demonstration

47 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk47 Network Latency Most transactions are sampled But three transactions are always recorded See PMTRANSDEFN.PM_SAMPLING_ENABLE –108: User Session logout, expiration, timeout, or error –109: User Session began (user logged in) –116: Redirected round trip time (network latency)

48 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk48 Transaction 116 Network round trip from webserver to browser and back again –Includes network transmission time –Browser response time –Client IP address Although that could be load balancer or NAT –Operator ID LOCATION from HR database?

49 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk49

50 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk50 Analysis by Client IP IP addresses are of Routers not Clients Mon Jun 30 page 1 Login Durations by Client IP IP Address MIN AVG MED MAX VAR NUM_EVENTS , , , , , ,

51 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk51 HR location of OPRID First three lines are different network topology Operator may not actually be at stated location –esp home workers Mon Jun 30 page 1 Login Durations by Location (min 10 logins) LOCATION CITY Cty MIN AVG MED MAX VAR EVENTS EXT-00-IN Bangalore IND , EXT-00-CZ Prague CZE , EXT-MK-UK Milton Keynes GBR , HOME Home Worker* GBR INT-L3 Liverpool GBR INT-MK Milton Keynes GBR INT-WS Walsall GBR

52 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk52

53 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk53 Summary Very impressive feature rich piece of code –It could also put me out of business! In house support tool –An end user could be enabled to collect a performance trace.

54 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk54 My Questions Will Fusion have a Performance Monitor? –I think it should Will it be separately licensed product? –PeopleSoft’s performance monitor isn’t

55 Your Questions?

56 Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk56 Practical Guidance on the Use of PeopleSoft Performance Monitor David Kurtz Go-Faster Consultancy Ltd.


Download ppt "Practical PSFT PPMUKOUG2008 ©www.go-faster.co.uk1 Practical Guidance on the Use of PeopleSoft Performance Monitor David Kurtz Go-Faster Consultancy Ltd."

Similar presentations


Ads by Google