Practical Guidance on the Use of PeopleSoft Performance Monitor

Practical Guidance on the Use of PeopleSoft Performance Monitor
David Kurtz Go-Faster Consultancy Ltd. Practical PSFT PPM UKOUG2008 ©

UKOUG2008 ©www.go-faster.co.uk
Who Am I? Oracle Database Specialist Independent consultant System Performance tuning PeopleSoft ERP Oracle RDBMS Book Practical PSFT PPM UKOUG2008 © 2

Agenda Instrumentation Oracle RDBMS PeopleSoft PeopleTools Using PeopleSoft Performance Monitor Problems Performance Trace Digging in the data Practical PSFT PPM UKOUG2008 ©

Resources If you can’t hear me say so now. Please feel free to ask questions as we go along. The presentation will be available from Conference website Article in UKOUG Oracle Scene magazine Practical PSFT PPM UKOUG2008 ©

Performance Tuning We do not use ratio based tuning any more. Especially the buffer cache hit ratio We do use timed event based tuning. Further reading: YAPP – Kolk, Yamaguchi, and Viscusi Optimising Oracle Performance - Millsap & Holt The Goal – Eli Goldratt When I was a young, clean shaven DBA, performance tuning was simple. If the system was slow you measured the buffer cache hit ratio and if it was below 90% you added more memory. Practical PSFT PPM UKOUG2008 ©

Performance Instrumentation
Oracle does understand instrumentation. Instrumentation is built in throughout the database kernel. PeopleTools 8.44 upwards includes Performance Monitor. Instrumentation built into PeopleSoft technology. … but to explain why app instrumentation is so important lets go back to the database Practical PSFT PPM UKOUG2008 ©

Oracle RDBMS Instrumentation
Dynamic Performance Views Statspack AWR / ADDM / ASH (licensed extras) Trace/Dump to operating system files Enhanced by setting events Profilers Practical PSFT PPM UKOUG2008 ©

Event Level 8 a.k.a. SQL*Trace with timed event information Includes Every SQL Statement How long it took to parse/execute/fetch Row source operation list How long each operation took STATISTICS_LEVEL = ALL Every event for which the database waits Can then profile with TKPROF or other profilers Practical PSFT PPM UKOUG2008 ©

What does this instrumentation do for us?
If you have a performance problem, then you can determine exactly what the database is doing, and how long that is taking. It can also prove that the problem is not located in the database! Hence, you can work out what to do about it. Practical PSFT PPM UKOUG2008 ©

Nørgaard’s Law Every 18 months one of your vendors will add another tier or layer of software somewhere in your stack. Below the database SAN, Cache, Raid Controller Above the database Application Server, Web Server, Java, E-WAN (Ever Wider Area Network), Encryption, Browser, JavaScript Practical PSFT PPM UKOUG2008 ©

You probably don’t need event 10046!
You probably won’t get all you need from event 10046! You probably don’t need event 10046! SQL*Net Message from Client Idle Event? Database is Idle, but is the user idle? Or is something else active in the technology stack (so the user is still waiting)? This event cannot distinguish between these conditions. Practical PSFT PPM UKOUG2008 ©

Timed Event Information
Oracle timed events tell us about the database. We need similar information for every element in the technology chain. PeopleSoft realised this and instrumented their entire technology stack. And they got it right! Practical PSFT PPM UKOUG2008 ©

PeopleTools Performance Utilities
Instrumentation in PeopleTools 8.4 Query Statistics 8.44, usable from 8.45 PeopleSoft Ping 8.42, back-ported to 8.19 Performance Monitor 8.44 Chapter 10 PeopleTools Performance Utilities In PeopleTools 8.4, PeopleSoft has started to introduce some utilities to help detect and identify performance problems in the online part of the application. In this chapter, I discuss some of aspects of these utilities. This chapter does not seek to replace PeopleSoft’s documentation. Instead, you are advised to read this chapter in conjunction with PeopleBooks. Practical PSFT PPM UKOUG2008 ©

Performance Monitor PeopleTools 8.44 Fully instrumented Including a timed-event interface Event for the application Useful PeopleBook Additional analytics in PT8.45 Performance Monitor PeopleSoft Performance Monitor, introduced in PeopleTools 8.44, represents a huge effort by PeopleSoft to fully instrument the application and produce a “wait interface.” To put it in terms that an Oracle DBA would recognize, it provides for the application what an Oracle SQL trace with the wait interface (event 10046, level 8) does for the database. At the time of this writing, I am working with PeopleTools 8.44, and I have yet to use Performance Monitor in a real production environment, so the comments in this section are based upon some limited experiments that I have performed. One of the most significant unknowns remains what the overhead (measurement intrusion effect) is of using this facility on an active production system. In this section I provide only a brief overview of the utility. The product documentation in PeopleTools 8.44 PeopleBook: PeopleSoft Performance Monitor is very detailed and well worth reading. Practical PSFT PPM UKOUG2008 ©

Performance Monitor Architecture
Based upon existing PeopleSoft technology Monitored System Send information to servlet in monitoring system Monitoring System Monitor servlet writes results to database via PSPPMSRV process in application server Ideally PeopleTools only system database This minimises measurement intrusion effect Architectural Overview PeopleSoft has separated the job of obtaining the metrics from the various components being monitored from the job of storing those metrics in a database. So the overhead— especially the database processing and space overhead—of inserting and manipulating the performance metrics data can be separated from the production database. Thus if a company runs more than one PeopleSoft system that it wishes to monitor, it could build another, perhaps PeopleTools-only, system just to hold the metrics (although this may have a licensing implication). So there is a concept of both a monitoring system and a monitored system (see Figure 10-8). A system can be configured to self-monitor, but there are various warnings against doing so on a production system due to the associated overhead. However, self-monitoring cannot lead to a recursive loop. Practical PSFT PPM UKOUG2008 ©

0345f1003x Application Server Web Browser Server DBMS
(application logic) APPQ PSAPPSRV DBMS (application data & meta-data SQL Tuxedo Message http / https Browser (presentation & JavaScript) Monitoring System PIA Servlet PPMI Servlet PSPPMSRV Monitor Servlet Browser (presentation & JavaScript) Screen Paint Java Script Web Server (presentation logic) PIA Servlet Application Server (application logic) APPQ PSAPPSRV DBMS (application data & meta-data http / https Tuxedo Message SQL You can set this up to self-monitor, and I do on my laptop because I have limited resources. But it is not recommended, because you start to create a significant measurement intrusion effect. PSMONITORSRV Monitored System Practical PSFT PPM UKOUG2008 ©

Performance Monitor Architecture
Instrumentation in Application Server processes PSMONITORSRV collects host resource statistics Memory CPU Process Scheduler PIA servlet To obtain the metrics from the monitored system, PeopleSoft has added code to the PIA servlet, and to the various servers in the application server and Process Scheduler. The monitor server (PSMONITORSRV) was added in PeopleTools 8.44 to collect host resource usage metrics. It also connects to the Tuxedo Bulletin Board to collect information about the Tuxedo domain (see Chapter 2 for a discussion of the Tuxedo architecture). In the Performance Monitor package, these various sources of metrics are referred to as agents. The agents send the metrics to the monitor servlet in the monitoring system. The netstat command reports a number of persistently established connections between the agents and the web server. Note PeopleSoft systems usually have multiple application servers and multiple web servers. Metrics must be collected from each application server and each web server. On the monitoring system, the metrics are collected by the monitoring servlet and sent on to the performance collator server, PSPPMSRV, which then inserts the metrics into the database. Practical PSFT PPM UKOUG2008 ©

Performance Monitor Metrics
Transactions User activities in PIA that cause communications with application server Sampled Enabled to form a trace Events Periodic samples Usually initiated by monitoring agents eg. CPU, Tuxedo counters Metrics The metrics collected by Performance Monitor break down into two major groups: * Transactions consist of the activities that occur when operator does something in the PIA that causes communication with the servlet, such as clicking a Save button in a component. * Events are usually initiated periodically by the monitoring agents to collect metrics, such as CPU or Tuxedo metrics. I discuss each group in further detail in the sections that follow. Practical PSFT PPM UKOUG2008 ©

Performance Monitor Transactions
User activity in PIA Performance Monitoring Unit Hierarchy of transactions Similar to Oracle event trace recursive actions Transactions Transactions are the activities that occur in the web server and application server when a user initiates a conversation with the web server by clicking a link or button, or navigating between fields in the PIA. A transaction can be composed of other transactions. For example, a PIA request can comprise one or more Jolt requests, the Jolt request(s) will give rise to a Tuxedo service, and so on. Each of these elements is a separate transaction. This is similar to an Oracle trace, where user SQL statements can give rise to recursive SQL statements, or a fetch operation can include sequential and/or scattered read operations. All of the transactions that occur in response to a single user action are collectively called a Performance Monitoring Unit (PMU). The transactions in a PMU exist in a hierarchy and can be displayed as a tree structure in the PIA. Figure 10-9 shows the PMU for a single PeopleSoft Ping. The level of detail in a PMU is controlled by the filtering level, which is discussed later in this chapter Transactions are pieces of processing that take a certain amount of time to execute, so they all have a duration in addition to the various other metrics. Practical PSFT PPM UKOUG2008 ©

Transactions Stored to PSPMTRANSCURR table As PMUs are closed moved to PSPMTRANSHIST Later deleted or archived to PSPMTRANSARCH ERD downloadable from Customer Connection All of the output from Performance Monitor is stored in the database of the monitoring system.[1] Current or open PMUs are temporarily placed in the table PSPMTRANSCURR, and when they are completed they are moved to PSPMTRANSHIST. Old PMUs can be deleted or archived to PSPMTRANSARCH. These three tables have similar structures (see Table 10-2). It is worth taking some time to understand the structure of these tables, so that you can build your own queries. Table Structure of PSPMTRANS% Tables [1] An ERD diagram of the performance monitoring tables (PSPM%) can be downloaded from the Peoplesoft Connection website under Documentation Updates -> Enterprise -> PeopleTools -> Performance Monitor. There are also two use-case flowcharts for troubleshooting. Practical PSFT PPM UKOUG2008 ©

ERD of Transaction Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk
PSPMTRANSDEFN (E) PSTRANSHIST (C) PSPMAGENT (A) PM_TRANS_DEFN_SET, PM_TRANS_DEFN_ID PM_AGENTID search criteria PM_SYSTEMID PSPMSYSDEFN (B) PSPMMETRICDEFN (M1) PSPMMETRICDEFN (M2) PM_METRICIDn (n=1-7) PSPMMETRICDEFN (M3) PM_CONTEXTID_n (n=1-3) PSPMMETRICDEFN (M4) PSPMMETRICDEFN (M5) PSPMCONTEXTDEFN (C1) The interrelationship of the various tables that hold and describe transactions is shown in Figure If you are collecting information for multiple monitored systems on the one monitoring system, then the database name is related to the agent. PSPMMETRICDEFN (M6) PSPMCONTEXTDEFN (C2) PSPMMETRICDEFN (M7) PSPMCONTEXTDEFN (C3) Practical PSFT PPM UKOUG2008 ©

Metrics Metric IDs specified on transaction definition PSPMTRANSDEFN Metrics Types defined on PSPMMETRICDEFN Type 1: Counters (including timers) Metric 4: Total Servlet Request time (ms) Type 2: Gauges Metric 102: %CPU Used Type 3: Numeric Identifier Metric 20: HTTP response code Type 4: String Identifier Metric 27: File Name Each transaction can hold up to seven metrics, one of which can be a character string. The PM_METRICID can be looked up on PSPMMETRICDEFN. Each metric definition has a metric type (PM_METRICTYPE): *1: Counter. For example, metric 4 is “Total servlet request time (ms)”. The metric accumulates the time and reports it each time transaction 152 occurs. In order to find the servlet time accumulated since the last transaction 152 occurred, you need to subtract the value of the metric on the last transaction record from the current value. *2: Gauge. For example, metric 102 is “%CPU Used”. This type of metric is a scalar quantity. That is to say it contains a measurement of a particular quantity. The value can be understood in isolation, as opposed to a counter that must be compared. *3: Numeric identifier. For example, metric 20 is the HTTP response code. This is a number that has a meaning, but you don’t do any arithmetic with it. However, there are some type 3 metrics that should really be type 2, such as metric 37, “Number of Tuxedo Servers”. *4: String identifier. For example, metric 27 is a file name. Practical PSFT PPM UKOUG2008 ©

Transaction 101 Reported at entry and exit of PIA servlet Context 1 Action=View Page Context 2 IP Address= Context 3 Session ID=AN7tpzSwpZc4kt9k Additional Description psc/ps/EMPLOYEE/HRMS/c/UTILITIES.PTPERF_TES T.GBL In Listing 10-13, transaction 101 is “Reported at entry and exit of PIA servlet”. It is the root transaction of the PMU shown in Figure It has three contexts: * Context 1: Action=View Page * Context 2: IP Address= * Context 3: Session ID=AN7tpzSwpZc4kt9k However, the additional description is often more useful. In this case, it is the URL of the component: Practical PSFT PPM UKOUG2008 ©

Transaction 101 4 metrics Metric 19: Response Size (bytes) =17613 Metric 20: Response Code =200 Metric 22: Static Content Count =0 Metric 23: Is this a Pagelet? Transaction 101 has 4 metrics: * Metric 19: “Response Size (bytes)” * Metric 20: “Response Code” * Metric 22: “Static Content Count” * Metric 23: “Is this a Pagelet?” Practical PSFT PPM UKOUG2008 ©

Transaction Query Results
PM_TOP_INST_ID PM_INSTANCE_ID PM_PARENT_INST_ID DBNAME PM_HOST_PORT PM_DOMAIN_NAME PM_AGENT_TYPE PM_INSTANCE PM_AGENT_STRT_DTTM PM_MON_STRT_DTTM OPRID PM_PERF_TRACE PM_PROCESS_ID PM_TRANS_DEFN_ID DESCR60 'CONTEXT1:'||C.PM_CONTEXTID_1||'-'||C1.PM_CONTEXT_LABEL||'='||C.PM_CONTEXT_VALUE … PM_TRANS_DURATION 'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1 … PM_ADDTNL_DESCR HR88 go-faster-3:7201:7202 ps WEBSERVER :12: :12: PS PS: :01: 101 Reported at entry and exit of PIA servlet Context1:3-Session ID=AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt! ! Context2:2-IP Address= Context3:1-Action=View Page 1322 Metric1:Response Size (bytes)=17613 Metric2:Response Code=200 Metric3:Static Content Count=0 Metric4:Is this a Pagelet?=0 Metric5:=0 Metric6:=0 Metric7:= Practical PSFT PPM UKOUG2008 ©

Events Certain events are defined for which the monitor agents collect metrics Regular cycle Host resources In response to user action PeopleSoft Ping On an exception Jolt Exception or Query Timeout Events Certain events are defined that also cause the monitor agents to collect various related metrics. These events can occur * On a regular cycle, such as host resource or Tuxedo server metrics * In response to a user action, such as a PeopleSoft Ping * On an exception, such as a Jolt exception or query timeout Practical PSFT PPM UKOUG2008 ©

Events Do not have an explicit context Collecting agent provide context Stored in PSPMEVENTHIST Later deleted or archived to PSPMEVETARCH E Practical PSFT PPM UKOUG2008 ©

ERD of Events Practical PSFT PPM UKOUG2008 ©www.go-faster.co.uk
PSPMEVENTDEFN (E) PSEVENTHIST (C) PSPMAGENT (A) PM_EVENT_DEFN_SET, PM_EVENT_DEFN_ID PM_AGENTID search criteria PM_SYSTEMID PSPMSYSDEFN (B) PSPMMETRICDEFN (M1) PSPMMETRICDEFN (M2) PM_METRICIDn (n=1-7) PSPMMETRICDEFN (M3) PSPMMETRICDEFN (M4) The structure for events (see Figure 10-11) is less complicated than transactions because there are no contexts PSPMMETRICDEFN (M5) PSPMMETRICDEFN (M6) PSPMMETRICDEFN (M7) Practical PSFT PPM UKOUG2008 ©

Event Query Results DBNAME PM_HOST_PORT PM_AGENT_TYPE PM_DOMAIN_NAME PM_INSTANCE PM_AGENT_DTTM PM_INSTANCE_ID PM_EVENT_DEFN_ID DESCR60 'METRIC1:'||M1.PM_METRICLABEL||'='||C.PM_METRIC_VALUE1 'METRIC2:'||M2.PM_METRICLABEL||'='||C.PM_METRIC_VALUE2 'METRIC3:'||M3.PM_METRICLABEL||'='||C.PM_METRIC_VALUE3 'METRIC4:'||M4.PM_METRICLABEL||'='||C.PM_METRIC_VALUE4 'METRIC5:'||M5.PM_METRICLABEL||'='||C.PM_METRIC_VALUE5 'METRIC6:'||M6.PM_METRICLABEL||'='||C.PM_METRIC_VALUE6 'METRIC7:'||M7.PM_METRICLABEL||'='||C.PM_METRIC_VALUE7 PM_ADDTNL_DESCR HR go-faster-3:7201:7202 WEBSERVER ps :12: 600 PSPING metrics fowarded from browser Metric1:Network Latency (ms)=435 Metric2:WebServer Latency (ms)=100 Metric3:AppServer Latency (ms)=561 Metric4:DB Latency (millisecs)=451 Metric5:=0 Metric6:=0 Metric7:IP Address= PS;AN7tpzSwpZc4kt9k8QNaCcYUWWh9FaFt! ! Practical PSFT PPM UKOUG2008 ©

Using PeopleSoft Performance Monitor
Bugs & Fixes Performance Tuning the Performance Monitor Purge Process Performance Trace Demo Network Latency Practical PSFT PPM UKOUG2008 ©

Bugs and Fixes Ports left in CLOSE_WAIT on Monitored System Until Unix/Linux runs out of ports Windows doesn’t have a limit on number of ports ‘Application Server is Down’ message Affects PT8.49 From at least Fixed in Practical PSFT PPM UKOUG2008 ©

Tuning Performance Monitor
Some of the delivered analytics do not perform well with even moderate data volumes Set up the monitoring system to self monitor Then you can generate PPM traces on the analytics You will need additional indexes (YMMV) Practical PSFT PPM UKOUG2008 ©

Purge Process Data normally held in history tables PSPMTRANSHIST, PSPMEVENTHIST Clone tables PSPMTRANSHISTCL, PSPMEVENTHISTCL PPM writes to tables specified in PSPMTABLEMAP Archive process switches this to clone tables select * from pspmtablemap; PM_TRANS_TBL_NAME PM_EVENT_TBL_NAME PSPMTRANSHIST PSPMEVENTHIST Practical PSFT PPM UKOUG2008 ©

PSPMTABLEMAP Archive/Purge switches PPM destination Prevents concurrent INSERT and DELETE/Query operations Saves read consistency problems on Oracle Saves page locks on other databases PPM appears not to collect data during this process But it is written to clone tables Archive process moves it to main hist tables after purge Practical PSFT PPM UKOUG2008 ©

Practical PSFT PPM UKOUG2008 ©

Purge Process Can’t Keep Up
Platform generic query leads to full scan Even if data has been deleted (manually) high water marks (HWM) on tables not reset Customisation Oracle specific statement May need to rebuild HIST tables to reset HWM In which case manually set PSPMTABLEMAP to clones, rebuild history tables, run archive process Practical PSFT PPM UKOUG2008 ©

Performance Fix for Purge
Vanilla Code &TransHistSQL.Open("SELECT … AND %DateTimeDiff(X.PM_MON_STRT_DTTM, %CurrentDateTimeIn) >= (PM_MAX_HIST_AGE * 24 * 60)"); Expands to … AND ROUND((CAST(( SYSDATE) AS DATE) - CAST((X.PM_MON_STRT_DTTM) AS DATE)) * 1440, 0) >= (PM_MAX_HIST_AGE * 24 * 60) My Suggestions … AND X.PM_MON_STRT_DTTM < SYSDATE - Z.PM_MAX_HIST_AGE See blog entry Practical PSFT PPM UKOUG2008 ©

How much data? Control sampling Proportion of transactions collected Depends upon activity on system On busy self-service system as little as 1in 5000 Event sampling frequency For each agent 5 minutes – 15 minutes Depends on whether you want to be able to see shortlived behaviours. Practical PSFT PPM UKOUG2008 ©

Agent Filter Levels Controls amount of detail recorded in PMUs 4- standard 5 – verbose Includes SQL fetch operations Performance penalty 6 - debug If the Ping PMU shown in Figure 10-9 had been monitored with agents only at the standard filter level, it would not have shown any detail below the ICPanel service. By setting the filter level to 5 (verbose), the SQL statement and fetch operations were reported. However, there are warnings in both the Agent Filters component and the documentation that increasing the filter level to 5 (verbose) or 6 (debug) can have a significant performance impact. Practical PSFT PPM UKOUG2008 ©

Agent Filter Levels Agent Filter Levels The event and transaction definitions include a filter level (see Figure 10-12). The agents generate the metrics only if the filter level of the event is less than or equal to the filter level for the agent. The default agent filter level is 4 for all agents. Practical PSFT PPM UKOUG2008 ©

Analytics: System Performance

Analytics: Top Components

Performance Trace Generates a group of PMUs for activity in a user session Choose an ID to identify records later Performance Trace The performance trace is essentially a way of grouping an operator’s PMUs. So an operator can enable a trace session and perform a series of actions, giving rise to a series of PMUs. When the performance trace is enabled, a default name composed of the operator’s ID and the current date and time is generated (see Figure 10-13), but this can be overridden. The performance trace name is also written to those PMU records. Later, just the transactions for that operator’s session can be retrieved and analyzed. Thus you can find out where and how much time that operator spent waiting for the system to respond. Figure Performance Trace Console dialog If the operator overrides the filter settings for their trace, event 601 is also generated. Practical PSFT PPM UKOUG2008 ©

Network Latency Most transactions are sampled But three transactions are always recorded See PMTRANSDEFN.PM_SAMPLING_ENABLE 108: User Session logout, expiration, timeout, or error 109: User Session began (user logged in) 116: Redirected round trip time (network latency) Practical PSFT PPM UKOUG2008 ©

Transaction 116 Network round trip from webserver to browser and back again Includes network transmission time Browser response time Client IP address Although that could be load balancer or NAT Operator ID LOCATION from HR database? Practical PSFT PPM UKOUG2008 ©

Analysis by Client IP IP addresses are of Routers not Clients Mon Jun page 1 Login Durations by Client IP IP Address MIN AVG MED MAX VAR NUM_EVENTS , , , , , , Practical PSFT PPM UKOUG2008 ©

HR location of OPRID First three lines are different network topology Operator may not actually be at stated location esp home workers Mon Jun page 1 Login Durations by Location (min 10 logins) LOCATION CITY Cty MIN AVG MED MAX VAR EVENTS EXT-00-IN Bangalore IND , EXT-00-CZ Prague CZE , EXT-MK-UK Milton Keynes GBR , HOME Home Worker* GBR INT-L3 Liverpool GBR INT-MK Milton Keynes GBR INT-WS Walsall GBR Practical PSFT PPM UKOUG2008 ©

Summary Very impressive feature rich piece of code It could also put me out of business! In house support tool An end user could be enabled to collect a performance trace. Practical PSFT PPM UKOUG2008 ©

My Questions Will Fusion have a Performance Monitor? I think it should Will it be separately licensed product? PeopleSoft’s performance monitor isn’t Practical PSFT PPM UKOUG2008 ©

Your Questions?

Practical Guidance on the Use of PeopleSoft Performance Monitor
David Kurtz Go-Faster Consultancy Ltd. Practical PSFT PPM UKOUG2008 ©

Practical Guidance on the Use of PeopleSoft Performance Monitor

Similar presentations

Presentation on theme: "Practical Guidance on the Use of PeopleSoft Performance Monitor"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Practical Guidance on the Use of PeopleSoft Performance Monitor

Similar presentations

Presentation on theme: "Practical Guidance on the Use of PeopleSoft Performance Monitor"— Presentation transcript:

Similar presentations

About project

Feedback