Performance Tuning for PeopleSoft Administrators

Performance Tuning for PeopleSoft Administrators
Row Migration can Aggravate Contention on Cache Buffer Chains Latch 01/04/2017 Performance Tuning for PeopleSoft Administrators David Kurtz Go-Faster Consultancy Ltd. ©2011 Go-Faster Consultancy Ltd.

Row Migration can Aggravate Contention on Cache Buffer Chains Latch
01/04/2017 01/04/2017 Who Am I? Oracle Database Specialist Independent consultant Performance tuning PeopleSoft ERP Oracle RDBMS Book Oak Table Who am I slides are boring, mainly because there are boring, but also because by the time I have said anything you have already read everything. However, it does give an opportunity to put up the book cover for the second edition! UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2011 Go-Faster Consultancy Ltd. ©2011 Go-Faster Consultancy Ltd. 2

Draft Presentation 01/04/2017 It depends… The answer to all questions should be deemed to be implicitly prefixed ‘it depends’ if they are not already explicitly so prefixed. The reasons for this include, but are not limited to: There may be exceptions where the answer is either false or not completely true There may be exceptions to the exceptions and so on ad infinitum. The question may not explicitly scope all the conditions upon which the answer depends Just about every Oracle presentation I have seen for the last 5 years or so has started with a ‘safe harbor’ statement. So, I thought I should have one too. Oracle’s SH statement basically says ‘don’t quote me on this’. Mine is similar. Inevitably, this kind of presentation deals in generalisations. In practice, most performance issues are related to something unique about the particular system, business. I may say things that don’t apply universally. That may occur because … <see slide> Let me illustrate. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Draft Presentation 01/04/2017 ‘Not Completely True’ ( A discrepancy in Mercury's orbit pointed out in 1859, by the French mathematician and astronomer Urbain Le Verrier showed up a flaw in Newton's theory. By the end of the 19th century, it was known that its orbit showed slight perturbations that could not be accounted for entirely under Newton's theory, but all searches for another perturbing body (such as a planet orbiting the Sun even closer than Mercury) had been fruitless. The issue was resolved in 1915 by Albert Einstein's new theory of general relativity, which accounted for the small discrepancy in Mercury's orbit. Einstein proposed that spacetime is curved by matter, and that free-falling objects are moving along locally straight paths in curved spacetime. These straight paths are called geodesics. The perihelion of Mercury's orbit precesses around the Sun at an excess of 43 arcseconds per century beyond what is predicted by Newtonian mechanics, a phenomenon that was explained in the 20th century by Albert Einstein's General Theory of Relativity. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

‘Not Adequately Specified’
Draft Presentation 01/04/2017 ‘Not Adequately Specified’ 342 -173 =147 =1478 Tom Lehrer – base 8 The question was not adequately specified. When we talk about performance we need to be precise and rigerous. For me, this is exactly like being back in the lab. Performance problems are a form of experimental science, and they require use of the scientific method. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Performance Performance is exactly what the user perceives it to be. No more, no less. Performance is ‘poor’ when the user’s perception does not match their expectation. Two serious points to make during this slide:We need to know what we mean by performance You have to be able to measure the user experience (or at least close to) You need a working quantitative definition of acceptable performance. A performance SLA. This is a threshold for justifying a performance improvement activity, and calling a halt to it when you have achieved enough. UKOUG Performance Tuning for PeopleSoft Administrators ©2012

“Data, data, data, I cannot make bricks without clay.”
You don’t need fancy monitoring software to tell you that a system is performing poorly. That is what users are for. But you do need to record and monitor metrics to provide diagnostic data I am not saying you don’t need metrics or that you don’t need to record and monitor them because you do. But not all metrics are indicators of system performance. Many have valuable diagnostic value. However, so you have to use thoughtfully. Disk stats are a good example of this. Disk folk talk about Iops. Average response time is one thing, but they don’t measure total time spent. Mainly because they can’t without significant overhead. Often stats are averaged over large periods of time. The slide title is from: The Adventure of the Copper Beeches UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Performance Tuning is a search for lost time.
Detection is, or ought to be, an exact science. It should be treated in the same cold and unemotional manner. The Sign of Four, Arthur Conan-Doyle When you look at performance focus on time spent doing stuff or waiting for stuff to happen. Follow the lost time. I could have structured this presentation as a series of tips, but that is not good enough, it would be doing a disservice to you the audience. Top tips and best practices are not silver bullets, and the point I really want to make in this preamble is that if you are going to address performance issues then you need to understand how things work under the bonnet (or should that been hood). Performance Tuning requires a logical, rigorous and repeatable evidence-based approach. The method you use to approach performance issues is more important than the individual pieces of advice that I will get onto in the rest of this presentation, because if you don’t use the right method, there are going to be times when you reach the wrong answer. So, I am going to pick some things that I think are important and explain how they work rather and what that could mean to you. And even then I am not going into nearly as much detail as some things rethan run through a huge list of facts. UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Draft Presentation 01/04/2017 We do not use guesswork Say rather ... we balance probabilities and choose the most likely. It is the scientific use of the imagination, but we have always some material basis on which to start our speculation. The Hound of the Baskervilles, Arthur Conan-Doyle I could have structured this presentation as a series of tips, but that is not good enough, it would be doing a disservice to you the audience. Top tips and best practices are not silver bullets, and the point I really want to make in this preamble is that if you are going to address performance issues then you need to understand how things work under the bonnet (or should that been hood). "We are coming now rather into the region of guesswork," said Dr. Mortimer. "Say, rather, into the region where we balance probabilities and choose the most likely. It is the scientific use of the imagination, but we have always some material basis on which to start our speculation. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

PeopleTools Performance Monitor
Draft Presentation 01/04/2017 PeopleTools Performance Monitor UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Performance Monitor Part of PeopleTools
Draft Presentation 01/04/2017 Performance Monitor Part of PeopleTools PeopleTools to monitor PeopleTools Since PeopleTools 8.44 Fully instrumented Including a timed-event interface for the component processor Event for the application Useful PeopleBook No separate licence Performance Monitor PeopleSoft Performance Monitor, introduced in PeopleTools 8.44, represents a huge effort by PeopleSoft to fully instrument the application and produce a “wait interface.” To put it in terms that an Oracle DBA would recognize, it provides for the application what an Oracle SQL trace with the wait interface (event 10046, level 8) does for the database. The product documentation in PeopleTools PeopleBook: PeopleSoft Performance Monitor is very detailed and well worth reading. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

0345f1003x Web Application Server Browser Server DBMS
(application logic) APPQ PSAPPSRV DBMS (application data & meta-data SQL Tuxedo Message http / https Browser (presentation & JavaScript) Monitoring System PIA Servlet PPMI Servlet PSPPMSRV Monitor Servlet Browser (presentation & JavaScript) Screen Paint Java Script Web Server (presentation logic) PIA Servlet Application Server (application logic) APPQ PSAPPSRV DBMS (application data & meta-data http / https Tuxedo Message SQL You can set this up to self-monitor, and I do on my laptop because I have limited resources. But it is not recommended, because you start to create a significant measurement intrusion effect. However, I do recommend setting up self-monitoring on the monitoring system It proves that PPM is working You can use PPM to tease out the performance issues in the PPM analytic components PSMONITORSRV Monitored System UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Performance Monitor Metrics
Transactions User activities in PIA that cause communications with application server Sampled Enabled to form a trace PSPMTRANSHIST Events Periodic samples Usually initiated by monitoring agents eg. CPU, Tuxedo counters PSPMEVENTHIST Metrics The metrics collected by Performance Monitor break down into two major groups: * Transactions consist of the activities that occur when operator does something in the PIA that causes communication with the servlet, such as clicking a Save button in a component. * Events are usually initiated periodically by the monitoring agents to collect metrics, such as CPU or Tuxedo metrics. I discuss each group in further detail in the sections that follow. UKOUG Performance Tuning for PeopleSoft Administrators ©2012

PeopleSoft for the Oracle DBA
Tuxedo Service Trace Web Server Access Log PeopleTools Trace Oracle SQL*Trace Browser (presentation & JavaScript) Web Server (presentation logic) Servlet Thread Application Server (application logic) APPQ PSAPPSRV DBMS (application data & meta-data Screen Paint Java Script http / https Tuxedo Message SQL In the good old days you had some basic logging. The web server has an access log. It will list every request that comes through the web server. Need to bounce web server to enable/disable. Tuxedo service trace will tell you want the service is, but nearly everything is an ICPanel. Need to reconfigure and reboot Tuxedo domain to enable or disable (you could use the Tuxedo administrative applet too). PeopleTools trace lists most of the SQL issues by PeopleTools, some of the timing are a bit suspect. PeopleCode trace has a very significant overhead. You wouldn’t really want to run a production system with these traces running, they are really diagnostic traces for non-Prod systems. SQL*Trace is a session trace, and that can also be a bit of a blunderbus when it comes to the application server. 101: PIA Request 400: Tuxedo Service 400: PeopleCode only 115: Jolt Time 401: ICPanel 406/407/408: SQL Exec UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©Go-Faster Consultancy Ltd.

Analytics: System Performance
The system performance component mainly shows event info. It gives you a nice overview of a system, and then you can drill into areas in detail UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Analytics: Top Components
Top compoents by cumulative time. App Server time but can tell you if you have a problem. This is real data from a real customer site, and there is a long story that goes with it, but it did show a very clear problem. UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Performance Trace Generates a group of PMUs for activity in a user session Choose an ID to identify records later Verbose Includes SQL Significant overhead. Don’t use as default. Trace only. Performance Trace The performance trace is essentially a way of grouping an operator’s PMUs. So an operator can enable a trace session and perform a series of actions, giving rise to a series of PMUs. When the performance trace is enabled, a default name composed of the operator’s ID and the current date and time is generated (see Figure 10-13), but this can be overridden. The performance trace name is also written to those PMU records. Later, just the transactions for that operator’s session can be retrieved and analyzed. Thus you can find out where and how much time that operator spent waiting for the system to respond. Figure Performance Trace Console dialog If the operator overrides the filter settings for their trace, event 601 is also generated. UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Performance Monitor Transactions
User activity in PIA Performance Monitoring Unit Hierarchy of transactions Similar to Oracle event trace recursive actions Verbose Transactions Transactions are the activities that occur in the web server and application server when a user initiates a conversation with the web server by clicking a link or button, or navigating between fields in the PIA. A transaction can be composed of other transactions. For example, a PIA request can comprise one or more Jolt requests, the Jolt request(s) will give rise to a Tuxedo service, and so on. Each of these elements is a separate transaction. This is similar to an Oracle trace, where user SQL statements can give rise to recursive SQL statements, or a fetch operation can include sequential and/or scattered read operations. All of the transactions that occur in response to a single user action are collectively called a Performance Monitoring Unit (PMU). The transactions in a PMU exist in a hierarchy and can be displayed as a tree structure in the PIA. Figure 10-9 shows the PMU for a single PeopleSoft Ping. The level of detail in a PMU is controlled by the filtering level, which is discussed later in this chapter Transactions are pieces of processing that take a certain amount of time to execute, so they all have a duration in addition to the various other metrics. UKOUG Performance Tuning for PeopleSoft Administrators ©2012

How much data? Control sampling Proportion of transactions collected
Draft Presentation 01/04/2017 How much data? Control sampling Proportion of transactions collected Depends upon activity on system On busy self-service system as little as 1 in 1000 It depends! You will have to decide for yourself. Event sampling frequency For each agent 5 minutes – 15 minutes Depends on whether you want to be able to see short-lived behaviours. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

UKOUG2012 - Performance Tuning for PeopleSoft Administrators
Draft Presentation 01/04/2017 UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Performance Tuning the Performance Monitor
Draft Presentation 01/04/2017 Performance Tuning the Performance Monitor Always configure self-monitoring of the monitoring database So you know it works So you can work out why the analytics are slow! Performance Tuning the Performance Monitor Archive Process Additional indexes UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Performance Monitor Data
Draft Presentation 01/04/2017 Performance Monitor Data Delivered analytics will only get you so far Take time to understand the data model Write your own analytics Data stored in PSPMTRANSHIST PSPMEVENTHIST Archived data in PSPMTRANSARCH PSPMEVNTARCH UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Custom Analytics with Excel
Draft Presentation 01/04/2017 Custom Analytics with Excel Write a query to extract the data Create ODBC source using Oracle’s ODBC driver Use MSQuery to extract the data directly via into Excel workbook Possibly directly into a pivot table Chart the data When you update the data the chart will also update. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Application Server Processes by Service over a week
Draft Presentation Application Server Processes by Service over a week 01/04/2017 UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Performance Monitor Further Reading
Draft Presentation 01/04/2017 Performance Monitor Further Reading Performance Monitor PeopleBook PeopleSoft Performance Monitor Red Paper Doc ID: PeopleSoft for the Oracle DBA Chapter 10 Practical Guidance on the Use of PeopleSoft Performance Monitor UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

PeopleTools Instrumentation for Oracle Database
Draft Presentation 01/04/2017 PeopleTools Instrumentation for Oracle Database UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Oracle DBMS: Module & Action
Draft Presentation 01/04/2017 Oracle DBMS: Module & Action Enterprise Manager / Grid Control Active Session Hstory (ASH) Oracle Extended Trace From 10g set trace on Module/Action Oracle Resource Manager UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Instrumentation for Oracle Database up to PeopleTools 8.49
Draft Presentation 01/04/2017 Instrumentation for Oracle Database up to PeopleTools 8.49 DBMS_APPLICATION_INFO Sets a Session Attribute CLIENT_INFO since PT7.53 Useful for auditing with database triggers Module and Action not set Module defaults to program name This is hugely significant, because you start to tie PeopleSoft instrumentation to Oracle analysis tools and you can easily track back into specific areas of the product from standard database performance tuning tools. The module and action recorded on v$session turn up in many different places. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Oracle DBMS: Module & Action
Draft Presentation 01/04/2017 Oracle DBMS: Module & Action Oracle + PeopleSoft instrumentation lets you work out database response time in application. Challenge: relating a piece of SQL to a step in a PeopleSoft process Identification Comments in code UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

PeopleTools 8.50 Instrumentation for Oracle Database
Draft Presentation 01/04/2017 PeopleTools 8.50 Instrumentation for Oracle Database MODULE and ACTION On-Line Module/Action Component/Page, Query Name, Subscription Message Client ID = Operator ID Scheduled Processes But only sets ACTION to PRCSNAME This is hugely significant, because you start to tie PeopleSoft instrumentation to Oracle analysis tools and you can easily track back into specific areas of the product from standard database performance tuning tools. The module and action recorded on v$session turn up in many different places. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Custom PeopleTools Instrumentation
Draft Presentation 01/04/2017 Custom PeopleTools Instrumentation Trigger on PSPRCSRQST Set Module to PRCSNAME Set Acton to Process Instance Or Operator ID & Run Control ID Override PeopleTools 8.50 instrumentation Module = program, Action = Process Name UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

PeopleTools 8.52 Instrumentation for Oracle Database
Draft Presentation 01/04/2017 PeopleTools 8.52 Instrumentation for Oracle Database Application Engine in PeopleTools 8.52 Module = PSAE . program name . session ID Action = Program . Section . Step Overrides custom trigger ASH data for specific process instance using PSPRCSQUE.SESSION_ID_NUM UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Oracle RDBMS:Active Session History
Row Migration can Aggravate Contention on Cache Buffer Chains Latch 01/04/2017 Oracle RDBMS:Active Session History Samples active sessions every second Circular buffer in memory v$active_Session_history It should hold about 1 hour of data 1 in 10 samples stored in database DBA_HIST_ACTIVE_SESS_HISTORY Flushed out during AWR snapshot Active Session History (ASH) was introduced in Oracle 10g. It samples the activity of each active database session every second. The data is held in a buffer in memory in the database. The design goal is to keep about an hour (your mileage will vary). If a session is not active it will not be sampled. The in-memory buffer is exposed via a view called v$active_session_history. You could sort of simulate some of ASH by taking a snapshot of v$session every session, but the overhead would be prohibitive. ASH is built into the Oracle kernel, so its overhead is minimal. When an AWR snapshot is taken, 1 row in 10 from the ASH buffer is copied down into the AWR repository. It can also be flushed to disk between snapshots when the buffer reaches 66% full, so there is no missed data.The data is stored in WRH$_ACTIVE_SESSION_HISTORY and it is exposed via DBA_HIST_ACTIVE_SESS_HISTORY. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2011 Go-Faster Consultancy Ltd.

01/04/2017 Licensing ASH is a part of the Diagnostics Pack only available with Enterprise Edition of Oracle database. That’s means it costs money. I don’t like it either, but that is how it is! ASH is enabled by default, but before you rush off to use it, be aware that it is a licenced feature. It is part of the Diagnostic Pack, so you have to pay for it. I don’t like that either, but that’s how it is. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2011 Go-Faster Consultancy Ltd.

01/04/2017 ASH in OEM You can run ASH reports via EM Of course, OEM provides a way to run ASH reports, and here you see I have picked a particular time window, and I have specified a module name – in this case the main payroll calculation process. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2011 Go-Faster Consultancy Ltd.

01/04/2017 Example ASH Report These processes were responsible for 86% of total DB activity Average 14.8 active sessions (out 32 processes) If I go on I get SQL statements But I don’t get execution plans. And this is great. The report is easy to produce, and it tells you lots of things. Which SQL statements are consuming the most time, which objects have the most I You can see in this example I picked a module that was responsible for 86% of the total, and there were an average of 14.8 active sessions (I know there were 32 concurrent processes). But, you don’t get execution plans, and for that you will need to dig deeper yourself, and learn to use the DBMS_XPLAN package. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2011 Go-Faster Consultancy Ltd.

01/04/2017 What does ASH retain? Most of the columns are on v$session Session Session ID and serial, query coordinator Wait event id, name and parameters SQL SQL_ID, plan hash, opcode Plan line numbers from 11g Object object, file and block numbers row numbers from 11g Application module, action, client_id … Most of the columns on v$active_session_history are taken directly from column of the same name on v$session, some have different name, and there is some additional information that is not available elsewhere. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2011 Go-Faster Consultancy Ltd.

Active Session History
Draft Presentation 01/04/2017 Active Session History Query ASH repository directly DBA_HIST_ACTIVE_SESS_HISTORY Profile DB Time by Module / Action SQL_ID SQL Plan Hash Value (if lots of different literals) UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

01/04/2017 Background Reading Sifting through the ASHes, Graham Wood The ASHes of (DB) Time, Graham Wood UKOUG 2009 Video of presentation at MOW2010 Doug Burns’ Oracle Blog Introduction to DBMS_XPLAN Sifting through the ASHes, Graham Wood, Oracle ( The ASHes of (DB) Time, Graham Wood at UKOUG2009 (and other conferences). Introduction to DBMS_XPLAN ( UKOUG2008 UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2011 Go-Faster Consultancy Ltd.

Further Reading Practical use of Active Session History
Draft Presentation 01/04/2017 Further Reading Practical use of Active Session History With examples drawn from PeopleSoft #Practical_ASH.ppt UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Traces PeopleTools SQL Trace Profiler Application Server Log
Draft Presentation 01/04/2017 Traces PeopleTools SQL Trace Profiler PSTraceViewer Unsupported TraceMagic Doc Application Server Log LogFence=4 Diagnostic Large Files Difficult to extract performance metrics Timing Inaccuracies Incomplete Measurement Intrusion UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Batch Timings Reports Application Engine and COBOL
Draft Presentation 01/04/2017 Batch Timings Reports Application Engine and COBOL Time spent in each step AE can write this to database Can get negative numbers Only ever one, can get around this with some arithmetic. Fixed in PT8.53? UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Web Server Access Log Logs every request Large files
Draft Presentation 01/04/2017 Web Server Access Log Logs every request Large files Time taken for every request Client IP address But it could just be a network component not the actual client Oracle Doc UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

There are only two kinds of performance problems
Draft Presentation 01/04/2017 There are only two kinds of performance problems You are working too hard. Consuming resource CPU Disk Memory You are being prevented from working Queuing Database Locking Tuxedo Queuing CPU overload … Waiting for somebody else working too hard! UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

0345f1003x It is generally better to queue higher up the stack.
The impact of queuing in lower tiers can propagate across the system through mechanisms in higher tiers. Protect lower tiers for overload by correct configuration But do explain this to DB/OS/disk admins! Browser (presentation & JavaScript) Screen Paint Java Script Web Server (presentation logic) PIA Servlet Application Server (application logic) APPQ PSAPPSRV DBMS (application data & meta-data http / https Tuxedo Message SQL Eg. At database lock (perhaps held by a batch process) might cause on-line processes to wait. As each on-line users join the queue of locks, so another PSAPPSRV process is stuck in a busy state. After the tuxedo service timeout the process will be killed by tuxedo and the user will get an error. Monitored System UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Good Queuing! While it is best not to have to queue at all
Draft Presentation 01/04/2017 Good Queuing! While it is best not to have to queue at all better to queue on the APPQ in the application server domain, than run out of CPU and queue on the operating system run queue. Use Process Scheduler configuration limit the number of concurrent processes prioritise processing of queued processes This can result in increased system throughput UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Cost-Based Optimizer Statistics
Draft Presentation 01/04/2017 Cost-Based Optimizer Statistics UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Cost-Based Optimizer Statistics
Draft Presentation 01/04/2017 Cost-Based Optimizer Statistics Performance of SQL is a significant aspect of any OLTP system PeopleSoft is not an exception to this rule. All SQL databases* use volumetric statistics to make SQL Optimizer determine the ‘best’ execution plan. ⇒ You need to get your statistics right *at least the ones on which PeopleSoft is certified! But I am only going to talk about Oracle RDBMS UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

How should you collect Optimizer Statistics in Oracle?
Draft Presentation 01/04/2017 How should you collect Optimizer Statistics in Oracle? Maria Colgan and others – Oracle Database Optimizer: An Insider’s View of How the Optimizer Works (Wednesday) Oracle Database Optimizer: Harnessing the Power of Optimizer Hints (Tuesday) Carlos Sierra How to Create in 5 Minutes a SQL Tuning Test Case Using SQLTXPLAIN (Mon) How to Improve SQL Performance with the New SQL Health Check Tool (Tuesday – actually now!) Jonathan Lewis - Creating Test Cases (Monday) B-Tree Indexes (Mon) Christian Antognini - How the Query Optimizer Learns from Its Mistakes (Tuesday – also now!) Peter Brink Cardinality Feedback UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

How should you collect Optimizer Statistics in Oracle?
Draft Presentation 01/04/2017 How should you collect Optimizer Statistics in Oracle? Tell the Optimizer the truth about your data. Or at least as much of the truth as it needs to make the right answer. Making it handle too much of the truth can be prejudicial without being probative. Data in Table has a lifecycle Create / Change / Delete data, Collect Statistics Use it UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

What does Oracle do by default?
Draft Presentation 01/04/2017 What does Oracle do by default? Oracle collects statistics in the maintenance window 10pm-6am weekdays, and weekends Collects statistics on stale objects Automatically determines optimal sample size for collecting statistics Automatically determines whether data is sufficiently skewed to require histograms. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Implications for PeopleSoft of Default Oracle Behaviour
Draft Presentation 01/04/2017 Implications for PeopleSoft of Default Oracle Behaviour PeopleSoft is parse intensive. Literal values in dynamic SQL Non-use of ReUseStatement in Application Engine %ProcessInstance, %Oprid and similar macros always resolve to literals Temporary Record corresponds to many tables. Non-sharable SQL Skewed Data Default values to avoid NULL 0, single space, 1st January 1900. Excessive histogram generation Processing histograms adds to both statistics collection and parse overhead. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

New Optimizer Features in 11g
Draft Presentation 01/04/2017 New Optimizer Features in 11g Table Preferences Default dbms_stats options at table level CASCADE DEGREE ESTIMATE_PERCENT However, there are new features that only work properly with automatic sample size. GRANULARITY INCREMENTAL METHOD_OPT Can control histogram collection STALE_PERCENT Control when statistics are refreshed Use these, and you can safely use standard, default approaches to maintaining statistics with PeopleSoft. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

%UpdateStats macro DDL Model calls dbms_stats.gather_table_statistics
Draft Presentation 01/04/2017 %UpdateStats macro DDL Model calls dbms_stats.gather_table_statistics In 11g, can remove most parameters and use table preferences to control the behaviour of DBMS_STATS. This applies to whenever statistics are collected. There is still a case for intercepting calls from %UpdateStats to dbms_stats with custom PL/SQL package. Some processes make excessive use of %UpdateStats eg. TL_TIMEADMIN Statistics collection can be significant overhead during batch process Sometimes a process may call %UpdateStats on a permanent table. If that table is large then incremental difference small. Might only want to collect statistics if sufficient change, ie statistics are STALE. Fine grain control over %UpdateStats can be very useful. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Warning: Statistics History
Draft Presentation 01/04/2017 Warning: Statistics History NB: Default Oracle Statistics History Retention policy is 31 days. Global policy – can specify at table level Every %UpdateStats call generates history UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

pscbo_stats – 1322888.1 It is an attempt to solve a genuine problem
Draft Presentation 01/04/2017 pscbo_stats – It is an attempt to solve a genuine problem It has continued to evolve since its initial release However, it is a 10g solution. It does not attempt to manage 11g table preferences Uses excessively high sample sizes Too many buckets for histograms Fails to recognise all PS objects are NOPARALLEL No partition support 11g Incremental Statics Synopses on partitioned objects only work properly on default sample size Based upon my experience of PeopleSoft on Oracle: I think better effects can be achieved with better choices for histograms and numbers of buckets and default sample size. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Draft Presentation 01/04/2017 GFCPSSTATS11 Package Hold table preferences on meta-data table keyed on RECNAME DDL Trigger to set table preferences as table created DML Trigger to set table preferences when metadata changed. DDL model also calls GFCPSSTATS11 package Via meta-data, can suppress call to dbms_stats Or only collect statistics if ‘stale’. So, no PeopleSoft application code change required. Force collection of locked statistics Catch exception when attempt to update locked statistics UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Draft Presentation 01/04/2017 Temporary Records No point collecting statistics during maintenance window. Truncated at start of AE But Truncate doesn’t clear statistics!!! Can only collect accurate statistics during process %UpdateStats macro Recommendation: Lock and Delete Optimizer Statistics OPTIMIZER_DYMANIC_SAMPLING=4 UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Global Temporary Tables?
Draft Presentation 01/04/2017 Global Temporary Tables? Application Engine Temporary table instances State Records COBOL Regular Table Record Consider using Oracle Global Temporary Table Bot can’t do this for restartable AEs UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

MOS Document References
Draft Presentation 01/04/2017 MOS Document References SQLT – [ID ] Useful for detailed performance analyses of individual SQL statements OS Watcher Black Box – [ID ] Automates collection of OS-Level performance and diagnostic metrics Extensible scripting can include application-specific information. (stay tuned) UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Further Reading http://blog.psftdba.com www.go-faster.co.uk
Draft Presentation 01/04/2017 Further Reading E-ORA Advice for the PeopleSoft Oracle DBA [ID ] Although I have some reservations about some of the content! UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Tuxedo Application Server Sizing
Draft Presentation 01/04/2017 Tuxedo Application Server Sizing UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Application Server Sizing
Draft Presentation 01/04/2017 Application Server Sizing How many server processes should I configure? It is very hard to answer that question with a simply numeric answer. Not too many. Not too few. It depends! UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Draft Presentation 01/04/2017 Application Server Sizing Too Few? Queuing All servers busy Too Many? Run out of memory/CPU Paging to disk Queue on run queue >= 10 per queue IPC Queue Contention Overload database causing Application Server to back up The trouble is you can get both conditions! UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Detecting Application Server Queuing
Draft Presentation 01/04/2017 Detecting Application Server Queuing Queuing not reported in PPM due to bug Look instead at application server process service status Event 302 spawning. Following chart derived from this data Interrogate Tuxedo domain directly with tmadmin CLI with shell/batch script. See UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Application Server Processes by Service over a week
Draft Presentation Application Server Processes by Service over a week 01/04/2017 UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Application Server Spawning
Draft Presentation 01/04/2017 Application Server Spawning Spawning for enabled when Min servers<Max servers For PSAPPSRV, PSQRYSRV, PSBRKHND, PSSUBHND, PSPUBHND Idea is to adjust number of servers dynamically to meet the demand Spawning gets a bad press because Number of Min/Max servers not correctly configured Fallacious belief that application server queuing is universally bad It isn’t. It can be the least worst option. Excessive spawning/recycling is a problem UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Recycle Count PSAPPSRV PSQRYSRV Further Reading
Draft Presentation 01/04/2017 Recycle Count PSAPPSRV Historically, the delivered value has varied between 1000 and in different versions of PeopleTools It used to be used as a coarse method of controlling memory consumption of application server processes More effective to use MaxCacheMemory >= 500Mb PSQRYSRV Entire query results sets copied into memory A lower recycle count can be used to release memory Particularly on Windows Further Reading Document ID UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Draft Presentation 01/04/2017 Application Server Sizing Avoid creating things that you don’t need! PSSAMSRV – Only one or two are ever needed. PSQCKSRV – Only used by for 3-tier Application Designer sessions (and then only optional) Ensure enough connections are available Turning off Jolt Pooling will require more JSHs Understand Application Server Memory Usage My Oracle Support - Document ID Tim – Added UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Avoiding Contention Tuxedo Queues
Draft Presentation 01/04/2017 Avoiding Contention Tuxedo Queues Contention on IPC queues occurring if messages on queue and not all servers busy Recommendation No more than 10 server processes/queue Create multiple queues in a single domain Rather than multiple domains each with a single queue Need to manually set Bulletin Board sizing parameters in psappsrv.ubx file. UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Unix IPC Message Sizing
Draft Presentation 01/04/2017 Unix IPC Message Sizing (This affects windows too!) PeopleSoft Messages to Tuxedo typically around Kb application dependent – so you need to measure this for your system Default message size 64Kb (except AIX) Messages larger than max message size (default ¾ queue size) written to and read from disk Messages that ¾ fill queue also written to disk UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Practical PPM Practical PPM select * from ( select jbs.pm_systemid , jbs.dbname , jbs.pctile , jbs.jolt_bytes_send , jbr.jolt_bytes_rcvd from ( select pm_systemid, dbname, pctile, min(jolt_bytes_send) jolt_bytes_send select pm_systemid, dbname, jolt_bytes_send , ntile(100) over (partition by pm_systemid, dbname order by jolt_bytes_send) as pctile from dmk_trans_anal ) group by pm_systemid, dbname, pctile ) jbs full outer join ( select pm_systemid, dbname, pctile, min(jolt_bytes_rcvd) jolt_bytes_rcvd select pm_systemid, dbname, jolt_bytes_rcvd , ntile(100) over (partition by pm_systemid, dbname order by jolt_bytes_rcvd) as pctile ) jbr on jbr.pctile = jbs.pctile and jbr.pm_systemid = jbs.pm_systemid and jbr.dbname = jbs.dbname order by 1,2,3 UKOUG Performance Tuning for PeopleSoft Administrators ©2012 Go-Faster Consultancy Ltd.

Unix IPC Message Sizing
Draft Presentation 01/04/2017 Unix IPC Message Sizing General Recommendation Set queue size 256Kb Although, this will consume more shared memory. ipcs shows queues that Tuxedo PrintQueue doesn’t! Set maximum message size 128Kb Large PS/Query results will still ping to disk and there is nothing you can do about it UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Application Server – Integration Broker Considerations (Asynch)
Draft Presentation 01/04/2017 Application Server – Integration Broker Considerations (Asynch) Isolate PUB/SUB processes by dedicating one or more domains to IB. DO NOT allow multiple domains to concurrently process the same service operations For domain failover, use IB failover groups For further isolation, use dedicated dispatchers To Scale, use slave or template slave dispatchers PeopleSoft Integration Performance and Tuning for PeopleTools 8.50 [ID ] ©2012 Go-Faster Consultancy Ltd.

Integration Broker Monitoring
Draft Presentation 01/04/2017 Integration Broker Monitoring Integration Broker Service Operations Monitor IB Profiling – runtime performance detail for both Synch and Asynch operations Enable at both the Integration Gateway and Application Server Navigation: PeopleTools->Integration Broker->Service Operations Monitor->Statistics Tim - Added UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Further Reading PeopleSoft for the Oracle DBA Tuxedo Internals
Draft Presentation 01/04/2017 Further Reading PeopleSoft for the Oracle DBA Chapter 13 Tuxedo Internals This presentation from 2002 is based on PT7.5x UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Web Server Characteristics
Draft Presentation 01/04/2017 Web Server Characteristics Primary purpose is serialization/deserialization to build and serve presentation to browsers Maintains user session state Tuning mostly involves correctly sizing the JVM. Scale horizontally by adding web instances Tim - Added UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Web Server Sizing Tuning goal is to properly size JVMs, allowing for enough user sessions without extensive GC pauses. JVM size is set with Java start-up options (-Xmx, -Xmn) Try to keep JVM size between 512M and 2048M depends on which JRE Watch out for native heap -Xss to reduce thread stack size if necessary Native heap limitations largely a non-issue in PT 8.51+ 64-bit addressing Use verbose GC logging to monitor the Garbage Collector UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Draft Presentation 01/04/2017 Web Server Logging GC logging (-verboseGC –XX:+PrintGCDetails -XX:+PrintGCTimeStamps) Provides timing and frequency of Java VM Garbage Collection cycles Useful for evaluating Java Heap sizing under load. Tim - Added UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Web Server Configuration
Draft Presentation 01/04/2017 Web Server Configuration Timeouts can drastically affect Web Server performance and scalability Web Profile Cache settings and Timeouts are important Saved States, State Discard Interval, Home Page Stale Interval Consider a dedicated Web Instance for Integration Broker Gateway Things like session affinity and Http keepalives don’t matter as much as for the IB Gateway. Tim - Added UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Further Reading Web/Network Infrastructure
Draft Presentation 01/04/2017 Further Reading Web/Network Infrastructure E-PIA: Red Paper On Implementing Clustering And High Availability For PeopleSoft [ID ] Tim Added - UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

Conclusion It depends…
Draft Presentation 01/04/2017 Conclusion It depends… UKOUG Performance Tuning for PeopleSoft Administrators ©2012 ©2012 Go-Faster Consultancy Ltd.

You know my methods, apply them.
A Study in Scarlet, Arthur Conan-Doyle Last slide - More Sherlock Holmes. UKOUG Performance Tuning for PeopleSoft Administrators ©2012

Performance Tuning for PeopleSoft Administrators

Similar presentations

Presentation on theme: "Performance Tuning for PeopleSoft Administrators"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Performance Tuning for PeopleSoft Administrators

Similar presentations

Presentation on theme: "Performance Tuning for PeopleSoft Administrators"— Presentation transcript:

Similar presentations

About project

Feedback