Presentation is loading. Please wait.

Presentation is loading. Please wait.

PeopleSoft Performance Tuning at ab

Similar presentations


Presentation on theme: "PeopleSoft Performance Tuning at ab"— Presentation transcript:

1 PeopleSoft Performance Tuning at ab
Technology Session Number 610 October 23, Performance Tuning at UBS AG

2 About UBS Group UBS is a global, integrated investment services firm and the leading bank in Switzerland: UBS Wealth Management & Business Banking consists of the world's leading private banking and Swiss retail banking business UBS Global Asset Management is a leading institutional asset manager and mutual fund provider. UBS Warburg operates globally as a client-driven securities, investment banking & wealth management firm UBS PaineWebber is one of the top US wealth managers

3 All users via PIA Self-Service
Project Overview HRMS 8 Global Payroll Swiss Extension 40000 employees (34000 payroll) > 200Gb Data 400 HR CRM users All users via PIA Self-Service Performance Tuning at UBS AG

4 HP/UX11, MC ServiceGuard, EMC Oracle 8.1.7.2.1, PT8.15, Apache
Configuration Production HP V Class, 2x20CPU, 24/10Gb HP/UX11, MC ServiceGuard, EMC Oracle , PT8.15, Apache

5 Large HRMS implementation Lots of customisations
Challenges Large HRMS implementation Lots of customisations Payroll is a ‘financial’ batch Oracle bugs Performance Problems 35000 employees is a large HRMS implementation, but by no means the largest. Nonetheless, we have done work on the performance of the security views. This system has been heavily customised, and runs in 3 languages. HR doesn’t really have much by way of a batch process. However, the payroll has all the characteristics of a financial batch program. We have had to contend with a number of Oracle Bugs, one of which was a major driver in the upgrade to Oracle and was a significant factor in the poor performance of the 7.5 payroll system Performance Tuning at UBS AG

6

7 What is Performance Tuning? On-Line Batch
Agenda What is Performance Tuning? On-Line Batch What did we achieve at UBS? Questions? Independent consultant specialising in the performance tuning of PeopleSoft systems. I’ve spent 13 years working with Oracle databases, the last 6 of which have spent working with PeopleSoft. In this presentation I am going to discuss what I mean by ‘Performance Tuning’, and what I believe is the correct to approach it. Then I am going to get into some technical detail. I will handle on-line and batch processing separately. I will talk about the kind of performance metrics that you need to watch in order to work out what has to be tuned, Then I will round off with a description of what was achieved at UBS. I was a part of the project IT team at the UBS Swiss Payroll project in Basel. Part of what we did was ensure the performance of the system met some fairly stringent targets. The technical stuff is not just theory - we did it at Ubs, and I’ve done it elsewhere - you can do it too. Performance Tuning at UBS AG

8 If you can’t hear me - say so now!
Rules of Engagement If you can’t hear me - say so now! Feel free to ask on-topic questions during the presentation. This presentation (with notes) is available on the conference website (after the conference) and on I have to put this into a slide otherwise I forget to say it. Performance Tuning at UBS AG

9 Aphorism Performance is exactly what the user perceives it to be. No more, no less. I have realised that over the years that I have collected a number of catchphrases and quotes. When you are tackling performance problems you need to keep an eye on what you are trying to achieve. Often a user will report that the system is slow, so you have measure what the use is experiencing The availability and response time that a system produces are sum of all the components in the chain. Database - Application Server - Network - PC It takes time for requests to be transmitted over a network, through application server, be processed by the database and returned. There is a certain amount of processing on the client and that takes time too. If any component fails then the system fails. If you go to a cash point and you cant draw money, you don’t care whether the central mainframe is down, or the network, or the cash point itself is down, or even if it have run out of money. The bottom line is that the system is not available, and you don’t get your money. The users can only observe one thing. Their own PC Performance Tuning at UBS AG

10 Aphorism Poor performance is when the user’s perception does not match their expectation. One of the problems with performance is that it can all be a bit touchy feely. Users will only ever say two things. 1) The system is slow 2) The system is OK You need some sort of metric. Performance Level Agreements as a part of the SLA. The only thing that means anything to a user is time. How long do they spend waiting for the hourglass? Perhaps it also means that there is another component of your system to tune - your users Performance Tuning at UBS AG

11 Performance Tuning is a search for lost time.
Aphorism Performance Tuning is a search for lost time. Where are you loosing how much of your time? In order to improve the performance of a system (and by system I mean anything not just a computer system) you have to understand how it work. Performance Tuning at UBS AG

12 Aphorism When you have eliminated the impossible, whatever remains, however improbable, must be the truth. A Study in Scarlet, Arthur Conan-Doyle Answering that questions is not trivial. Because when you do the analysis properly and you may get an unexpected answer. The answer might be counter-intuitive, it may be so peculiar that you might have rejected it out of hand. This is experimental science I’ve had a presentation will all this Sherlock Holmes stuff for several years. And I put some of it into this presentation a long time ago now. Just last week ( ) the Royal Society of Chemistry (who are a serious bunch of scientists) very publicly awarded a gold medal to Holmes (and not his creator Conan-Doyle) because of his contributions to science. So there is a serious point here, and I am not simply padding out this presentation. Detection is, or ought to be, an exact science. It should be be treated in the same cold and unemotional manner. The Sign of Four, Arthur Conan-Doyle Performance Tuning at UBS AG

13 PIA (On-Line) Performance
Architecture brief overview Sources of Metrics Graphical Techniques Tracing Techniques So we are looking for ways to measure time I am going to take the on-line side of PeopleSoft, the PIA first. You are going to see that V8 provides many different metrics, which tell you different things about different parts of the technology, so I will start with a very high-level overview of the technology. Performance Tuning at UBS AG

14 Tuxedo Application Server (application data & meta-data
PIA Architecture PIA is 4-tier architecture Each component can contribute to response time Need to measure each component Show of hands - how many of you are on V8.x? In the beginning PeopleSoft was a two-tier system. A windows client interacted directly with the database. Today on the application designer and data mover still do that. In 1997 PeopleTools 7.0 introduced a full three-tier windows client that used the Tuxedo Application server. In 2000 PeopleTools 8.x appeared with a fully internet client. The database still stores the application and the application server is responsible for extracting the relevant application objects from the database as well as the application data. That hans’t really changed. The application server bundles the component (the thing we used to call the panel group) will all the html, JavaScript fragments and graphics and send that to the servlet. The servlet, which is closely bound with the webserver unpacks that message, sends the page to the browser and rest of those objects are loaded by the browser from the webserver. The Servlet also holds session information for each connected user. There is a lot of work going on at different tiers in the chain. Depending on circumstances different componenets can contribute differently to responsetimes. Our objective is to measure all of this. (see for detailed PIA/Tuxedo architectural overview) Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data SQL Tuxedo Message http / https Performance Tuning at UBS AG

15 Browser Web Server BEA/Tuxedo Database (Oracle) 3rd Party software
Sources of Metrics Browser 3rd Party software Local agent on PC Proxy Server Web Server Access Log BEA/Tuxedo Service Trace TMADMIN Database (Oracle) SQL_Trace PSFT Application Server Trace There are lots of things to measure in the PeopleTools 8. Each tier in the diagram on the pervious slide produces its own set of metrics. The key is to relate different sets. In the light of what I have just said about perception, the most important place to measure is as close to the client as possible. That can be difficult. There is 3rd party software that will put agents on the clients. Page detailer is a part of the Websphere suite. One technique that I have used is to introduce a proxy server. Most webservers can be configured as proxy servers. So I put a proxy server physically close to the clients that I want to measure. Similarly the web server can also record all the same requests but now their duration will be the time at the webserver. This time that the request took to actually process. The difference in these duration is the time lost across the network. Performance Tuning at UBS AG

16 Standard format for access log
Web/Proxy Access Log Standard format for access log (Apache is different - see manual) Fields Date Time Time Taken accurate to OS time units (1/100ths or 1/1000ths) only 1s on Apache Message bytes IP addresses & DNS names URI stem & Query The access log will record all the requests including their duration as measured at the proxy server. There is a standard format for webserver logs. Weblogic conforms to this stanard, Apache does not. There are a number of things that you can records. Performance Tuning at UBS AG

17 How to Enable Weblogic Trace
#weblogic.properties #required weblogic.httpd.enableLogFile=true weblogic.httpd.logFileName=access.log weblogic.httpd.logFileFormat=extended #recommended weblogic.httpd.logRotationType=date weblogic.httpd.logRotationPeriodMins=1440 weblogic.httpd.logRotationBeginTime= :00:00 PeopleTools 8.4 has weblogic server The configuration file is a lttle different. It is now an XML file. You can edit it, but there is now a much improved console to control it. Performance Tuning at UBS AG

18 Sample Weblogic Access Log
#Version: 1.0 #Fields: date time time-taken bytes c-ip c-dns cs-method sc-status cs-uri-stem cs-uri-query :51: go-faster-1 GET 200 /servlets/iclientservlet/peoplesoft8/signon.html - :51: go-faster-1 GET 200 /peoplesoft8/signin.css - :51: go-faster-1 GET 200 /peoplesoft8/images/secondaryPSFTLogo.gif - :51: go-faster-1 POST 200 /servlets/iclientservlet/peoplesoft8/ ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=PERSONAL_DATA&RL=&target=main2 And this is what a sample trace looks like Look at the first two lines. These look like comments, but they actually determine the format of the access log. You create this two lines yourself as a seed logfile, and when the server restarts it reads these, and the format of the logfile that follows is determined. Performance Tuning at UBS AG

19 How to Enable Apache Access Log
#httpd.conf LogFormat "%{%Y.%m.%d %H:%M:%S}t|%T|%B|%u|%h|%{User-Agent}i|%>s|%m|%U|%q" monitoring CustomLog logs/access.log monitoring But apache is no longer supported in 8.4 Performance Tuning at UBS AG

20 Sample Apache Access Log
:57:06|0|275|-| |Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|GET|/peoplesoft8/cache/PT_NEXTTAB_ENG_1.gif| :57:17|0|31847|-| |Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA :57:26|0|30959|-| |Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|POST|/servlets/iclientservlet|?ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&PanelGroupName=JOB_DATA Pipe symbols have been used as separators because it’s very unlikely to appear elsewhere in the trace. Performance Tuning at UBS AG

21 What does the access log give us?
A date and time for every request Who made the request? How long did it take to serve? What was requested? Which component (panel group)? So to summarise the enhanced access log will tell you about activity at the web or proxy server. Everything you need to know about a request. Performance Tuning at UBS AG

22 BEA/Tuxedo Service Trace
Edit psappsrv.ubx file CLOPT=“-r -e APPQ.stderr…” -r enables trace to stderr file -e qualifies name of stderr file The Tuxedo service trace will log every service that goes via the application server. There is almost, but not quite, a one-to-one relationship between web server and app server requests. The difference between web server service time and app server service time is the time spent by the servlet on the JVM. Using SQL trace it is possible to aggregate SQL time over a period. You can determine the proportion of SQL time in Tuxedo time. Performance Tuning at UBS AG

23 Add a new variable to the trace section
Edit psappsrv.cfg Add a new variable to the trace section Thus service trace can be enabled/disabled from configuration file New variable will appear in the configuration dialogue [Trace] TuxedoServiceTrace=-r So that the service trace can be controlled easily I usually add a new variable in the trace file. Performance Tuning at UBS AG

24 Change every CLOPT entry in psappsrv.ubx. File name is set manually.
Edit psappsrv.ubx Change every CLOPT entry in psappsrv.ubx. File name is set manually. Same as queue name RQADDR="APPQ" CLOPT="{$Trace\TuxedoServiceTrace} -e {LOGDIR}{FS}APPQ.stderr {$PSAPPSRV\Spawn Server} -- -C {CFGFILE} -D {$Domain Settings\Domain ID} -S PSAPPSRV" Now customise the ubx fileto use the new variable. LOGDIR is a delivery variable that is the location of the LOGS directory. Thus the stderr logs go to same directory as the other log files. I generally make the name of the stderr file the same as the queue so I can easily distinguish between queues Performance Tuning at UBS AG

25 Resultant psappsrv.ubb
When domain is configured .ubb file is generated. All variables are fully resolved. CLOPT=”-r -e D:\ps\hr8\appserv\hr8d\LOGS\APPQ.stderr -p 1,600:3,10 -sICQuery -sSqlQuery:SqlRequest -- -C psappsrv.cfg -D HR8D -S PSAPPSRV" When you configure a domain the variable names are fully resolved in .ubb. So it’s a good place to check that the domain is correctly reconfigured. Performance Tuning at UBS AG

26 Sample Tuxedo Service Trace
Each service called to domain is logged SERVICE PID SDATE STIME EDATE ETIME @ICScript @ICScript @ICScript @ICScript @ICScript @ICScript @ICScript @ICPanel @ICPanel @ICPanel The trace is a text file,which is easy to work with. Performance Tuning at UBS AG

27 Service Pid Sdate/Edate Stime/Etime Name of Service
Tuxedo Service Trace Service Name of Service Pid An OS PID of listener or handler Sdate/Edate Start/End Date in Seconds since 00:00hrs GMT 1st January 1970. Stime/Etime Start/End Time in OS time units (1/ 100ths or 1/1000ths) Hence calculate accurate service duration Time on Server - not queuing The fields in the service trace Performance Tuning at UBS AG

28 SERVICE SUMMARY REPORT
Txrpt Utility SERVICE SUMMARY REPORT SVCNAME a-10a TOTALS Num/Avg Num/Avg ICScript / /1.43 ICPanel / /3.47 JavaMgrGetObj 1/ /0.04 GetCertificate 1/ /0.73 TOTALS / /2.06 TXPRT utility shipped by BEA with tuxedo to analyse the trace - gives hourly figures. Performance Tuning at UBS AG

29 BEA/Tuxedo TMADMIN Utility
Interactive command line interface BEA/Tuxedo utility not PeopleSoft Wrapped inside psadmin Can be included in scripts -r for read only in monitoring scripts pq (printqueue) psr (printserver) pclt (printclient) Performance Tuning at UBS AG

30 How many servers on queue How many requests queued
PQ What queues in domain How many servers on queue How many requests queued (How much work on queue) Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine JSL.exe GO-FASTER+ JREPSVR.exe GO-FASTER+ PSSAMSRV.exe SAMQ GO-FASTER+ BBL.exe GO-FASTER+ WSL.exe GO-FASTER+ PSAPPSRV.exe APPQ GO-FASTER+ Performance Tuning at UBS AG

31 How many requests have they handled? What are they doing right now?
PSR What servers exist? How many requests have they handled? What are they doing right now? Prog Name Queue Name Grp Name ID RqDone Load Done Current Service BBL.exe GO-FAST ( IDLE ) PSAPPSRV.exe APPQ APPSRV ICPanel PSSAMSRV.exe SAMQ APPSRV ( IDLE ) WSL.exe BASE ( IDLE ) JSL.exe JSLGRP ( IDLE ) JREPSVR.exe JREPGRP ( IDLE ) Performance Tuning at UBS AG

32 Are they doing something? When did they connect?
PCLT Who is connected? Are they doing something? When did they connect? LMID User Name Client Name Time Status Bgn/Cmmt/Abrt GO-FASTER NT WSH :21:14 IDLE 0/0/0 GO-FASTER NT JSH :21:13 IDLE 0/0/0 GO-FASTER NT tmadmin :00:00 IDLE 0/0/0 GO-FASTER PS JavaClient :02:30 BUSY/W 0/0/0 Performance Tuning at UBS AG

33 Find long running SQL statements Total SQL Time for session
Oracle SQL Trace Tuning tool Session trace Format it with TKPROF Find long running SQL statements Total SQL Time for session call count cpu elapsed disk query current rows Parse Execute Fetch total Performance Tuning at UBS AG

34 Tuxedo Application Server (application data & meta-data
Location of Metrics Tuxedo tmadmin script Tuxedo Service Trace Proxy Server Access Log Oracle SQL*Trace Webserver Access Log So now you can use pairs of measures to measure the segment between the points of measurement The difference between proxy server and web server is network time The difference between Webserver log and Tuxedo Log is servlet time + tuxedo queue time Tuxedo queuing can be estimated from Tuxedo tmadmin scripts. An average queue of 2 requests on a queue suggests a queuing time of twice the average service time Oracle trace can only be used on a statistical basis. What proportion of Tuxedo Service time is SQL Important tip - if these traces are coming off difference servers get them time synchronised at least with each other if not a proper time source. Java Servlet (presentation logic) Tuxedo Application Server (application logic) DBMS (application data & meta-data SQL Tuxedo Message http / https Performance Tuning at UBS AG

35 Load Data into Database
Graphical Analysis Don’t drown in numbers Draw a picture Load Data into Database SQL_Loader Oracle 9i External Table Process Data Matching Aggregation Oracle Analytic Function Load data into Excel Draw Graph My tactic for dealing with almost any log file is the same. Load it into a database, manipulate the data with SQL - you may have to match entries in different logs Performance Tuning at UBS AG

36 Connect Excel directly to a database Aggregate data carefully
Tips about Graphs Connect Excel directly to a database Aggregate data carefully Too little and excel draws it badly because it adds a black border around an area Too much and you lose details Pivot tables are useful but Excel97 buggy Scatters and Trendlines joining dots can obscure data Excel supports points per series Performance Tuning at UBS AG

37 Example: CPU Utilisation
Performance Tuning at UBS AG

38 Example: Apache Servlet Times

39 Longest running http conversations

40 Example: Tuxedo Service Time
Performance Tuning at UBS AG

41 tmadmin - PCLT - Connect Client Sessions

42 tmadmin - PCLT - Busy/Wait Users

43 tmadmin - PQ - Queue Length

44 tmadmin - PSR - print server processes

45 Relating different metrics produces new information
Performance Tuning at UBS AG

46 Sources of Metrics Graph Batch Performance
Performance Tuning at UBS AG

47 Process Scheduler Request Table
Sources of Metrics Process Scheduler Request Table PSPRCSRQST need to archive this table maintain history in another table with on DELETE trigger Application Engine Steps PS_BAT_% tables AE_TRACE=1024 In Process Scheduler configuration (psprcs.cfg) Report to database and/or trace file Cobol Statement Report to trace file SQL Trace Performance Tuning at UBS AG

48 Graphical Analysis Performance Tuning at UBS AG

49 Top AE Statements Performance Tuning at UBS AG AE PC % Cum % % Cum %
Stmt Execute SQL SQL PC SQL SQL Total Total Rank Statement ID Count Time Time Time Time Time Time Time 1 GP_PMT_PREP.CA000.CF000.S 2 PeopleCode.Rowset.Fill GP_RSLT_ACUM 3 GP_PMT_PREP.FA000.FH000.S 4 GP_PMT_PREP.FA000.FC000.S 5 GP_PMT_PREP.FA000.FG000.S 6 GP_PMT_PREP.FA000.FB000.S 7 PeopleCode.Rowset.Fill GP_RSLT_PIN 8 PeopleCode.CNuK_ 9 PeopleCode.CDfF0_PPNw Performance Tuning at UBS AG

50 Oracle SQL*Trace & TKPROF
Enable SQL Trace with Trigger on psprcsrqst.runstatus from !7 to 7 Process trace with TKPROF report Top n statements sort by Parse/Fetch/Execute CPU/Elapsed Time statement timings/execution plan call count cpu elapsed disk query current rows Parse Execute Fetch total Performance Tuning at UBS AG

51 Aphorism The problems change from release to release, but the methods by which they are investigated remain the same. Jonathan Lewis (author of Practical Oracle 8i) Performance Tuning at UBS AG

52 Questions? Performance Tuning at UBS AG

53 You know my methods, apply them.
Aphorism You know my methods, apply them. A Study in Scarlet, Arthur Conan-Doyle I will leave you with a final piece of Sherlock Holmes and my thanks. Performance Tuning at UBS AG

54 PeopleSoft Performance Tuning at ab
Technology Session Number 610 October 23, Performance Tuning at UBS AG


Download ppt "PeopleSoft Performance Tuning at ab"

Similar presentations


Ads by Google