Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein

Similar presentations


Presentation on theme: "1 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein"— Presentation transcript:

1 www.hotsos.comSlide 1 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein (larry.klein@hotsos.com) Hotsos Enterprises, Ltd. Charlotte Oracle Users Group Thursday May 10, 2007

2 www.hotsos.comSlide 2 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Presenter’s Background Larry Klein 3 years, VP of Consulting, Hotsos Enterprises, Ltd. 10 years, Oracle Consulting and Development database and application performance and architecture 17 years, IBM and Candle Corp’s mainframe diagnostics, performance, and capacity planning Patrick Robbins, VP of Sales, Hotsos Enterprises, Ltd.

3 www.hotsos.comSlide 3 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Presentation Agenda Workload and Characterization Concepts Data Sources Case Studies Questions?

4 www.hotsos.comSlide 4 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. What is Workload? The Exercise of Application Functionality for Business Benefit Application Functions Users Oracle E-Business Suite

5 www.hotsos.comSlide 5 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Workload has a “Heaviness” or Cost Application Functions LightHeavy Users Few Workload Light Workload Medium Many Workload Medium Workload Heavy

6 www.hotsos.comSlide 6 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network

7 www.hotsos.comSlide 7 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network Hotsos sees Others do much “Downstream” tuning database ratios init.ora disk speeds …

8 www.hotsos.comSlide 8 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network Downstream efforts often are unfocused, wasted – like rearranging deck chairs on a certain luxury liner At Hotsos we believe there is much to be gained by focusing at the Workload level

9 www.hotsos.comSlide 9 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Tech Stack Workload Drives Resource Utilizations Operating Costs Good or Bad Behavior

10 www.hotsos.comSlide 10 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. http://www.angelsandearthlythings.com/atlas.html Do You Remember Atlas? Greek god Lost battle against Zeus Punished by being made to carry on his shoulders the weight of the heavens So, Now that we Know Workload, What is Workload Characterization?

11 www.hotsos.comSlide 11 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. http://www.angelsandearthlythings.com/atlas.html Poor Atlas… Ouch!!! I’m Really Sore!!!

12 www.hotsos.comSlide 12 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. http://www.angelsandearthlythings.com/atlas.html Atlas Visits a “Traditional Tuning” Doctor… Ouch!!! I’m Really Sore!!! The Flex in your Knees exceeds a comfortable range wear some knee braces The Flex in your Back exceeds a comfortable range wear a back brace

13 www.hotsos.comSlide 13 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. http://www.angelsandearthlythings.com/atlas.html Atlas Visits a “Workload Characterizing” Doctor… Ouch!!! I’m Really Sore!!! That’s a Pretty Big Rock you’ve got there Let’s find a way to do Something about that Rock

14 www.hotsos.comSlide 14 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Workload Characterization? What is it? –The process of finding in an Application System The big rocks Many other very useful, powerful things Why do you care? –The big rocks impact other, more important workloads Resource consumption, contention, latching –Every system has big rocks Wasting resources Causing contention Causing one or more unplanned, unbudgeted hardware upgrades

15 www.hotsos.comSlide 15 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. When a Customer Calls Hotsos for Help What they say “Help, my Whole System is Slow” What they mean “Help, one or more individual workloads is slower than I need it to be” We help them by –Characterizing the workload –Looking for the big rocks –Considering their other, important issues –Putting together a game plan to analyze and fix

16 www.hotsos.comSlide 16 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. What Happens in our Hotsos Consulting work? We use the Hotsos Method R* Listen to the Customer Understand then compile an Issues List –Order Entry form OEXOEMOE takes 60 seconds to book an order, need it to be 5 seconds –Custom report XYZ takes 50 hours to run, need it to be 2 hrs because it’s on the critical path of month end processing –Holy cow, Big Rock DiscoverABC consumed 52% of yesterday’s total database processing costs Trace (10046 level 12) Profile Analyze Test and Resolve *R = Response Time

17 www.hotsos.comSlide 17 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. E-Business Customers Almost Always get into Workload Trouble, Particularly Big Rock Trouble Project Implementation Timeline Buy Software Go Live and Pray Guess #Users, Features, Sizing Order Hardware Map & Gap Install Hardware CRP’s ? More Users, Features! Build 1000 Custom- izations per the Gap, Uncon- strained batch schedule We’re out of time. We don’t need no stinking load testing

18 www.hotsos.comSlide 18 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. As a result, the Initial Server Sizing could be off… Congratulations! Your Project has generated 10 (or maybe 100?) pounds of workload for a 5 pound server

19 www.hotsos.comSlide 19 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. For E-Business Customers We Often find 2 Categories of Issues…

20 www.hotsos.comSlide 20 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Common E-Business Foundation Issues IssueAction Not running certified releases Check Metalink for Certification status Creative init.oraMust follow legal settings (especially CBO) per Metalink Support Note 216205.1 (Under)sizingSee sizing table at end of Note 216205.1 No, inconsistent, or stale statistics Run FNDGSCST or equivalent, be careful of interaction with 10g rdbms GATHER_STATS job No pinningPin at database startup, frequently executed packages, functions, triggers No purging Run FNDWFPR to purge workflow tables Run FNDCPPUR, FNDSCPRG to purge FND tables …

21 www.hotsos.comSlide 21 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Common E-Business Workload Issues IssueAction MicroThe thing I traced is its own problem. There is inefficient SQL or whatever. Go fix the SQL or whatever… MacroThe thing I traced, in and of itself, is good. it’s SQL is efficient on a no-load system it runs great under load it runs poorly who is (are) the culprits – consuming resource – contending with this task The thing I traced is good it’s a batch job the user is complaining about turnaround time runtime actually was quite good waittime before run is quite high – hmmmm…

22 www.hotsos.comSlide 22 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. To find out the Culprits You Need Workload Characterization Who What When Where How How Much?

23 www.hotsos.comSlide 23 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Workload Management Process Detect Workload Issues Measure and Characterize Workload Fix/Manage Workload

24 www.hotsos.comSlide 24 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. How do you get the Data for Workload Characterization? Oracle Apps Attribute Data Source Who OS Username, DB Username, Apps Username What Program, Module, Action When Logon time, Logoff time (if Conc Mgr job, Waittime too) Where Host, Terminal, Node How Much Cost Logical reads, Physical reads, cpu How Many Transactions Application Transaction Tables

25 www.hotsos.comSlide 25 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Hotsos Method/Product Terminology for Workload Characterization? Oracle Apps Attribute Entity Who Workers What When Where How Much Cost Cost by Worker, detail then summary How Many Xactions Widgets

26 www.hotsos.comSlide 26 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. For a given Workload, Both Response Time and Cost are Important AND Related… (AND Need to be Managed…) A Workload’s response time R = function(cost, contention) CostContentionResponse Time Low Fast LowHighNot so Fast HighLowNot so Fast High Certainly Slow

27 www.hotsos.comSlide 27 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Eliminating unnecessary workload often results in better-than-linear performance improvement. Eliminating work… –Saves top-line response time –Eliminates dependant work –Reduces queue lengths Makes remaining calls faster Helps everyone Eliminating unnecessary work reverses exponential performance degradation From Hotsos Problem Determination Class, Chapter 6 Day 3

28 www.hotsos.comSlide 28 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Said another way… Like Grocery store Checkout Lines, Computer Systems can be described by a “Queueing Model” The shape of the “hockey stick” is based on the number of clerks (cpu’s) and other factors With respect to the “knee” of curve left is “good” right is “bad” Utilization Response Time

29 www.hotsos.comSlide 29 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Said another way… If you have R pain, you might be “right” of the knee How do you go left to reduce utilization? Utilization Response Time Add Capacity (at what cost$$$?) Reduce Workload

30 www.hotsos.comSlide 30 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “The Whole ERP System is Slow” Hotsos Consulting, recent project

31 www.hotsos.comSlide 31 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “The Whole ERP System is Slow” Ms. Customer, On any one work day 80% of your ERP daily cost is going to “non- APPS” workloads As it turns out, all of “non-APPS” is custom Cognos reporting ERP is for transaction processing Your options are: shutdown or identify and tune the costly Cognos reports implement a reporting environment separate from transaction processing

32 www.hotsos.comSlide 32 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “Order Entry Meltdown” ChallengeApproach Client XYZ Company Custom Order Entry Application Application not meeting needs of the business Database Server max’ed out Tune Logical Reads Identify/Trace Work Measure and report progress 5 week effort

33 www.hotsos.comSlide 33 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Progress Report to the Client’s CIO Total Daily Logical Read 86% Reduction, over 5 Weeks

34 www.hotsos.comSlide 34 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Client Feedback… Client CIO, “I Guess that’s good work, but What’s a Logical Read?” “How do I know you didn’t reduce costs by: shutting down the database sending users home?

35 www.hotsos.comSlide 35 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study’s Alternative Ways to Report Success 500% Increase in Widget Production Over 5 weeks 40:1 “Betterment” In Cost per Order Over 5 weeks Client bought me dinner!!!

36 www.hotsos.comSlide 36 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “My CPU’s are maxed out already?  ” How did my Sizing Guestimates compare to Actual Usage?

37 www.hotsos.comSlide 37 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “My CPU’s are maxed out already?  ” How do my Sizing Guestimates compare to Actual Usage?

38 www.hotsos.comSlide 38 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “Performance is Highly Variable, day to day”

39 www.hotsos.comSlide 39 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “Performance is Highly Variable, day to day”

40 www.hotsos.comSlide 40 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “OE really slows down during Month End” Hotsos Consulting project, March 2007 Mr. Customer, during your OE window of 6am to 6pm on month end days you are incurring 80% of your day’s batch cost too let’s identify expensive day-time running batch move it to nighttime tune it

41 www.hotsos.comSlide 41 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – “Month End Batch runs too long” … Mr. Customer, on your busiest month end day (6444+1750)/13360 = 60% of your STANDARD jobs ran in 5 seconds or less The STANDARD queue has a sleep setting of 30 seconds Let’s setup a FAST queue, and a SLOW queue, and reassign jobs…

42 www.hotsos.comSlide 42 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Case Study – How “Valid” is the Stress Test??? Clone PROD to TEST, then Upgrade TEST Load test TEST as if it were PROD –Determine important PROD user activity from 2-4pm –Induce similar user activity in TEST for 2 hours, monitor –But No automated test scripts No orchestrated, manual test scripts Select PROD users to log in to TEST and “do their thing” Determine if TEST workload came close to PROD Determine if TEST suffered latching, too, or problem solved Decide Upgrade go-live based on TEST “closeness” and TEST system performance Large E-Business Customer Needs to Upgrade because of serious latching issues due to rdbms bug Will the Upgraded System Sustain PROD Volumes?

43 www.hotsos.comSlide 43 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Measuring TEST compared to PROD - Workers Workers BatchWorkers Online Workers Total Count Run Mins CountDistinctCount Prod 2-4pm 254639731342949650 TEST1990166326691626628 Hmmm – TEST light on Workers heavy on Batch Runtime Minutes

44 www.hotsos.comSlide 44 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Measuring TEST compared to PROD - Widgets Order Lines Journal Lines... Total Xactions Prod 2-4pm 41670116255982 TEST1049684112212 Hmmm – TEST was light on Widgets

45 www.hotsos.comSlide 45 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Measuring Day in Life TEST – Assembly Line Costs Logical Reads (millions) Physical Reads (millions) APPS Non- APPSAPPS Non- APPS Prod 2-4pm 4868111 TEST10382170 Hmmm – TEST much higher Costs

46 www.hotsos.comSlide 46 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. How Much like PROD was TEST? Probably, Manual Stress TEST users “underloaded” transaction processing “overloaded” reporting TEST was not valid!!! “Upon review, I think that many TEST users must have” used TEST as “pedal to the metal” disregarded instructions and pacing loaded up TEST with favorite longrunning month-end reports

47 www.hotsos.comSlide 47 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. How Much like PROD was TEST? Customer realized the TEST was invalid But hey, TEST had no latching problems Customer decided to upgrade anyway Argh…

48 www.hotsos.comSlide 48 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Summary Workload Characterization is a process To measure and understand the details of –Workers –Costs –Widgets (worker outputs) that occur in your E-Business or any Application System To apply different views against the details To look for details, anomalies, summaries, and trends To help you reduce utilization by knowing the right workloads to optimize To help you improve performance without hardware upgrades

49 www.hotsos.comSlide 49 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Summary Workload Characterization Has been practiced for 40 years in the mainframe world Is not popular nor widely applied in the Oracle/UNIX world Consequently, many customers “take the default” and buy more hardware in response to Workload Performance issues It’s simple and very cheap, if you know how, to Slice and dice and dissect all the work in your system Look for and “big rock”, costly work or other bad workload behavior Reduce utilization and “buy back” capacity quickly and easily

50 www.hotsos.comSlide 50 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Questions???

51 www.hotsos.comSlide 51 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Thank You!


Download ppt "1 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein"

Similar presentations


Ads by Google