Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein

Similar presentations


Presentation on theme: "1 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein"— Presentation transcript:

1 1 Copyright © by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein Hotsos Enterprises, Ltd. Charlotte Oracle Users Group Thursday May 10, 2007

2 2 Copyright © by Hotsos Enterprises, Ltd. Presenter’s Background Larry Klein 3 years, VP of Consulting, Hotsos Enterprises, Ltd. 10 years, Oracle Consulting and Development database and application performance and architecture 17 years, IBM and Candle Corp’s mainframe diagnostics, performance, and capacity planning Patrick Robbins, VP of Sales, Hotsos Enterprises, Ltd.

3 3 Copyright © by Hotsos Enterprises, Ltd. Presentation Agenda Workload and Characterization Concepts Data Sources Case Studies Questions?

4 4 Copyright © by Hotsos Enterprises, Ltd. What is Workload? The Exercise of Application Functionality for Business Benefit Application Functions Users Oracle E-Business Suite

5 5 Copyright © by Hotsos Enterprises, Ltd. Workload has a “Heaviness” or Cost Application Functions LightHeavy Users Few Workload Light Workload Medium Many Workload Medium Workload Heavy

6 6 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network

7 7 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network Hotsos sees Others do much “Downstream” tuning database ratios init.ora disk speeds …

8 8 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network Downstream efforts often are unfocused, wasted – like rearranging deck chairs on a certain luxury liner At Hotsos we believe there is much to be gained by focusing at the Workload level

9 9 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Tech Stack Workload Drives Resource Utilizations Operating Costs Good or Bad Behavior

10 10 Copyright © by Hotsos Enterprises, Ltd. Do You Remember Atlas? Greek god Lost battle against Zeus Punished by being made to carry on his shoulders the weight of the heavens So, Now that we Know Workload, What is Workload Characterization?

11 11 Copyright © by Hotsos Enterprises, Ltd. Poor Atlas… Ouch!!! I’m Really Sore!!!

12 12 Copyright © by Hotsos Enterprises, Ltd. Atlas Visits a “Traditional Tuning” Doctor… Ouch!!! I’m Really Sore!!! The Flex in your Knees exceeds a comfortable range wear some knee braces The Flex in your Back exceeds a comfortable range wear a back brace

13 13 Copyright © by Hotsos Enterprises, Ltd. Atlas Visits a “Workload Characterizing” Doctor… Ouch!!! I’m Really Sore!!! That’s a Pretty Big Rock you’ve got there Let’s find a way to do Something about that Rock

14 14 Copyright © by Hotsos Enterprises, Ltd. Workload Characterization? What is it? –The process of finding in an Application System The big rocks Many other very useful, powerful things Why do you care? –The big rocks impact other, more important workloads Resource consumption, contention, latching –Every system has big rocks Wasting resources Causing contention Causing one or more unplanned, unbudgeted hardware upgrades

15 15 Copyright © by Hotsos Enterprises, Ltd. When a Customer Calls Hotsos for Help What they say “Help, my Whole System is Slow” What they mean “Help, one or more individual workloads is slower than I need it to be” We help them by –Characterizing the workload –Looking for the big rocks –Considering their other, important issues –Putting together a game plan to analyze and fix

16 16 Copyright © by Hotsos Enterprises, Ltd. What Happens in our Hotsos Consulting work? We use the Hotsos Method R* Listen to the Customer Understand then compile an Issues List –Order Entry form OEXOEMOE takes 60 seconds to book an order, need it to be 5 seconds –Custom report XYZ takes 50 hours to run, need it to be 2 hrs because it’s on the critical path of month end processing –Holy cow, Big Rock DiscoverABC consumed 52% of yesterday’s total database processing costs Trace (10046 level 12) Profile Analyze Test and Resolve *R = Response Time

17 17 Copyright © by Hotsos Enterprises, Ltd. E-Business Customers Almost Always get into Workload Trouble, Particularly Big Rock Trouble Project Implementation Timeline Buy Software Go Live and Pray Guess #Users, Features, Sizing Order Hardware Map & Gap Install Hardware CRP’s ? More Users, Features! Build 1000 Custom- izations per the Gap, Uncon- strained batch schedule We’re out of time. We don’t need no stinking load testing

18 18 Copyright © by Hotsos Enterprises, Ltd. As a result, the Initial Server Sizing could be off… Congratulations! Your Project has generated 10 (or maybe 100?) pounds of workload for a 5 pound server

19 19 Copyright © by Hotsos Enterprises, Ltd. For E-Business Customers We Often find 2 Categories of Issues…

20 20 Copyright © by Hotsos Enterprises, Ltd. Common E-Business Foundation Issues IssueAction Not running certified releases Check Metalink for Certification status Creative init.oraMust follow legal settings (especially CBO) per Metalink Support Note (Under)sizingSee sizing table at end of Note No, inconsistent, or stale statistics Run FNDGSCST or equivalent, be careful of interaction with 10g rdbms GATHER_STATS job No pinningPin at database startup, frequently executed packages, functions, triggers No purging Run FNDWFPR to purge workflow tables Run FNDCPPUR, FNDSCPRG to purge FND tables …

21 21 Copyright © by Hotsos Enterprises, Ltd. Common E-Business Workload Issues IssueAction MicroThe thing I traced is its own problem. There is inefficient SQL or whatever. Go fix the SQL or whatever… MacroThe thing I traced, in and of itself, is good. it’s SQL is efficient on a no-load system it runs great under load it runs poorly who is (are) the culprits – consuming resource – contending with this task The thing I traced is good it’s a batch job the user is complaining about turnaround time runtime actually was quite good waittime before run is quite high – hmmmm…

22 22 Copyright © by Hotsos Enterprises, Ltd. To find out the Culprits You Need Workload Characterization Who What When Where How How Much?

23 23 Copyright © by Hotsos Enterprises, Ltd. Workload Management Process Detect Workload Issues Measure and Characterize Workload Fix/Manage Workload

24 24 Copyright © by Hotsos Enterprises, Ltd. How do you get the Data for Workload Characterization? Oracle Apps Attribute Data Source Who OS Username, DB Username, Apps Username What Program, Module, Action When Logon time, Logoff time (if Conc Mgr job, Waittime too) Where Host, Terminal, Node How Much Cost Logical reads, Physical reads, cpu How Many Transactions Application Transaction Tables

25 25 Copyright © by Hotsos Enterprises, Ltd. Hotsos Method/Product Terminology for Workload Characterization? Oracle Apps Attribute Entity Who Workers What When Where How Much Cost Cost by Worker, detail then summary How Many Xactions Widgets

26 26 Copyright © by Hotsos Enterprises, Ltd. For a given Workload, Both Response Time and Cost are Important AND Related… (AND Need to be Managed…) A Workload’s response time R = function(cost, contention) CostContentionResponse Time Low Fast LowHighNot so Fast HighLowNot so Fast High Certainly Slow

27 27 Copyright © by Hotsos Enterprises, Ltd. Eliminating unnecessary workload often results in better-than-linear performance improvement. Eliminating work… –Saves top-line response time –Eliminates dependant work –Reduces queue lengths Makes remaining calls faster Helps everyone Eliminating unnecessary work reverses exponential performance degradation From Hotsos Problem Determination Class, Chapter 6 Day 3

28 28 Copyright © by Hotsos Enterprises, Ltd. Said another way… Like Grocery store Checkout Lines, Computer Systems can be described by a “Queueing Model” The shape of the “hockey stick” is based on the number of clerks (cpu’s) and other factors With respect to the “knee” of curve left is “good” right is “bad” Utilization Response Time

29 29 Copyright © by Hotsos Enterprises, Ltd. Said another way… If you have R pain, you might be “right” of the knee How do you go left to reduce utilization? Utilization Response Time Add Capacity (at what cost$$$?) Reduce Workload

30 30 Copyright © by Hotsos Enterprises, Ltd. Case Study – “The Whole ERP System is Slow” Hotsos Consulting, recent project

31 31 Copyright © by Hotsos Enterprises, Ltd. Case Study – “The Whole ERP System is Slow” Ms. Customer, On any one work day 80% of your ERP daily cost is going to “non- APPS” workloads As it turns out, all of “non-APPS” is custom Cognos reporting ERP is for transaction processing Your options are: shutdown or identify and tune the costly Cognos reports implement a reporting environment separate from transaction processing

32 32 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Order Entry Meltdown” ChallengeApproach Client XYZ Company Custom Order Entry Application Application not meeting needs of the business Database Server max’ed out Tune Logical Reads Identify/Trace Work Measure and report progress 5 week effort

33 33 Copyright © by Hotsos Enterprises, Ltd. Progress Report to the Client’s CIO Total Daily Logical Read 86% Reduction, over 5 Weeks

34 34 Copyright © by Hotsos Enterprises, Ltd. Client Feedback… Client CIO, “I Guess that’s good work, but What’s a Logical Read?” “How do I know you didn’t reduce costs by: shutting down the database sending users home?

35 35 Copyright © by Hotsos Enterprises, Ltd. Case Study’s Alternative Ways to Report Success 500% Increase in Widget Production Over 5 weeks 40:1 “Betterment” In Cost per Order Over 5 weeks Client bought me dinner!!!

36 36 Copyright © by Hotsos Enterprises, Ltd. Case Study – “My CPU’s are maxed out already?  ” How did my Sizing Guestimates compare to Actual Usage?

37 37 Copyright © by Hotsos Enterprises, Ltd. Case Study – “My CPU’s are maxed out already?  ” How do my Sizing Guestimates compare to Actual Usage?

38 38 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Performance is Highly Variable, day to day”

39 39 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Performance is Highly Variable, day to day”

40 40 Copyright © by Hotsos Enterprises, Ltd. Case Study – “OE really slows down during Month End” Hotsos Consulting project, March 2007 Mr. Customer, during your OE window of 6am to 6pm on month end days you are incurring 80% of your day’s batch cost too let’s identify expensive day-time running batch move it to nighttime tune it

41 41 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Month End Batch runs too long” … Mr. Customer, on your busiest month end day ( )/13360 = 60% of your STANDARD jobs ran in 5 seconds or less The STANDARD queue has a sleep setting of 30 seconds Let’s setup a FAST queue, and a SLOW queue, and reassign jobs…

42 42 Copyright © by Hotsos Enterprises, Ltd. Case Study – How “Valid” is the Stress Test??? Clone PROD to TEST, then Upgrade TEST Load test TEST as if it were PROD –Determine important PROD user activity from 2-4pm –Induce similar user activity in TEST for 2 hours, monitor –But No automated test scripts No orchestrated, manual test scripts Select PROD users to log in to TEST and “do their thing” Determine if TEST workload came close to PROD Determine if TEST suffered latching, too, or problem solved Decide Upgrade go-live based on TEST “closeness” and TEST system performance Large E-Business Customer Needs to Upgrade because of serious latching issues due to rdbms bug Will the Upgraded System Sustain PROD Volumes?

43 43 Copyright © by Hotsos Enterprises, Ltd. Measuring TEST compared to PROD - Workers Workers BatchWorkers Online Workers Total Count Run Mins CountDistinctCount Prod 2-4pm TEST Hmmm – TEST light on Workers heavy on Batch Runtime Minutes

44 44 Copyright © by Hotsos Enterprises, Ltd. Measuring TEST compared to PROD - Widgets Order Lines Journal Lines... Total Xactions Prod 2-4pm TEST Hmmm – TEST was light on Widgets

45 45 Copyright © by Hotsos Enterprises, Ltd. Measuring Day in Life TEST – Assembly Line Costs Logical Reads (millions) Physical Reads (millions) APPS Non- APPSAPPS Non- APPS Prod 2-4pm TEST Hmmm – TEST much higher Costs

46 46 Copyright © by Hotsos Enterprises, Ltd. How Much like PROD was TEST? Probably, Manual Stress TEST users “underloaded” transaction processing “overloaded” reporting TEST was not valid!!! “Upon review, I think that many TEST users must have” used TEST as “pedal to the metal” disregarded instructions and pacing loaded up TEST with favorite longrunning month-end reports

47 47 Copyright © by Hotsos Enterprises, Ltd. How Much like PROD was TEST? Customer realized the TEST was invalid But hey, TEST had no latching problems Customer decided to upgrade anyway Argh…

48 48 Copyright © by Hotsos Enterprises, Ltd. Summary Workload Characterization is a process To measure and understand the details of –Workers –Costs –Widgets (worker outputs) that occur in your E-Business or any Application System To apply different views against the details To look for details, anomalies, summaries, and trends To help you reduce utilization by knowing the right workloads to optimize To help you improve performance without hardware upgrades

49 49 Copyright © by Hotsos Enterprises, Ltd. Summary Workload Characterization Has been practiced for 40 years in the mainframe world Is not popular nor widely applied in the Oracle/UNIX world Consequently, many customers “take the default” and buy more hardware in response to Workload Performance issues It’s simple and very cheap, if you know how, to Slice and dice and dissect all the work in your system Look for and “big rock”, costly work or other bad workload behavior Reduce utilization and “buy back” capacity quickly and easily

50 50 Copyright © by Hotsos Enterprises, Ltd. Questions???

51 51 Copyright © by Hotsos Enterprises, Ltd. Thank You!


Download ppt "1 Copyright © 1999-2007 by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein"

Similar presentations


Ads by Google