We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byBryan Atlee
Modified about 1 year ago
1 Copyright © by Hotsos Enterprises, Ltd. Characterizing Workload for the Oracle E-Business Suite Larry Klein Hotsos Enterprises, Ltd. Charlotte Oracle Users Group Thursday May 10, 2007
2 Copyright © by Hotsos Enterprises, Ltd. Presenter’s Background Larry Klein 3 years, VP of Consulting, Hotsos Enterprises, Ltd. 10 years, Oracle Consulting and Development database and application performance and architecture 17 years, IBM and Candle Corp’s mainframe diagnostics, performance, and capacity planning Patrick Robbins, VP of Sales, Hotsos Enterprises, Ltd.
3 Copyright © by Hotsos Enterprises, Ltd. Presentation Agenda Workload and Characterization Concepts Data Sources Case Studies Questions?
4 Copyright © by Hotsos Enterprises, Ltd. What is Workload? The Exercise of Application Functionality for Business Benefit Application Functions Users Oracle E-Business Suite
5 Copyright © by Hotsos Enterprises, Ltd. Workload has a “Heaviness” or Cost Application Functions LightHeavy Users Few Workload Light Workload Medium Many Workload Medium Workload Heavy
6 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network
7 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network Hotsos sees Others do much “Downstream” tuning database ratios init.ora disk speeds …
8 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Database Operating System CPU, Memory, Disk, Network Downstream efforts often are unfocused, wasted – like rearranging deck chairs on a certain luxury liner At Hotsos we believe there is much to be gained by focusing at the Workload level
9 Copyright © by Hotsos Enterprises, Ltd. Why is “Workload” Important? Workload Tech Stack Workload Drives Resource Utilizations Operating Costs Good or Bad Behavior
10 Copyright © by Hotsos Enterprises, Ltd. Do You Remember Atlas? Greek god Lost battle against Zeus Punished by being made to carry on his shoulders the weight of the heavens So, Now that we Know Workload, What is Workload Characterization?
11 Copyright © by Hotsos Enterprises, Ltd. Poor Atlas… Ouch!!! I’m Really Sore!!!
12 Copyright © by Hotsos Enterprises, Ltd. Atlas Visits a “Traditional Tuning” Doctor… Ouch!!! I’m Really Sore!!! The Flex in your Knees exceeds a comfortable range wear some knee braces The Flex in your Back exceeds a comfortable range wear a back brace
13 Copyright © by Hotsos Enterprises, Ltd. Atlas Visits a “Workload Characterizing” Doctor… Ouch!!! I’m Really Sore!!! That’s a Pretty Big Rock you’ve got there Let’s find a way to do Something about that Rock
14 Copyright © by Hotsos Enterprises, Ltd. Workload Characterization? What is it? –The process of finding in an Application System The big rocks Many other very useful, powerful things Why do you care? –The big rocks impact other, more important workloads Resource consumption, contention, latching –Every system has big rocks Wasting resources Causing contention Causing one or more unplanned, unbudgeted hardware upgrades
15 Copyright © by Hotsos Enterprises, Ltd. When a Customer Calls Hotsos for Help What they say “Help, my Whole System is Slow” What they mean “Help, one or more individual workloads is slower than I need it to be” We help them by –Characterizing the workload –Looking for the big rocks –Considering their other, important issues –Putting together a game plan to analyze and fix
16 Copyright © by Hotsos Enterprises, Ltd. What Happens in our Hotsos Consulting work? We use the Hotsos Method R* Listen to the Customer Understand then compile an Issues List –Order Entry form OEXOEMOE takes 60 seconds to book an order, need it to be 5 seconds –Custom report XYZ takes 50 hours to run, need it to be 2 hrs because it’s on the critical path of month end processing –Holy cow, Big Rock DiscoverABC consumed 52% of yesterday’s total database processing costs Trace (10046 level 12) Profile Analyze Test and Resolve *R = Response Time
17 Copyright © by Hotsos Enterprises, Ltd. E-Business Customers Almost Always get into Workload Trouble, Particularly Big Rock Trouble Project Implementation Timeline Buy Software Go Live and Pray Guess #Users, Features, Sizing Order Hardware Map & Gap Install Hardware CRP’s ? More Users, Features! Build 1000 Custom- izations per the Gap, Uncon- strained batch schedule We’re out of time. We don’t need no stinking load testing
18 Copyright © by Hotsos Enterprises, Ltd. As a result, the Initial Server Sizing could be off… Congratulations! Your Project has generated 10 (or maybe 100?) pounds of workload for a 5 pound server
19 Copyright © by Hotsos Enterprises, Ltd. For E-Business Customers We Often find 2 Categories of Issues…
20 Copyright © by Hotsos Enterprises, Ltd. Common E-Business Foundation Issues IssueAction Not running certified releases Check Metalink for Certification status Creative init.oraMust follow legal settings (especially CBO) per Metalink Support Note (Under)sizingSee sizing table at end of Note No, inconsistent, or stale statistics Run FNDGSCST or equivalent, be careful of interaction with 10g rdbms GATHER_STATS job No pinningPin at database startup, frequently executed packages, functions, triggers No purging Run FNDWFPR to purge workflow tables Run FNDCPPUR, FNDSCPRG to purge FND tables …
21 Copyright © by Hotsos Enterprises, Ltd. Common E-Business Workload Issues IssueAction MicroThe thing I traced is its own problem. There is inefficient SQL or whatever. Go fix the SQL or whatever… MacroThe thing I traced, in and of itself, is good. it’s SQL is efficient on a no-load system it runs great under load it runs poorly who is (are) the culprits – consuming resource – contending with this task The thing I traced is good it’s a batch job the user is complaining about turnaround time runtime actually was quite good waittime before run is quite high – hmmmm…
22 Copyright © by Hotsos Enterprises, Ltd. To find out the Culprits You Need Workload Characterization Who What When Where How How Much?
23 Copyright © by Hotsos Enterprises, Ltd. Workload Management Process Detect Workload Issues Measure and Characterize Workload Fix/Manage Workload
24 Copyright © by Hotsos Enterprises, Ltd. How do you get the Data for Workload Characterization? Oracle Apps Attribute Data Source Who OS Username, DB Username, Apps Username What Program, Module, Action When Logon time, Logoff time (if Conc Mgr job, Waittime too) Where Host, Terminal, Node How Much Cost Logical reads, Physical reads, cpu How Many Transactions Application Transaction Tables
25 Copyright © by Hotsos Enterprises, Ltd. Hotsos Method/Product Terminology for Workload Characterization? Oracle Apps Attribute Entity Who Workers What When Where How Much Cost Cost by Worker, detail then summary How Many Xactions Widgets
26 Copyright © by Hotsos Enterprises, Ltd. For a given Workload, Both Response Time and Cost are Important AND Related… (AND Need to be Managed…) A Workload’s response time R = function(cost, contention) CostContentionResponse Time Low Fast LowHighNot so Fast HighLowNot so Fast High Certainly Slow
27 Copyright © by Hotsos Enterprises, Ltd. Eliminating unnecessary workload often results in better-than-linear performance improvement. Eliminating work… –Saves top-line response time –Eliminates dependant work –Reduces queue lengths Makes remaining calls faster Helps everyone Eliminating unnecessary work reverses exponential performance degradation From Hotsos Problem Determination Class, Chapter 6 Day 3
28 Copyright © by Hotsos Enterprises, Ltd. Said another way… Like Grocery store Checkout Lines, Computer Systems can be described by a “Queueing Model” The shape of the “hockey stick” is based on the number of clerks (cpu’s) and other factors With respect to the “knee” of curve left is “good” right is “bad” Utilization Response Time
29 Copyright © by Hotsos Enterprises, Ltd. Said another way… If you have R pain, you might be “right” of the knee How do you go left to reduce utilization? Utilization Response Time Add Capacity (at what cost$$$?) Reduce Workload
30 Copyright © by Hotsos Enterprises, Ltd. Case Study – “The Whole ERP System is Slow” Hotsos Consulting, recent project
31 Copyright © by Hotsos Enterprises, Ltd. Case Study – “The Whole ERP System is Slow” Ms. Customer, On any one work day 80% of your ERP daily cost is going to “non- APPS” workloads As it turns out, all of “non-APPS” is custom Cognos reporting ERP is for transaction processing Your options are: shutdown or identify and tune the costly Cognos reports implement a reporting environment separate from transaction processing
32 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Order Entry Meltdown” ChallengeApproach Client XYZ Company Custom Order Entry Application Application not meeting needs of the business Database Server max’ed out Tune Logical Reads Identify/Trace Work Measure and report progress 5 week effort
33 Copyright © by Hotsos Enterprises, Ltd. Progress Report to the Client’s CIO Total Daily Logical Read 86% Reduction, over 5 Weeks
34 Copyright © by Hotsos Enterprises, Ltd. Client Feedback… Client CIO, “I Guess that’s good work, but What’s a Logical Read?” “How do I know you didn’t reduce costs by: shutting down the database sending users home?
35 Copyright © by Hotsos Enterprises, Ltd. Case Study’s Alternative Ways to Report Success 500% Increase in Widget Production Over 5 weeks 40:1 “Betterment” In Cost per Order Over 5 weeks Client bought me dinner!!!
36 Copyright © by Hotsos Enterprises, Ltd. Case Study – “My CPU’s are maxed out already? ” How did my Sizing Guestimates compare to Actual Usage?
37 Copyright © by Hotsos Enterprises, Ltd. Case Study – “My CPU’s are maxed out already? ” How do my Sizing Guestimates compare to Actual Usage?
38 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Performance is Highly Variable, day to day”
39 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Performance is Highly Variable, day to day”
40 Copyright © by Hotsos Enterprises, Ltd. Case Study – “OE really slows down during Month End” Hotsos Consulting project, March 2007 Mr. Customer, during your OE window of 6am to 6pm on month end days you are incurring 80% of your day’s batch cost too let’s identify expensive day-time running batch move it to nighttime tune it
41 Copyright © by Hotsos Enterprises, Ltd. Case Study – “Month End Batch runs too long” … Mr. Customer, on your busiest month end day ( )/13360 = 60% of your STANDARD jobs ran in 5 seconds or less The STANDARD queue has a sleep setting of 30 seconds Let’s setup a FAST queue, and a SLOW queue, and reassign jobs…
42 Copyright © by Hotsos Enterprises, Ltd. Case Study – How “Valid” is the Stress Test??? Clone PROD to TEST, then Upgrade TEST Load test TEST as if it were PROD –Determine important PROD user activity from 2-4pm –Induce similar user activity in TEST for 2 hours, monitor –But No automated test scripts No orchestrated, manual test scripts Select PROD users to log in to TEST and “do their thing” Determine if TEST workload came close to PROD Determine if TEST suffered latching, too, or problem solved Decide Upgrade go-live based on TEST “closeness” and TEST system performance Large E-Business Customer Needs to Upgrade because of serious latching issues due to rdbms bug Will the Upgraded System Sustain PROD Volumes?
43 Copyright © by Hotsos Enterprises, Ltd. Measuring TEST compared to PROD - Workers Workers BatchWorkers Online Workers Total Count Run Mins CountDistinctCount Prod 2-4pm TEST Hmmm – TEST light on Workers heavy on Batch Runtime Minutes
44 Copyright © by Hotsos Enterprises, Ltd. Measuring TEST compared to PROD - Widgets Order Lines Journal Lines... Total Xactions Prod 2-4pm TEST Hmmm – TEST was light on Widgets
45 Copyright © by Hotsos Enterprises, Ltd. Measuring Day in Life TEST – Assembly Line Costs Logical Reads (millions) Physical Reads (millions) APPS Non- APPSAPPS Non- APPS Prod 2-4pm TEST Hmmm – TEST much higher Costs
46 Copyright © by Hotsos Enterprises, Ltd. How Much like PROD was TEST? Probably, Manual Stress TEST users “underloaded” transaction processing “overloaded” reporting TEST was not valid!!! “Upon review, I think that many TEST users must have” used TEST as “pedal to the metal” disregarded instructions and pacing loaded up TEST with favorite longrunning month-end reports
47 Copyright © by Hotsos Enterprises, Ltd. How Much like PROD was TEST? Customer realized the TEST was invalid But hey, TEST had no latching problems Customer decided to upgrade anyway Argh…
48 Copyright © by Hotsos Enterprises, Ltd. Summary Workload Characterization is a process To measure and understand the details of –Workers –Costs –Widgets (worker outputs) that occur in your E-Business or any Application System To apply different views against the details To look for details, anomalies, summaries, and trends To help you reduce utilization by knowing the right workloads to optimize To help you improve performance without hardware upgrades
49 Copyright © by Hotsos Enterprises, Ltd. Summary Workload Characterization Has been practiced for 40 years in the mainframe world Is not popular nor widely applied in the Oracle/UNIX world Consequently, many customers “take the default” and buy more hardware in response to Workload Performance issues It’s simple and very cheap, if you know how, to Slice and dice and dissect all the work in your system Look for and “big rock”, costly work or other bad workload behavior Reduce utilization and “buy back” capacity quickly and easily
50 Copyright © by Hotsos Enterprises, Ltd. Questions???
51 Copyright © by Hotsos Enterprises, Ltd. Thank You!
© 2009 Wellesley Information Services. All rights reserved. 20 technical tips and tricks to speed SAP NetWeaver Business Intelligence query, report, and.
Chapter 12 Technology. INTRODUCTION This chapter considers technology in general, with some limited emphasis on software. The life cycle and software.
Help Desk Procedures Topic: Tasks of the Help Desk Operator Written by Greg Webb while at Information Technology, Sydney Institute of Technology. Current.
Software Development QA Best Practices May 20, 2010 Suzette Hackl, CSM Senior Project Manager Skyline Technologies, Inc.
® IBM Software Group © 2008 IBM Corporation A new feature providing mainframe development flexibility David Myers Rational Developer for System z Product.
1 GREY BOX TESTING Web Apps & Networking Session 10 Boris Grinberg
SeETL The Worlds Most Productive ETL Tool 1/1/2012
Partition, Compress, Archive and Purge Keep Your System on the Road David Kurtz Go-Faster Consultancy Ltd.
Public Information Version 3.1: 1/1/2012 Introducing Instant Business Intelligence To IT BI Project Managers What you need, when you need it
What is an Operating System? A program that acts as an intermediary between a user of a computer and the computer hardware. Operating system goals: Execute.
A Publication of Bridgemark Solutions Six Keys to Generating More Sales Leads AND WINNING MORE MARKET RESEARCH PROJECTS.
© 2009 Wellesley Information Services. All rights reserved. A comprehensive guide to SAP NetWeaver Visual Composer Dr. Bjarne Berg.
Information Systems Using Information (Higher and Intermediate 2)
Linux, and Windows and VMWare, oh my! By Ron Kaminski
Oracle Enterprise Manager Grid Control: Day in the Life of An Admin Wilson N. López – Solution Specialist.
Quality Tools and Techniques in the School and Classroom.
Exchange 2010 Hosting Service How Hosted Exchange Works and Benefits Businesses.
ADAMA UNIVERSITY SCHOOL OF ENGENEERING & INFORMATION TECHNOLOGIES Sem. II,2010/11 INSTRUCTOR TARIKU W OPERATING SYSTEM (IT3016)
Knowledge in implementing/managing the IS/IT project CASE-The Brose Group Implements Page
INTERMEDIATE 1 PHYSICAL EDUCATION STRUCTURES AND STRATEGIES INFORMATION PACK Name : _____________________________________ Class : _________ Year : ______.
1 Advanced Database Application Development Performance Tuning Performance Benchmarks Standardization E-Commerce Legacy Systems.
1 Data Warehousing Denis Manley Enterprise Systems FT228/3.
Managing IT Budgets in Uncertain Economic Times: IT Optimization.
Software Quality Engineering Software Metrics-II.
1 GREY BOX TESTING Web Apps & Networking Session 7 Boris Grinberg
A Recruiting Workshop for Nutrition Educators Developed by Howard Armstrong and Linda Melcher Revised by Linda Melcher, MS, RD Creating Meaningful Conversations.
© 2016 SlidePlayer.com Inc. All rights reserved.