1. Copyright © 2005 HyPerformix, Inc. Moving Beyond Test and Guess Using Modeling with Load Testing to Improve Web Application Readiness Keith Smith Authors: Richard Gimarc, Amy Spellmann & Jim Reynolds HyPerformix, Inc. Kansas City CMG 2005

2. Copyright © 2005 HyPerformix, Inc.2 Agenda Load testing goals & effectiveness Bridging the gap from test to production Natural partnership: Load Testing & Modeling Case study – FMStocks –Data collection & analysis –Model construction & validation –What-If scenario evaluation –Evaluating predictive accuracy Summary

3. Copyright © 2005 HyPerformix, Inc.3 Why Load Test? Verify that applications meet SLAs for availability and performance Mitigate risk Provide assurance that applications will deploy successfully application readinessDemonstrate application readiness

4. Copyright © 2005 HyPerformix, Inc.4 Application Readiness The practice of fully preparing a business software application for deployment to customers, partners and internal users application readinessThe goal of application readiness is to verify that an application provides –Functionality that meets user needs –Accuracy to consistently produce the correct results –Reliability to perform according to SLAs

5. Copyright © 2005 HyPerformix, Inc.5 Load Testing Goals Software applications are ready –Ensure applications are ready to be deployed from a performance & functionality perspective Hardware –Verify hardware requirements for production Tuning –Tune/optimize application performance

6. Copyright © 2005 HyPerformix, Inc.6 Load Testing Challenges Time consuming –Significant time required to perform tests limits the number of configurations that can be evaluated –Breadth of exploration is limited by time and test environment Expensive –Test environments that mirror production can be prohibitively expensive –Difficult to match production workloads, databases, mainframes, etc. Predicting human behavior –Difficult to predict exactly how the system will be used –Which use cases will mimic reality?

7. Copyright © 2005 HyPerformix, Inc.7 Modeling Improves Load Test Effectiveness Time – Tasks that take hours in Test take minutes to model – Utilizes a scientific, repeatable methodology – Quickly evaluates hardware and software configurations – Guides the load test – Increases the scope and coverage of Test Money – The model is a virtual test lab – Does not require hardware to mirror production – Extrapolates test to production Future – Rapid analysis of new & different usage scenarios – Provides ongoing capacity planning capability

8. Copyright © 2005 HyPerformix, Inc.8 Value of Modeling with Load Testing “We saved $500k in hardware investments for our production environment by combining modeling with our load testing effort.” “We identified excess server capacity for the planned application deployment – which we were able to use for other capacity constrained applications.”

9. Copyright © 2005 HyPerformix, Inc.9 The BIG Question If – You have successfully completed your load testing – Your load test environment does not mirror production – You were time-constrained, did not evaluate all usage scenarios Then – Are you confident your production deployment will succeed? What do you do? – Test & Guess – Apply Performance Modeling

10. Copyright © 2005 HyPerformix, Inc.10 Load Test Decision Points: Where Modeling Assists Extrapolate from the test environment to production Determine the most cost effective usage scenarios and test configurations Design Production Test Develop

11. Copyright © 2005 HyPerformix, Inc.11 Coverage of Usage Scenarios Load testing should focus on the most critical scenario(s) –High frequency –Resource intensive Challenges –Which scenarios are most likely to match reality? –How does performance vary with different use cases? –What if end users do not actually use the application the way we tested it? –Which configurations are likely to provide the best performance? Modeling efficiently evaluates different scenarios to guide the focus of the load test

12. Copyright © 2005 HyPerformix, Inc.12 Example of System Sensitivity to Usage Scenarios

13. Copyright © 2005 HyPerformix, Inc.13 Bridging the Gap: Test to Production

14. Copyright © 2005 HyPerformix, Inc.14 Bridging the Gap: Test to Production

15. Copyright © 2005 HyPerformix, Inc.15 Bridging the Gap #1 : Test Lab Topology

16. Copyright © 2005 HyPerformix, Inc.16 Bridging the Gap #1 : Production Topology 6 App Servers Private WAN 2 data centers

17. Copyright © 2005 HyPerformix, Inc.17 Bridging the Gap #2: Test Lab Topology

18. Copyright © 2005 HyPerformix, Inc.18 Bridging the Gap #2 : Production Topology 5 App Servers V880 & E450 External Credit Auth Svc Firewall s DMZ Internet

19. Copyright © 2005 HyPerformix, Inc.19 Load Testing & Modeling – A Natural Partnership Well-established practices –Load & Stress Testing –Performance Modeling Common goal –Assuring application performance readiness Different perspective & approach –Load Tester – deals with reality –Modeler – creates a representation of reality Question:How do you integrate these two practices to form a synergistic partnership?

20. Copyright © 2005 HyPerformix, Inc.20 Two Views of Accuracy Load Testers –Their tests are the “the ultimate in performance simulation”* –Mirrored production, then they are 100% accurate –Differs from production, there is no way to estimate its accuracy, it’s guesswork Modelers –Models do not have to be 100% accurate to be useful –A repeatable proven process is always better than a guess –Providing directional guidance requires 70-80% –Detailed response time analysis may require 85-95% * Neil Gunther, “How to Get Unbelievable Load Test Results”

21. Copyright © 2005 HyPerformix, Inc.21 The Cost of Modeling Additional data requirements –Business function flow –Business function cost per tier –Differences between test & production Modeling is extra work –Requires running additional tests –Load testing offers a convenient & controlled environment –Extra data collection cost can be minimized with proper planning Modeling is less expensive than mirroring production

22. Copyright © 2005 HyPerformix, Inc.22 Case Study - FMStocks Demonstrate how to integrate Modeling into the Load Testing process Describe the additional data collection tasks Show how the model was built Validate the model’s accuracy Perform What-If scenarios to evaluate scalability Evaluate the model’s predictive accuracy

23. Copyright © 2005 HyPerformix, Inc.23 FMStocks Introducing FMStocks 2000: Scales to support tens of thousands of concurrent users! Microsoft Windows 2000, Microsoft SQL Server 7.0 and.Net Services.

24. Copyright © 2005 HyPerformix, Inc.24 FMStocks Terminology Business Functions – Transactions –Login –Buy –Sell –View Portfolio –View Summary –Logout Business Processes –Sequence of business functions View Account (80%) Stock Purchase (10%) Sell Stock (10%) Stock Purchase Login View Portfolio Buy View Portfolio Logout Stock Sell Login Sell View Portfolio Logout View Account Login View Summary View Portfolio Logout

25. Copyright © 2005 HyPerformix, Inc.25 Model Data Requirements - WIFR W orkload –Describes the load placed on the modeled system. Examples include business function arrival rate, number of users, and response time. I nfrastructure –Description of the hardware, network, and software subsystems. This execution environment supports the application and workload. F low –Documents the step-by-step progress of business functions through the infrastructure. R esource –Resource requirements are needed that describe the processing performed on each tier for each business function. Examples of resources are CPU seconds, I/Os, and memory.

26. Copyright © 2005 HyPerformix, Inc.26 Data Collection Tools –LoadRunner –Ethereal (LAN Analyzer) –Windows System Monitor (a.k.a. PerfMon) Three types of tests –Single business function trace: T1 tests Capture business function flow –Single business function load test: T2 tests Determine business function resource usage –Traditional load tests: T3 tests Validation targets

27. Copyright © 2005 HyPerformix, Inc.27 WIFR Population with the T Tests

28. Copyright © 2005 HyPerformix, Inc.28 FMStocks - Austin Test Lab Load Scaled By Vusers 2 Load Drivers WebServer: 4@1.5 GHz AppServer: 4@1.5 GHz DBServer: 2@2.4 GHz 100 Mb LAN FMStocks Application Processes

29. Copyright © 2005 HyPerformix, Inc.29 Data Collection: T1 Tests Created LoadRunner script to step through each business function LoadRunner submitted the business functions as shown above Network traffic collected using Ethereal Transformed the raw packet trace into a business function flow T1 Testing Time: 2 hours

30. Copyright © 2005 HyPerformix, Inc.30 T1 Result: Business Function Flow - Buy 32 Turns 5 Turns 4 Turns

31. Copyright © 2005 HyPerformix, Inc.31 Data Collection: T2 Tests Created LoadRunner script to drive single business functions LoadRunner submitted the business functions as shown above LoadRunner collected driver and server metrics Use utilization and throughput to calculate resource usage T2 Testing Time: 8 hours

32. Copyright © 2005 HyPerformix, Inc.32 T2 Result: Buy Throughput & Utilization Apply Utilization Law to determine business function resource usage: Trans Per Sec App Server Utilization 2 Users4 Users6 Users2 Users4 Users6 Users

33. Copyright © 2005 HyPerformix, Inc.33 FMStocks – Data Analysis & Model Creation Data Analysis: 4 days Model Creation: 1 day

34. Copyright © 2005 HyPerformix, Inc.34 Validation Methodology (T3 Tests) Vuser counts of 100, 200, 300, 400, 500, 600, 650, 700, 750 Perform T3 Tests Server utilization Network utilization Response Time Throughput Determine Low, Medium & High Load Levels Extract T3 Validation Metrics Evaluate Model Using Same Load Levels Compare Results Modeled versus Measured Successful Comparison? Don e Refine Model & Repeat You may have to refine the model to account for differences between measured and modeled results. LoadRunner Drives T3 Tests Collects Validation Metrics

35. Copyright © 2005 HyPerformix, Inc.35 FMStocks Validation Results Hardware utilization comparison was on target Modeled throughput matched measurements Modeled response times were not tracking with measurements Model predictions were always lower than the measured Significant deviation at 600 Vusers Indicates software contention (no hardware contention) Further data analysis identified contention in the Web Server

36. Copyright © 2005 HyPerformix, Inc.36 Initial Validation Results – Throughput Modeled transaction throughput matches measurements.

37. Copyright © 2005 HyPerformix, Inc.37 Initial Validation – CPU & Network Utilization Utilizations match measurements. App Server UtilizationDB Server Utilization Web Server UtilizationNetwork Utilization

38. Copyright © 2005 HyPerformix, Inc.38 Initial Validation Results – Response Time Response time increasing rapidly in measured system. Deviates from model results at 500 Vusers. BUT

39. Copyright © 2005 HyPerformix, Inc.39 Initial Validation Results Analysis: Software Contention in the Web Server Web server request rate grows as expected with load. Web server “contentions” curve matches increasing response time.

40. Copyright © 2005 HyPerformix, Inc.40 Validation Model Refinement Load test identified software queuing in the Web server at increasing load levels Determine refinement options √ Load dependent delay function for each business function –Queuing algorithm or thread limitation (requires the ability to decipher code & configuration parameters) Add contention function (ADN Behavior) to compute software delay in Optimizer Repeat Validation Runs

41. Copyright © 2005 HyPerformix, Inc.41 New Validation Results Updated the model to reflect the observed contention Repeat validation runs with the model Evaluate modeled vs. measured results –Updated model tracks accurately with measurements (10-20%) –Business function response time –CPU utilization –Network utilization Modeled results within 10-20% of measurements!

42. Copyright © 2005 HyPerformix, Inc.42 Validation Results: Response Time [100 to 750 Vusers] BuyView Portfolio View SummarySell

43. Copyright © 2005 HyPerformix, Inc.43 Validation Results: Utilization [100 to 750 Vusers] Web Server App ServerNetwork DB Server

44. Copyright © 2005 HyPerformix, Inc.44 Validation Results Summary Network utilization tracked well in both the initial and final validation CPU utilization tracked well in both cases Accuracy of response time predictions closely linked to representation of software constraints Confident the model can be used for What-if scenarios

45. Copyright © 2005 HyPerformix, Inc.45 What-If Modeling Scenarios Goal –Ensure application readiness to support 1,000 users Questions –What Load Test configuration is required to support 1,000 users? –What are the sensitive application components? –How accurate is the model’s extrapolation to production?

46. Copyright © 2005 HyPerformix, Inc.46 Modeling Scenario - Baseline Scalability Determine Baseline Capacity –How many users can our baseline configuration support? –Increase the number of users until the system “breaks” System component becomes saturated Response time degradation –Identify limiting component

47. Copyright © 2005 HyPerformix, Inc.47 Modeling Scenario – Baseline Scalability Business Function Response Time Knee at 900 Vuser s

48. Copyright © 2005 HyPerformix, Inc.48 Baseline Scalability Extrapolation: Response Time Comparison Response times track well up to 1,000 Vusers Buy SellView Portfolio View Summary 750* 800 900 1000

49. Copyright © 2005 HyPerformix, Inc.49 Modeling Scenario – Baseline Scalability Business Function Throughput Limit at 900 Vuser s

50. Copyright © 2005 HyPerformix, Inc.50 Modeling Scenario – Baseline Scalability Web Server Transaction Processing Software Thread Constrain t

51. Copyright © 2005 HyPerformix, Inc.51 Modeling Scenario – Baseline Scalability Server CPU Utilization Limit at 900 Vuser s

52. Copyright © 2005 HyPerformix, Inc.52 Modeling Scenario – Baseline Scalability Network Utilization Limit at 900 Vuser s

53. Copyright © 2005 HyPerformix, Inc.53 Modeling Scenario: Hardware Configuration Changes Web Server was the bottleneck in baseline –Software constraint –Highest CPU utilization Relieve Web Server software constraint –Replace single 4-way 1.5 GHz Web Server with two 2-way 2.4 GHz servers –This will decrease the load to the Web Server Rerun the scalability scenario –Increase the number of Vusers to 1,000+ –Can we defer response time degradation? –Can we safely support 1,000 users?

54. Copyright © 2005 HyPerformix, Inc.54 Baseline vs. Web Server Upgrade Response Time Comparison Upgrade knee at 1,100 Vusers Previous knee at 900

55. Copyright © 2005 HyPerformix, Inc.55 Modeling Scenario – Web Server Upgrade Business Function Throughput Limit at 1,100 Vusers

56. Copyright © 2005 HyPerformix, Inc.56 Modeling Scenario Summary Baseline Scalability –Simulated business function response times remain under 4 seconds for up to 1,200 Vusers –Response times start to degrade at 900 Vusers –Primary bottleneck is the Web Server software constraint –The Network is secondary bottleneck Hardware Upgrade to Two Web Servers –Improves the throughput –More fully utilizes hardware infrastructure –Response times start to degrade at 1,100 Vusers (+200 Vusers) Use upgraded configuration for Load Test

57. Copyright © 2005 HyPerformix, Inc.57 Extrapolating to Production Is the model accurate enough to extrapolate to the production load of 1,000 users with the upgrade? The model was demonstrated to be accurate up to 750 users (baseline validation) How accurate is the model with 750+ users? –Run load tests with same user loads & hardware changes that we just modeled in the scenarios Baseline Scalability Web Server Upgrade (increase to 2 Web Servers) –Compare model predictions to load test measurements

58. Copyright © 2005 HyPerformix, Inc.58 Baseline Scalability Extrapolation: Utilization Comparison Utilizations are good up to 1,000 Vusers 750* 800 900 1000 DB ServerNetwork Web ServerApp Server

59. Copyright © 2005 HyPerformix, Inc.59 Web Server Upgrade Extrapolation: Response Times Comparison Response times are good up to 800 Vusers 600* 800 1000 Buy SellView Portfolio View Summary

60. Copyright © 2005 HyPerformix, Inc.60 Web Server Upgrade Extrapolation: Utilization Comparison Utilizations are good up to 1,000 Vusers Web ServerApp Server DB ServerNetwork 600* 800 1000 DB server in load test changed to a 1-CPU server!!

61. Copyright © 2005 HyPerformix, Inc.61 Effect of Hyper-Threading on CPU Measurements With hyper-threading enabled, the operating system measures what it sees – the logical processors. The test results show that processor time measurements for a fixed amount of work depend on the level of activity for each logical processor. The level of activity determines whether the physical processor operates in single-task state or multi-task state. When in multi-task state, wait time due to the contention of logical processors appears to be included in the measurement of logical processor time. Johnson, S. “Measuring CPU Time from Hyper-Threading Enabled Intel Processors”, Proceedings of the Computer Measurements Group 2003 International Conference.

62. Copyright © 2005 HyPerformix, Inc.62 Throughput vs. Response Time Consistent response time across throughput range

63. Copyright © 2005 HyPerformix, Inc.63 CPU Utilization vs. CPU per Business Function HT CPU/BF gradually increases with utilization

64. Copyright © 2005 HyPerformix, Inc.64 Throughput vs. CPU per Business Function HT CPU/BF gradually increases with throughput

65. Copyright © 2005 HyPerformix, Inc.65 Simple Linear Analytical Model Constant CPU/BF is not a good predictor for HT server performance

66. Copyright © 2005 HyPerformix, Inc.66 Non-Hyper-Threaded Application Server IPS Optimizer Modeled utilization tracks nicely with measurement

67. Copyright © 2005 HyPerformix, Inc.67 Hyper-Threaded Application Server IPS Optimizer, without Hyper-Threading Simulation Modeled does not represent the increase in CPU/BF as load increases

68. Copyright © 2005 HyPerformix, Inc.68 Hyper-Threaded Application Server IPS Optimizer, with Hyper-Threading Simulation HT model results track measurements

69. Copyright © 2005 HyPerformix, Inc.69 Summary - Extrapolating to Production Can the model be used to extrapolate? –Yes! –The model’s predicted utilizations are good –Response time predictions tracked with the load test Are the model predictions good enough to –Guide the load testing effort? Yes! –Make a business decision about production deployment? Yes! Good enough!

70. Copyright © 2005 HyPerformix, Inc.70 Summary – Moving Beyond Test and Guess Load Testing is essential to ensuring application performance-readiness Load Testing has it’s limits: time and money Performance modeling improves the effectiveness of Load Testing Modeling has minimal impact to the Load Test effort Modeling & Load Testing are natural partners