Automated Capacity Planning and Support for Enterprise Applications Dharmesh Thakkar School of Computing, Queen’s University, Canada

Performance Activities in The Waterfall SDLC Verification: Performance test need to be conducted to verify that the required performance objectives are met 2

Performance Activities in SDLC Maintenance: Continuous monitoring, capacity planning and tuning need to be done to ensure that the system continues to meet the performance objectives 3

Overview of Thesis Customers engage the vendor’s support analyst to resolve the issues with system Performance issues arise in customers’ production environment, which need troubleshooting The capacity calculator helps customer determine required hardware resources to support the anticipated workload Build a capacity calculator based on the performance model Challenge: The goal of performance testing is not to find bugs, but to establish benchmarks 4 Challenge: The support analyst needs to troubleshoot the customer deployment Challenge: Load test and stress test workloads need to be compared against the customer workloads Modify the workload or the deployment scenario to put excessive stress on one or more system resources Run the application for long periods under high workloads

Overview of Thesis: Our Contributions A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China. Acceptance rate: 26% 5 A framework for building performance model based capacity calculator Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ. Acceptance rate: 38%

Verification Phase: An Example of A Capacity Calculator 6

Frequency (transactions per hour) Concurrency 250 Creating Customer ProfileT111T112T113T114 Customer LoginT121T122T123T124 Search TitleT131T132T133T134 Purchase TitleT141T142T143T144 Concurrency 500 Creating Customer ProfileT211T212T213T214 Customer LoginT221T222T223T224 Search TitleT231T232T233T234 Purchase TitleT241T242T243T244 Concurrency 1000 Creating Customer ProfileT311T312T313T314 Customer LoginT321T322T323T324 Search TitleT331T332T333T334 Purchase TitleT341T342T343T344 Concurrency 1500 Creating Customer ProfileT411T412T413T414 Customer LoginT421T422T423T424 Search TitleT431T432T433T434 Purchase TitleT441T442T443T444 Building a Measurement based Capacity Calculator Frequency (transactions per hour) Concurrency 250 Creating Customer ProfileT111T112T113T114 Customer LoginT121T122T123T124 Search TitleT131T132T133T134 Purchase TitleT141T142T143T144 7

Performance Modeling Challenges ■ Large number of tests to be executed ■ Limited time for performance modeling ■ Manual process of setup, execution and analysis ■ Risk of error and repeating the full modeling process 8

Addressing the Challenges of Performance Modeling ■ We created the framework to: –Automate the process –Speed up the process –Focus research on specific areas 9

Framework for Building Model based Capacity Calculator ■ Case Studies: –Dell DVD Store –A large enterprise application 10

Customization Cost Minimal Another Build Extensive Reasonable Extensive Reasonable Minimal Another Application 11

Maintenance Phase: Customer Support 12

Retrieving Relevant Reports ■ State of Practice: –No systematic techniques to retrieve and use information for future engagements –Keyword searching is limited: depends on the search skills and experience of the analyst and peculiarity of the problem 13

Customer Support Problem Statement ■ We want to find customers with similar operational and problem profiles ■ We can reuse prior solutions and knowledge  Heavy , Light Web, Light calendar  Light , Light Web, Light calendar  Light , Heavy Web, Light calendar  Heavy , Heavy Web, No calendar  Light , Light Web, Heavy calendar Other Customers Compare New Customer Engagement 14

Using Logs for Customer Support ■ Execution logs are readily available and contain –Operational Profile: usage patterns (heavy users of from device, or to device, or light users of calendar, etc.) –Signature Profile: specific error line patterns (connection timeout, database limits, messages queued up, etc.) ■ Find the most similar profile 15

Execution Logs ■ Contain time-stamped sequence of events at runtime ■ Readily available representatives of both feature executions and problems Queuing new mail msgid=ABC threadid=XYZ Instant message. Sending packet to client msgid=ABC threadid=XYZ New meeting request msgid=ABC threadid=XYZ Client established IMAP session id=ABC threadid=XYZ Client disconnected. Cannot deliver msgid=ABC threadid=XYZ New contact in address book id=ABC threadid=XYZ User initiated appointment deletion id=ABC threadid=XYZ 16

Example Other Customers Compare 17

Technique to Retrieve Relevant Engagement Reports 18

Case Studies ■ Case Study I –Dell DVD Store open source application –Code instrumentation done for event logging –Built the execution log repository by applying synthetic workloads, changing the workload parameters each time ■ Case Study II –Globally deployed commercial application –More than 500 unique execution events 19

Case Study Results ■ Dell DVD Store –100% precision and recall on both operational profile based and signature profile based retrieval ■ Commercial Application –100% precision and recall for signature profile based retrieval –Results for operational profile based retrieval: Experiment Count of Log Files K-L DistanceCosine Distance PrecisionRecallPrecisionRecall Single Feature Group %90.28%67.71%90.28% Multiple Feature Groups %80.95%75.00%100.00% All Feature Groups %97.22%62.50%83.33% Real World Log Files %72.22%68.75%91.67% All the Log Files %79.90%56.72%75.62% 20

Sources of Errors ■ Events that do not correspond directly to a particular operational feature, such as idle time events, server health check events, startup and shutdown events ■ Imbalance in the event logging 21

Imbalance in Event Logging 22

Summary: Our Contributions A framework for building performance model based capacity calculator Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ. A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China. 23

Details: Test Enumeration Frequency (transactions per hour) Concurrency 250 Creating Customer ProfileT111T112T113T114 Customer LoginT121T122T123T124 Search TitleT131T132T133T134 Purchase TitleT141T142T143T144 Concurrency 500 Creating Customer ProfileT211T212T213T214 Customer LoginT221T222T223T224 Search TitleT231T232T233T234 Purchase TitleT241T242T243T244 Concurrency 1000 Creating Customer ProfileT311T312T313T314 Customer LoginT321T322T323T324 Search TitleT331T332T333T334 Purchase TitleT341T342T343T344 Concurrency 1500 Creating Customer ProfileT411T412T413T414 Customer LoginT421T422T423T424 Search TitleT431T432T433T434 Purchase TitleT441T442T443T444 24

Environment Setup and Test Execution ■ Setup Environment and Test ■ Run Test ■ Shutdown Test 25

Test Transition ■ Conduct tests –Back to back –Add a delay –Use heuristics 26

Test Analysis ■ Validate the Test –Check for bugs or run problems ■ Analyze Performance Counters –Prune the warm-up and cool-down periods 27

Model Building ■ Create models using –Regression techniques –QN and other stochastic models 28

Test Reduction 29

Summary: Verification Phase 30

Log Lines to Event Distribution ■ Remove dynamic information –Example: Given the two log lines “Open inbox user=A” and “Open inbox user=B”, map both lines to the event “Open inbox user=?” ■ Use event percentages to compare event logs for different running lengths without bias 31

Compare Event Distributions ■ Kullback-Leibler Divergence ■ Cosine Distance 32

Identify Signature Events ■ Signature Events have a different frequency when compared to events in other log files –Example signature events: dropped connections, thread dumps, and full queues ■ Chi-square test identifies such events 33

Measuring Performance ■ Precision = 2/4 = 50% 100% precise if all the retrieved log files are relevant ■ Recall = 2/3 = 67% 100% recall if all the relevant log files are retrieved 34

The Big Picture 35

Summary: Maintenance Phase 36

Related Work ■ Data mining techniques on textual information [Hui and Jha, 2000] –Cons: Limited results, depending on analyst’s search skills and peculiarity of the problem ■ Using customer usage data [Elbaum and Narla, 2004] –Cons: Customer usage data rarely exists ■ Clustering HTTP execution logs [Menascé, 1999] –Cons: Complex process, works only for HTTP logs ■ Software Agent Deployment to build operational profile [Ramanujam et. al., 2006] –Cons: Intrusive, complex, costly 37

Software Development Life Cycle of Enterprise Applications 38

Performance Activities in SDLC Requirements: Performance objectives need to be clearly identified. Realistic, measureable and verifiable 39

Performance Activities in SDLC Design: Platform and technology selection should be done that would support the stated performance requirements. 40

Performance Activities in SDLC Implementation: Algorithmic, data structure, and programming decisions need to be made that would optimize the performance 41