© 2006, National Research Council Canada © 2006, IBM Corporation Solving performance issues in OTS-based systems Erik Putrycz Software Engineering Group National Research Council Canada Marius Slavescu IBM Rational Automated Software Quality Division IBM Canada

© 2006, National Research Council, Canada © 2006, IBM Corporation 2 Outline Introduction: performance problems in OTS-based systems Iterative process for diagnosing problems Applying the iterative process with Eclipse TPTP Example Conclusions

© 2006, National Research Council, Canada © 2006, IBM Corporation 3 Outline Introduction: performance problems in OTS-based systems –Memory leaks –Inefficient code –Current tools and techniques –OTS-based systems context Iterative process for diagnosing problems Applying the iterative process with Eclipse TPTP Example Conclusions

© 2006, National Research Council, Canada © 2006, IBM Corporation 4 Memory Leaks All applications require to reserve memory –… but sometimes the application doesn’t release memory that is not used anymore Execution environment or runtime responsible for managing memory –C/C++: manual reservations –Java,.NET, most other VMs: runtime responsible of reserving and releasing memory

© 2006, National Research Council, Canada © 2006, IBM Corporation 5 Memory Leaks (2) Diagnostic –Problem noticeable only until there is not more physical memory available Swapping and paging occurs Solution –“Black box” component get the vendor to fix it or change the way the product is use –Open source component Instrument and locate source of the problem

© 2006, National Research Council, Canada © 2006, IBM Corporation 6 Inefficient code Inefficient algorithms: –unnecessary complexity in the algorithm (that can be replaced by less complex versions); Inefficient use of caching: –without caching where possible (e.g., database, network or files), heavy resource access can lead to slow execution; Expensive interactions with other products: –some interactions within the system may be expensive and lead to a High resource utilization, or long latencies (e.g., remote interactions used inappropriately).

© 2006, National Research Council, Canada © 2006, IBM Corporation 7 Current tools and techniques Focused for custom built software If source code is available: –test performance –locate the performance issue in the code –fix the problem and iterate If source code is not available –use the performance troubleshooting section of the manual (if available) –get a product expert to help identifying the cause

© 2006, National Research Council, Canada © 2006, IBM Corporation 8 OTS-based Systems Context Possible sources of the cause –integration code –one or more components Possible actions –change the concerned component(s) configuration –notify the vendor of the bug –replace a product –modify the way the product is used

© 2006, National Research Council, Canada © 2006, IBM Corporation 9 Outline Introduction: performance problems in COTS-based systems Iterative process for diagnosing problems –Objectives –Process –Alternatives Applying the iterative process with Eclipse TPTP Example Conclusions

© 2006, National Research Council, Canada © 2006, IBM Corporation 10 Objectives Currently –Product experts in charge of analyzing logs, finding the cause of the problem Iterative process –Solve a performance problem in a systematic manner

© 2006, National Research Council, Canada © 2006, IBM Corporation 11 Iterative process: 4 steps 1.Data collection: trace files are collected on OTS products; 2.Performance problem reproduction: the problem has to be repeated in order to capture it in the traces; 3.Trace correlation: the trace files collected are correlated to understand the causality of events in the system; 4.Trace analysis: the correlated traces may need further analysis to focus on the relevant information

© 2006, National Research Council, Canada © 2006, IBM Corporation Data Collection Start by the least intrusive method through the different OTS-components Usually tracing end-user response time Subsequent steps: –Get data more focused Possible methods: –Adding instrumentation to the integration code –Enable logging on other products

© 2006, National Research Council, Canada © 2006, IBM Corporation Performance problem reproduction Test system: –Load testing tool generate load –Automated testing framework scenario to reproduce the problem Production system: –wait until the problem reappears

© 2006, National Research Council, Canada © 2006, IBM Corporation Trace correlation Finding the causality of events in the different traces Reconstruct the causality –Traces timestamps –Logs with only time: Correlate by similar time –Logs with time + duration Find events that happened during another one

© 2006, National Research Council, Canada © 2006, IBM Corporation Trace analysis Statistical –Locate in the traces the most significant events –Group the most repeated ones –Calculate where time is spent Visual –Most common method –Graph or UML interaction diagrams Analysis of profiler traces –[CASCON04]: traces are collected of all the interactions between integration code and OTS-products –Find which interactions are the cause of the performance problem

© 2006, National Research Council, Canada © 2006, IBM Corporation 16 Alternatives solutions to the iterative process Online analysis –Enable diagnostic on production system without a cycle JMX –Java Management: interface to query the state of java virtual machines JRockit (BEA JVM) –Integrated diagnosis of memory leaks –Integrated profiler Existing solutions limited to specific types of performance problems Stability concerns on production systems

© 2006, National Research Council, Canada © 2006, IBM Corporation 17 Outline Introduction: performance problems in COTS-based systems Iterative process for diagnosing problems Applying the iterative process with Eclipse TPTP –TPTP Platform Project –Tracing and Profiling Tools Project –Monitoring Tools Project –OTS-based Systems Support Example Conclusions

© 2006, National Research Council, Canada © 2006, IBM Corporation 18 TPTP Platform Project: The Foundation 1)Reference UIs and perspectives –Basic metaphors for interacting with target systems and resources - includes both remote and local systems. –Extensible UI Frameworks and common navigators, viewers, editors, and wizards 2)Standard data models, and assets repository –Information model implementations for test, trace, log and statistical data. –Framework for running rule based queries against data model instances and some simple queries. 3)Common data collection and execution framework –Execution environment that supports deployment, launch, and control of test cases and applications –Data collection and control frameworks and agents. –Communication service which is used by the distributed data collection and control frameworks. Eclipse Reference UIs Execution Framework XMI Assets

© 2006, National Research Council, Canada © 2006, IBM Corporation 19 TPTP Tracing and Profiling Tools Project Frameworks for tracing and profiling tools by extending the TPTP platform. Profiling tools for both single- system and distributed Java applications. A JVMPI monitoring agent that collects trace and profile data. –Collects and analyzes heap and stack information –JVMTI-based monitoring agent. A generic tool kit for probe insertion - can instrument byte code of Java applications.

© 2006, National Research Council, Canada © 2006, IBM Corporation 20 TPTP Monitoring Tools Project Frameworks for building monitoring tools by extending the TPTP platform. Includes tools for monitoring application servers (JBoss, Jonas, and Websphere) and system performance. Collects, analyzes, aggregates, and visualizes data captured in the log and statistical models. Supports Common Base Event (CBE), provides services for mapping of custom log formats to CBE, and regular expression based log filtering. Correlates data across multiple instances of log and statistical models; also across instances of trace and test history models –Enables symptom and pattern analysis.

© 2006, National Research Council, Canada © 2006, IBM Corporation 21 OTS-based systems support

© 2006, National Research Council, Canada © 2006, IBM Corporation 22 Outline Introduction: performance problems in COTS-based systems Iterative process for diagnosing problems Applying the iterative process with Eclipse TPTP Example –Description –Applying the iterative process Conclusions

© 2006, National Research Council, Canada © 2006, IBM Corporation 23 Example

© 2006, National Research Council, Canada © 2006, IBM Corporation 24 Applying the iterative process Iteration 1: –Data collection: Tomcat: default logging doesn’t provide execution time -> customized to include processing time Mysql: Slow query log enabled -> all slow queries recorded Change the default value of 2 seconds –Performance problem reproduction –Trace correlation Use tomcat and MySQL’s timestamps and execution times Custom log extractors + parsers –Trace analysis Visual analysis with UML2 Interaction diagrams  problem located in the database  Solution: modify integration code

© 2006, National Research Council, Canada © 2006, IBM Corporation 25 Applying the iterative process (2) Iteration 2: –Data collection ProbeKit: Instrument the integration code –Performance problem reproduction: same as it 1. –Data correlation: same algorithm as it 1. –Trace analysis: locate the source of the problem inside the integration code

© 2006, National Research Council, Canada © 2006, IBM Corporation 26 Applying the iterative process (3)

© 2006, National Research Council, Canada © 2006, IBM Corporation 27 Outline Introduction: taxonomy of performance problems in COTS-based systems Iterative process for diagnosing problems Applying the iterative process with Eclipse TPTP Example Conclusions

© 2006, National Research Council, Canada © 2006, IBM Corporation 28 Conclusion Diversity of OTS components –new challenge for solving performance problems Existing tools and methods are targeted for custom software –More powerful methods are required Solution: –Solve performance problems in a systematic manner with an iterative process –At each iteration focus on a specific part Availability of generic tools for performance analysis –Automate and customize the analysis Example and TPTP available at