1. Technical University of Darmstadt J2EE Performance and Scalability – From Measuring to Predicting Samuel Kounev Databases and Distributed Systems Group, TU-Darmstadt SPEC-OSG Java Subcommittee SPEC Benchmark Workshop 2006

2. Motivation MEASURING PREDICTING What performance would the application exhibit and how much hardware would be needed to meet SLAs? How do we ensure that the application design does not have any inherent scalability bottlenecks? Which platform would provide the best cost/performance ratio for a given application? How do we ensure that the selected platform does not have any inherent scalability bottlenecks? TU-Darmstadt © S. Kounev 2 J2EE applications becoming increasingly ubiquitous. Quality of service requirements of crucial importance! System architects and deployers faced with questions such as:

3. Roadmap & Agenda Will present a case study with SPECjAppServer2004. Benchmark used as a representative J2EE application. A systematic approach for performance prediction introduced. Agenda: Overview of SPECjAppServer2004 Approach and methodology for performance prediction Case study of SPECjAppServer2004 Validation of the approach Summary and conclusions TU-Darmstadt © S. Kounev 3

4. The SPECjAppServer2004 Benchmark Much more complex and substantially different from previous versions of SPECjAppServer A new enhanced workload exercising all major J2EE services including: The Web Container (Servlets and JSPs) The EJB Container (session beans and entity beans) Container-Managed Persistence JMS and Message Driven Beans Transaction Management Database Connectivity Exercises all parts of the underlying infrastructure including hardware, OS, JVM software, DBMS, JDBC drivers, system network TU-Darmstadt © S. Kounev 4

5. SPECjAppServer2004 Business Model TU-Darmstadt © S. Kounev 5

6. Sample Deployment Environment (Sun) 6

7. Approaches to Performance Prediction Educated Guess + Quick, easy and cheap. - Very inaccurate and risky. Performance Modeling + Often cheaper and quicker than load-testing. Could be applied at the design stage. - Extremely complex. Accuracy depends on how representative models are. Load Testing (brute force) + Accurate. Helps to identify bottlenecks and fine-tune system prior to production. - Expensive and time-consuming. Assumes system availability for testing. 7

8. Modeling Approach - Queueing Petri Net (QPN) models used - Combine queueing networks and Petri nets - Allow integration of queues into places of PNs - Ordinary vs. Queueing Places - Queueing Place = Queue + Depository -Benefits of using QPNs: Excellent modeling power and expressiveness. Can be exploited to accurately model both hardware and software aspects of system behavior (ISPASS03). The knowledge of the structure of QPNs can be exploited for efficient analysis by means of simulation (Performance Evaluation06). Can be used to combine qualitative and quantitative system analysis. TU-Darmstadt © S. Kounev 8

9. Performance Modeling Methodology 1.Establish performance modeling objectives. 2.Characterize the system in its current state. 3.Characterize the workload. 4.Develop a performance model. 5.Validate, refine and/or calibrate the model. 6.Use model to predict system performance. 7.Analyze results and address modeling objectives. TU-Darmstadt © S. Kounev 9 Performance Engineering of Distributed Component-Based Systems - Benchmarking, Modeling and Performance Prediction, © Shaker Verlag, 2006, ISBN: 3832247130.

10. Case Study: Modeling SPECjAppServer TU-Darmstadt © S. Kounev 10

11. Modeling Objectives Normal Conditions: 72 concurrent dealer clients (40 Browse, 16 Purchase, 16 Manage) and 50 planned production lines in the mfg domain. Peak Conditions: 152 concurrent dealer clients (100 Browse, 26 Purchase, 26 Manage) and 100 planned production lines in the mfg domain. Goals: Predict system performance under normal operating conditions with 4 and 6 application servers. Predict how much system performance would improve if the load balancer is upgraded with a slightly faster CPU. Study the scalability of the system as the workload increases and additional application server nodes are added. Determine which servers would be most utilized under heavy load and investigate if they are potential bottlenecks. TU-Darmstadt © S. Kounev 11

12. Performance Model 12

13. Performance Prediction TU-Darmstadt © S. Kounev 13

14. Performance Prediction (2) TU-Darmstadt © S. Kounev 14

15. 150 Browse Clients 200 Browse Clients TU-Darmstadt © S. Kounev 15 Performance Prediction (3)

16. 6. Predict System Performance (4) 16

17. 6. Predict System Performance (5) Sc.3: 300 B, 30 P, 30 M, 120 PL Max Error 16.8% Sc.4: 270 B, 90 P, 60 M, 120 PL Max Error 15.2% TU-Darmstadt © S. Kounev 17

18. Results Analysis TU-Darmstadt © S. Kounev 18

19. Summary and Conclusions TU-Darmstadt © S. Kounev 19 Studied a representative J2EE application and predicted its performance under realistic load conditions. The modeling error for response time did not exceed 21.2% and was much lower for throughput and utilization. QPN models can be exploited for accurate performance prediction in realistic scenarios. Proposed methodology provides a powerful tool for sizing and capacity planning. Performance prediction is hard, but its possible!

20. Acknowledgements TU-Darmstadt © S. Kounev 20 Kaivalya Dixit (in memoriam), SPECs Co-founder, Former SPEC President and Chair of the Java Subcommittee. All members of the OSG Java Subcommittee, eSPECially: John Stecher, IBM Tom Daly and Akara Sucharitakul, Sun Microsystems Ricardo Morin, Intel Steve Realmuto, BEA

21. Thank You for Your Patience Questions? TU-Darmstadt © S. Kounev 21 Enjoy your lunch!