1. 1 Best Practices for Performance Evaluation and Diagnosis of Java Applications Prashanth K Nageshappa Venkataraghavan Lakshminarayanachar IBM

2. 2 Agenda Inside a High Performance Java Virtual Machine (JVM) Performance Issues – Diagnosis Techniques The Healthcenter

3. 3 Inside a High Performance JVM

4. 4 DebuggerProfilersJava Application Code JVMTI SE5 Classes SE6 Classes Harmony Classes User Natives GCJITClass Library Natives Pluggable VM InterfacesJava Native Interface (JNI) Core VM (Interpreter, Verifier, Stack Walker) Trace & Dump Engines Port Library (Files, Sockets, Memory) Thread Library AIXLinuxWindowsz/OS PPC-32 PPC-64 x86-32 x86-64 PPC-32 PPC-64 390-31 390-64 x86-32 x86-64 390-31 390-64 Lifting the Hood Overall Architecture User Code VM Extensions Core VM Portability Layer Operating systems = User Code = Java Platform API = VM-aware = Core VM

5. 5 Java: Adaptive Compilation in J9/TR Methods start out being interpreted After N invocations (or via interpreter sampling) methods get compiled at ‘cold’ or ‘warm’ level Low overhead sampling thread is used to identify hot methods Methods may get recompiled at ‘hot’ or ‘scorching’ levels (for more optimizations) Transition to ‘scorching’ goes through a temporary profiling step cold hot scorching profiling interpreter warm

6. 6 Code Example public static int total = 55; public static int dummy(int i, int j, int N, int[] a) { int k = 0; for (i = 0; i < N; i++) { k = k + j + a[i] + (total + foo()); } return k; } public static int foo() { return 75; }

7. 7 Optimization and Effects Opt level Code Size (bytes) Compilation Time (us) Wall clock runtime (ms) Cold139226031685 Warm265490010078 Hot43689007765 Profiling13229000n/a Scorching57811,0006187

8. 8 Garbage Collection - Goals Tidying up… Fast allocation path – Large contributor to overall JVM performance. Low pause times and concurrent operation – Fit for purpose – different algorithms with different tradeoffs. Hardware exploitation – Multiple CPUs & varying memory architectures. – Algorithmic and processor parallelism. Accurate garbage collection – Earlier IBM JVMs did a ‘partially conservative’ GC, which was suboptimal.

9. 9 Compressed References > 32-bit Object (24 bytes – 100%) clazzflagsmonitor int field object field ClazzFlagsPadMonitor int field Padobject field ClazzFlagsMonitor int field object field > 64-bit Object (48 bytes – 50%) > 64-bit Compressed (24 bytes – 100%) > Use 32-bit values (offsets) to represent object fields With scaling, between 4 GB and 32 GB can be addressed > To enable the feature : -Xcompressedrefs

10. 10 Threading and Monitors Java uses monitors everywhere – Good – easy to use, safety built-in for many cases! – Bad – there’s a tax, even when there’s no contention. Central to performance in JVMs – Avoid it? Escape analysis (but remember JSR 133!). – Make it cheaper? Tasuki locks Lock reservations

11. 11 Bimodal lock – ‘thin’ or ‘inflated’ Single atomic operation (on enter) A Study of Locking Objects with Bimodal Fields (Tamiya Onodera & Kiyokuni Kawachiya, IBM Research, OOPSLA 1999) Lock Reservation: Java Locks Can Mostly Do Without Atomic Operations (Kiyokuni Kawachiya, Akira Koseki, Tamiya Onodera, IBM Research, OOPSLA 2002) Tasuki locks 0 0 1Inflated Monitor Thread ID 0 Unowned Thin owned Inflated owned

12. 12 Historical Perspective Is it just the hardware? We’ve come a long, long way… Why? –Processors – better control & understanding of the memory hierarchy –Language understanding (idiom recognition) –Processing budget (new instructions, more cores)

13. 13 SPECjbb Trademarks and Results SPEC and SPECjbb are registered trademarks of the Standard Performance Evaluation Corporation. Results referenced are current as of June, 2009. The SPECjbb2005 results are posted at www.spec.org, which contains a complete list of published SPECjbb2005 results. SPEC, SPECjbb reg tm of Standard Performance Evaluation Corporation. Data Pt Leap vs prev Accum Leap vs base JVMHardwareXeonChips Core (s) GHz SPECjbb 2005 bops SPECjbb 2005 bops/jvm wwwmath 1 base1.0JRockit R25.2Jun 05Dell PowerEdge SC1425DP223.624,208 link 2 5%1.05J9 5.0GAOct 05IBM eServer xSeries 346dual242.839,585 link1.11 – 1.06 = 1.05 3a 6%1.11HotSpot 5u5Dec 05FSC PRIMERGY TX300 S2DP223.828,314 link28314/24208*3.6/3.8=1.11 3bHotSpot 5u5Dec 05FSC PRIMERGY RX300 S2dual242.841,986 link41986/39585=1.06 417%1.3JRockit P26.0Mar 06FSC PRIMERGY TX300 S2dual242.849,233 link49233/41987=1.17; 1.11*1.17=1.30 5a 7%1.39JRockit P26.4Jun 06Dell PowerEdge 1850DP223.835,503 link35503/28314=1.25; 1.25*1.11=1.39 5bJRockit P26.4Jun 06FSC PRIMERGY BX620 S35160243.0100,407 link 614%1.59J9 5.0sr2July 06IBM System X 36505160243.0114,941 link114941/100407=1.14; 1.39*1.14=1.59 7a14%1.81JRockit P27.1Nov 06Dell PowerEdge 29505160243.0130,589 link130589/114941=1.14; 1.59*1.14=1.81 7bJRockit P27.1Nov 06Dell PowerEdge 29505355282.66210,065105,033link 84%1.88J9 5.0sr5Feb 07IBM System X 36505355282.66218,032109,016link218.032/210,065=1.04; 1.81*1.04=1.88 9a1%1.90JRockit P27.2Mar 07Dell PowerEdge 29505355282.66220,648110,324link220648/218032=1.01; 1.88*1.01=1.90 9bJRockit P27.2Aug 07Dell PowerEdge 29505365283.0238,47259,618link 10a6%2.01JRockit P27.4Nov 07Dell PowerEdge 29505365283.0252,40363,101link252403/238472=1.06;1.90*1.06=2.01 10bJRockit P27.4Nov 07Dell PowerEdge 2950 III5460283.16303,13075,783link 110.06%2.01HotSpot 6u5pFeb 08Sun fire X41505460283.16303,29775,824link303297/303130=1.0006 12a7%2.14J9 6sr1Mar 08IBM System X 36505460283.16323,17280,793link323172/303297=1.07; 2.01*1.07=2.14 12bJ9 6sr1Sep 08IBM System x36505470283.33330,60582,651link 134%2.23J9 6sr3Oct 08IBM System x36505470283.33344,43686,109link344436/330605=1.04; 2.14*1.04=2.23 14a7%2.38JRockit P28.0Mar 09FSC PRIMERGY RX200 S45470283.33368,03492,009link368034/344436=1.07; 2.23*1.07=2.38 14aJRockit P28.0Mar 09Cisco UCS B200-M15570282.93556,792278,396link 150.005%2.38HotSpot 6u14pMar 09Sun Fire X42705570282.93556,822278,411link556822/557/792=1.00005 169%2.58J9 6sr5Mar 09IBM BladeCenter HS225570282.93604,417151,104link604417/556822=1.09; 2.38*1.09=2.58

14. 14 Performance Issue : Diagnosis Techniques

15. 15 Debugging Performance Problems Four layers of deployment: – Operating System / Infrastructure – Java Runtime / Garbage Collection – Application Code – External Delays Simple process is to start at the bottom, and eliminate layers

16. 16 Infrastructure and Java Runtime Issues

17. 17 Application and External Issues

18. 18 MustGather

19. 19 “MustGather” Diagnostics Set of data requested by IBM Support initial problem diagnosis – Specified on a per-scenario basis Requests only the data relevant to the scenario – Specified on a per-platform basis Leverages OS specific tools and capabilities – Split into two parts: Setup: to be done before starting the Java application Gather: to be done when the problem has occurred Linked to from product support pages – Java: http://www.ibm.com/software/webservers/appserv/runtimes/support/ – WAS: http://www.ibm.com/software/webservers/appserv/was/support/

20. 20 System Resource Contention

21. 21 Resource Contention: Physical Memory Lack of physical memory will cause paging/swapping of memory Swapping is very costly for a Java process – Particularly affects Garbage Collection performance Garbage collection touches every point of memory in the process All memory therefore would need to be paged back in Leads to long “mark” and “sweep” phases of GC

22. 22 Resource Contention: CPU Insufficient CPU time availability will reduce performance – Normally surfaces when something periodically takes CPU time on the box, eg. Cron Jobs running batch applications Database backups

23. 23 System Resource Contention: Solutions Ensure there are enough resources! Where resource can contention occurs it is important to ensure the Java application has its pool of resources Isolation be achieved on some platforms using LPARs/WPARs/ Zones Otherwise move other applications onto separate machines

24. 24 Garbage Collection Performance

25. 25 Garbage Collection Performance GC performance issues can take many forms Definition of a performance problem is very user centric – User requirement may be for: Very short GC “pause” times Maximum throughput A balance of both First step is ensure that the correct GC policy has been selected for the workload type – Helpful to have an understanding of GC mechanisms Second step is to look for specific performance issues

26. 26 Object Allocation Requires a contiguous area of Java heap Driven by requests from: – The Java application – JNI code Most allocations take place in Thread Local Heaps (TLHs) – Threads reserve a chunk of free heap to allocate from Reduces contention on allocation lock Keeps code running in a straight line (fewer failures) Meant to be fast – Available for objects < 512 bytes in size Larger allocates take place under a global “heap lock” – These allocations are one time costs – out of line allocate – Multiple threads allocating larger objects at the same time will contend

27. 27 Object Reclamation (Garbage Collection) Occurs under two scenarios: – An “allocation failure” An object allocation is requested and not enough contiguous memory is available – A programmatically requested garbage collection cycle call is made to System.GC() or Runtime.GC() the Distributed Garbage Collector is running call to JVMPI/TI is made Two main technologies used to remove the garbage: – Mark Sweep Collector – Copy Collector IBM uses a mark sweep collector – or a combination for generational

28. 28 Global Collection Policies Garbage Collection can be broken down into 2 (3) steps – Mark: Find all live objects in the system – Sweep: Reclaim unused heap memory to the free list – Compact: Reduce fragmentation within the free list All steps are in a single stop-the-world (STW) phase – Application “pauses” whilst garbage collection is done Each step is performed as a parallel task within itself Four GC “Policies”, optimized for different scenarios – -Xgcpolicy:optthruputoptimized for “batch” type applications – -Xgcpolicy:optavgpauseoptimized for applications with responsiveness criteria – -Xgcpolicy:genconoptimized for highly transactional workloads – -Xgcpolicy:subpoolsoptimized for large systems with allocation contention

29. 29 Introduction to GCMV Garbage Collection and Memory Visualizer – Verbose GC data visualizer – Eclipse based tool available as plugin in ISA and as a standalone tool. – Parses and plots all verbose GC logs – Extensible to parse and plot other forms of input – Provides graphical display of wide range of verbose GC data values – Handles optthruput, optavgpause, and gencon GC modes – Has raw log, tabulated data and graph views and can save data to jpeg or.csv files (for export to spreadsheets)

30. 30 GCMV usage scenarios Investigate performance problems – Long periods of pausing or unresponsiveness Evaluate your heap size – Check heap occupancy and adjust heap size if needed Garbage collection policy tuning – Examine GC characteristics, compare different policies Look for memory growth – Heap consumption slowly increasing over time – Evaluate the general health of an application

31. 31 Application Code Performance

32. 32 The Healthcenter

33. 33 Evaluating Your Application through the Healthcenter Answers to.. – What is my Java application doing ? – Why is it doing that ? – Why is my application going so slowly ? – Is my application scaling well ? – Do we need to tune the JVM ? – Am I using the right options? Available from/as a part of – https://www.ibm.com/developerworks/java/jdk/tools/ – http://www.ibm.com/software/support/isa

34. 34 Health Center Overview

35. 35 Environment Subsystem Shows – Version information for the JVM – Operating system and architecture information for the monitored system – Process ID – All system properties – All environment variables

36. 36 – Shows all loaded classes – Shows classes loaded time – Visualizes classloading activity – Identifies shared classes – Makes recommendations Classes Subsystem

37. 37 GC Subsystem - Shows Used Heap (after collection) & GC pause times - Identify memory leaks - Provides tuning recommendations and analysis of GC data

38. 38 Locking Subsystem - Always-on lock monitoring - All lock usage is profiled such as lock request totals, blocking requests and hold times - Helps to identify points of contention that prevents the application from scaling

39. 39 Profiling Subsystem - Sampling based profiler - Instantly identifies hottest methods in an application - See full call stacks to identify where methods are being called from and what methods they call

40. 40 Features (New) I/O – Provides File open events – Provides File close events – Provides Details of files that are currently open Native Memory – Provides native memory usage of the process and system monitored – Does not provide a native memory perspective view for the z/OS® 31- bit or z/OS 64-bit platforms.

41. 41 Merci Grazie Gracias Obrigado Danke Japanese English French Russian German Italian Spanish Brazilian Portuguese Arabic Traditional Chinese Simplified Chinese Hindi Tamil Thai Korean Teşekkürler turkish Thank You

42. 42 Special notices © IBM Corporation 2010. All Rights Reserved. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. The following are trademarks of the International Business Machines Corporation in the United States and/or other countries: ibm.com/legal/copytrade.shtmlAIX, CICS, CICSPlex, DataPower, DB2, DB2 Universal Database, i5/OS, IBM, the IBM logo, IMS/ESA, Power Systems, Lotus, OMEGAMON, OS/390, Parallel Sysplex, pureXML, Rational, Redbooks, Sametime, SMART SOA, System z, Tivoli, WebSphere, and z/OS. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at ibm.com/legal/copytrade.shtml. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office Intel and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.