1. Page 1 © 2001 Hewlett-Packard Company Tools for Measuring System and Application Performance Introduction GlancePlus Introduction Glance Motif Glance Character Mode Glance Command-line Xverbosegc HPjmeter and –Xeprof Other Tools gdb

2. Page 2 © 2001 Hewlett-Packard Company GlancePlus Motif /opt/perf/bin/gpm

3. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 3 GlancePlus Motif Open multiple windows to simultaneously view: Machine’s behavior Process’s behavior Thread behavior We will learn how to use it with Java applications

4. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 4 GlancePlus Motif Main Window – Overview of System

5. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 5 GlancePlus Motif Main Window – Overview of System CPU I/O Memory Network

6. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 6 GlancePlus Motif Configure Sampling Time

7. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 7 GlancePlus Motif Configure Sampling Time Configure:Measurement Sample Interval: Minutes:0 Seconds:1 Graph Points: 50

8. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 8 GlancePlus Motif Main Window – CPU Usage Active Button: CPU Graph

9. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 9 GlancePlus Motif CPU Graph

10. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 10 GlancePlus Motif CPU Graph CPU Queue Length 8 CPUs CPU Usage Categories: Real Negative Nice (-20 to 0) Nice (greater than 0) Normal User System

11. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 11 GlancePlus Motif CPU Graph – Garbage Collection 1 Thread Active during GC

12. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 12 GlancePlus Motif Main Window - Reports

13. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 13 GlancePlus Motif Main Window - Reports:Process List Reports:Process List

14. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 14 GlancePlus Motif Process List

15. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 15 GlancePlus Motif Process List 8 CPU System: 800% Maximum CPU%

16. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 16 GlancePlus Motif Process List – Arrange Columns

17. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 17 GlancePlus Motif Process List – Arrange Columns Configure:Arrange Columns

18. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 18 GlancePlus Motif Process List – Arrange Columns

19. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 19 GlancePlus Motif Process List – Arrange Columns Interesting values: CPU% User CPU% System CPU% Virtual Memory Res(ident) Memory Physical I/O Configure:Choose Metrics: PROC_THREAD_COUNT

20. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 20 GlancePlus Motif Process List - Reports

21. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 21 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

22. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 22 GlancePlus Motif Process System Calls

23. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 23 GlancePlus Motif Process System Calls Sorted on: SysCall Rate

24. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 24 GlancePlus Motif Process System Calls - Sort Configure:Sort Fields

25. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 25 GlancePlus Motif Process System Calls - Sort

26. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 26 GlancePlus Motif Process System Calls - Sort

27. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 27 GlancePlus Motif Process System Calls System Calls for Idling JVM

28. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 28 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

29. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 29 GlancePlus Motif Process Memory Regions

30. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 30 GlancePlus Motif Process Memory Regions VSS – Virtual Set Size Total Reserved Space RSS – Resident Set Size Space Actively in Use Total Number of Memory Regions

31. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 31 GlancePlus Motif Process Memory Regions Data == C Heap Text == Executable Other == Java Heap Shared Libraries Thread Stacks Stack == For 1 st Thread

32. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 32 GlancePlus Motif Process Memory Regions - Sort

33. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 33 GlancePlus Motif Process Memory Regions - Sort

34. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 34 GlancePlus Motif Process Memory Regions - Sort Sort on VSS

35. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 35 GlancePlus Motif Process Memory Regions - Metrics Add Page Size Metrics

36. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 36 GlancePlus Motif Process Memory Regions - Metrics

37. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 37 GlancePlus Motif Process Memory Regions - Metrics Select All PROC_REGION_PAGE_COUNT*

38. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 38 GlancePlus Motif Process Memory Regions - Metrics For Address: Select PROC_REGION_VIRT_ADDRS

39. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 39 GlancePlus Motif Process Memory Regions – Pages Page Sizes Allocated In Java Heap All: 4 KB Too Many Total!!

40. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 40 GlancePlus Motif Process Memory Regions – Pages Page Sizes Allocated In Java Heap All: 4 MB Very Few Total!!

41. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 41 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

42. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 42 GlancePlus Motif Process Thread List

43. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 43 GlancePlus Motif Process Thread List Active Threads JVM Threads Total Number of Threads Thread Id

44. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 44 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

45. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 45 GlancePlus Motif Process Open Files

46. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 46 GlancePlus Motif Process Open Files File Type Sockets – Monitor Bytes Read and Written! File Name File Offset Total Number of Open Files

47. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 47 GlancePlus Motif Main Window - Reports System Info: System Tables Graph

48. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 48 GlancePlus Motif System Tables Graph Process Table File Table Shared Memory Table Semaphore Table File Locks Swap Space

49. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 49 GlancePlus Motif Identifying Performance Problems Patterns to recognize High System CPU time –Often associated with monitor contention –Check by looking at System Calls –High sched_yield, ksleep, kwakeup call rate confirms Increasing memory usage –Thread stacks –C Heap –Percentage of the Java Heap in active use

50. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 50 HotSpot JVM Object Model Efficient and low overhead header non-static fields method table ptr static fields GC maps header C++ vtable Class Object

51. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 51 HotSpot JVM Object Header Age (7 bits) Hash (23 bits) Lock (2 bits) lock unlock pthread mutex used (inflated) used by mark/sweep LockHashAge

52. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 52 HotSpot JVM Efficient, Low-Overhead Locking Stack header non-static fields Object U L header L If another thread tries to obtain the same monitor: Lock inflation (uses a pthread mutex)

53. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 53 How Does a pthread mutex Work? Try to get lock immediately If fail, be optimistic: – Enter queue and wait – Repeat: spin, sched_yield –Next waiter given lock Until: not optimistic – ksleep Has 2 Waiters

54. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 54 Contention: System Impact Monitors Implemented using a single word in memory that serves as the “lock word” On a system with multiple CPUs: All of the caches on the individual CPUs must be updated during the spin-sched_yield() cycle Consequences: Threads time is spent accessing one word – spinning –Result: High CPU usage with little application work Machine spends all of its time maintaining cache coherency!! More threads yield lower performance!

55. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 55 Contention: System Impact CPU 1 Cache CPU 2 Cache CPU 3 Cache CPU 4 Cache Memory thread 1 thread 3 thread 4 thread 5 thread 7 thread 6 thread 8 Cache Coherency LDCW

56. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 56 GlancePlus Motif Example of High Contention

57. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 57 GlancePlus Motif Example of High Contention HIGH SYSTEM CPU TIME LOW USER CPU TIME

58. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 58 GlancePlus Motif Example of High Contention

59. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 59 GlancePlus Motif Example of High Contention HIGH sched_yield() ksleep() kwakeup() CALL RATES

60. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 60 GlancePlus Motif Example of High Contention User System

61. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 61 GlancePlus Motif Example of High Contention - Fixed

62. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 62 GlancePlus Motif Example of High Contention

63. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 63 GlancePlus Motif Example of High Contention - Fixed 2 Fold Improvement! 5 Fold Improvement! 100x Reduction!

64. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 64 GlancePlus Motif Monitoring Memory Usage Select process ID for process in Process window and select Report:Memory Regions Data VSS (Virtual Set Size - allocated) represents the C heap Text VSS represents memory for the executable Other VSS represents the memory used by shared libraries, thread stacks, Java Heap, Code cache, … Private VSS (part of Other) contains the Java Heap and Thread stacks Corresponding RSS (Resident Set Size) for each area can also be monitored Watch for significant growth in any of these regions Use the Glance adviser syntax for long term monitoring

65. Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 65 GlancePlus Motif Conclusions Motif version is a powerful tool Exposes problems with Java performance Easy to do systematic analysis using displayed information