Page 1 © 2001 Hewlett-Packard Company Tools for Measuring System and Application Performance Introduction GlancePlus Introduction Glance Motif Glance Character Mode Glance Command-line Xverbosegc HPjmeter and –Xeprof Other Tools gdb

Page 2 © 2001 Hewlett-Packard Company GlancePlus Motif /opt/perf/bin/gpm

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 3 GlancePlus Motif Open multiple windows to simultaneously view: Machine’s behavior Process’s behavior Thread behavior We will learn how to use it with Java applications

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 4 GlancePlus Motif Main Window – Overview of System

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 5 GlancePlus Motif Main Window – Overview of System CPU I/O Memory Network

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 6 GlancePlus Motif Configure Sampling Time

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 7 GlancePlus Motif Configure Sampling Time Configure:Measurement Sample Interval: Minutes:0 Seconds:1 Graph Points: 50

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 8 GlancePlus Motif Main Window – CPU Usage Active Button: CPU Graph

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 9 GlancePlus Motif CPU Graph

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 10 GlancePlus Motif CPU Graph CPU Queue Length 8 CPUs CPU Usage Categories: Real Negative Nice (-20 to 0) Nice (greater than 0) Normal User System

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 11 GlancePlus Motif CPU Graph – Garbage Collection 1 Thread Active during GC

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 12 GlancePlus Motif Main Window - Reports

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 13 GlancePlus Motif Main Window - Reports:Process List Reports:Process List

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 14 GlancePlus Motif Process List

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 15 GlancePlus Motif Process List 8 CPU System: 800% Maximum CPU%

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 16 GlancePlus Motif Process List – Arrange Columns

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 17 GlancePlus Motif Process List – Arrange Columns Configure:Arrange Columns

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 18 GlancePlus Motif Process List – Arrange Columns

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 19 GlancePlus Motif Process List – Arrange Columns Interesting values: CPU% User CPU% System CPU% Virtual Memory Res(ident) Memory Physical I/O Configure:Choose Metrics: PROC_THREAD_COUNT

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 20 GlancePlus Motif Process List - Reports

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 21 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 22 GlancePlus Motif Process System Calls

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 23 GlancePlus Motif Process System Calls Sorted on: SysCall Rate

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 24 GlancePlus Motif Process System Calls - Sort Configure:Sort Fields

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 25 GlancePlus Motif Process System Calls - Sort

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 26 GlancePlus Motif Process System Calls - Sort

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 27 GlancePlus Motif Process System Calls System Calls for Idling JVM

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 28 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 29 GlancePlus Motif Process Memory Regions

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 30 GlancePlus Motif Process Memory Regions VSS – Virtual Set Size Total Reserved Space RSS – Resident Set Size Space Actively in Use Total Number of Memory Regions

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 31 GlancePlus Motif Process Memory Regions Data == C Heap Text == Executable Other == Java Heap Shared Libraries Thread Stacks Stack == For 1 st Thread

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 32 GlancePlus Motif Process Memory Regions - Sort

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 33 GlancePlus Motif Process Memory Regions - Sort

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 34 GlancePlus Motif Process Memory Regions - Sort Sort on VSS

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 35 GlancePlus Motif Process Memory Regions - Metrics Add Page Size Metrics

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 36 GlancePlus Motif Process Memory Regions - Metrics

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 37 GlancePlus Motif Process Memory Regions - Metrics Select All PROC_REGION_PAGE_COUNT*

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 38 GlancePlus Motif Process Memory Regions - Metrics For Address: Select PROC_REGION_VIRT_ADDRS

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 39 GlancePlus Motif Process Memory Regions – Pages Page Sizes Allocated In Java Heap All: 4 KB Too Many Total!!

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 40 GlancePlus Motif Process Memory Regions – Pages Page Sizes Allocated In Java Heap All: 4 MB Very Few Total!!

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 41 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 42 GlancePlus Motif Process Thread List

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 43 GlancePlus Motif Process Thread List Active Threads JVM Threads Total Number of Threads Thread Id

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 44 GlancePlus Motif Process List - Reports Process Resources Process Open Files Process Memory Regions Process System Calls Process Thread List

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 45 GlancePlus Motif Process Open Files

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 46 GlancePlus Motif Process Open Files File Type Sockets – Monitor Bytes Read and Written! File Name File Offset Total Number of Open Files

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 47 GlancePlus Motif Main Window - Reports System Info: System Tables Graph

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 48 GlancePlus Motif System Tables Graph Process Table File Table Shared Memory Table Semaphore Table File Locks Swap Space

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 49 GlancePlus Motif Identifying Performance Problems Patterns to recognize High System CPU time –Often associated with monitor contention –Check by looking at System Calls –High sched_yield, ksleep, kwakeup call rate confirms Increasing memory usage –Thread stacks –C Heap –Percentage of the Java Heap in active use

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 50 HotSpot JVM Object Model Efficient and low overhead header non-static fields method table ptr static fields GC maps header C++ vtable Class Object

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 51 HotSpot JVM Object Header Age (7 bits) Hash (23 bits) Lock (2 bits) lock unlock pthread mutex used (inflated) used by mark/sweep LockHashAge

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 52 HotSpot JVM Efficient, Low-Overhead Locking Stack header non-static fields Object U L header L If another thread tries to obtain the same monitor: Lock inflation (uses a pthread mutex)

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 53 How Does a pthread mutex Work? Try to get lock immediately If fail, be optimistic: – Enter queue and wait – Repeat: spin, sched_yield –Next waiter given lock Until: not optimistic – ksleep Has 2 Waiters

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 54 Contention: System Impact Monitors Implemented using a single word in memory that serves as the “lock word” On a system with multiple CPUs: All of the caches on the individual CPUs must be updated during the spin-sched_yield() cycle Consequences: Threads time is spent accessing one word – spinning –Result: High CPU usage with little application work Machine spends all of its time maintaining cache coherency!! More threads yield lower performance!

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 55 Contention: System Impact CPU 1 Cache CPU 2 Cache CPU 3 Cache CPU 4 Cache Memory thread 1 thread 3 thread 4 thread 5 thread 7 thread 6 thread 8 Cache Coherency LDCW

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 56 GlancePlus Motif Example of High Contention

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 57 GlancePlus Motif Example of High Contention HIGH SYSTEM CPU TIME LOW USER CPU TIME

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 58 GlancePlus Motif Example of High Contention

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 59 GlancePlus Motif Example of High Contention HIGH sched_yield() ksleep() kwakeup() CALL RATES

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 60 GlancePlus Motif Example of High Contention User System

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 61 GlancePlus Motif Example of High Contention - Fixed

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 62 GlancePlus Motif Example of High Contention

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 63 GlancePlus Motif Example of High Contention - Fixed 2 Fold Improvement! 5 Fold Improvement! 100x Reduction!

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 64 GlancePlus Motif Monitoring Memory Usage Select process ID for process in Process window and select Report:Memory Regions Data VSS (Virtual Set Size - allocated) represents the C heap Text VSS represents memory for the executable Other VSS represents the memory used by shared libraries, thread stacks, Java Heap, Code cache, … Private VSS (part of Other) contains the Java Heap and Thread stacks Corresponding RSS (Resident Set Size) for each area can also be monitored Watch for significant growth in any of these regions Use the Glance adviser syntax for long term monitoring

Tools for Measuring Performance © 2001 Hewlett-Packard Company Page 65 GlancePlus Motif Conclusions Motif version is a powerful tool Exposes problems with Java performance Easy to do systematic analysis using displayed information