© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Understanding the AIX Performance Data in a PowerVM Partition Pete Weilnau Philadelphia CMG Feb 29, 2008

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Agenda  Review of PowerVM  Instrumentation  Understanding the Instrumentation  Putting it into practice

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Disclaimer  All trademarks referenced in this presentation are the properties of their respective companies.  Some diagrams are from IBM Redbook SG Advanced POWER Virtualization on IBM System p5: Introduction and Configuration

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) PowerVM  Available on System p5 and higher  A marketing umbrella term for a number of hardware and software technologies: Dynamic LPAR (DLPAR) Shared Processor Pool (Micro-partitioning) Virtual IO Server (VIO) Simultaneous Multi-Threading (SMT) Virtual LAN (VLAN) Integrated Virtualization Manager (IVM) Capacity on Demand (CoD) Live Partition Mobility (POWER6 only) Shared Dedicated Capacity (POWER6 only)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) System p overview AIX LPAR Linux LPAR VIO Server LPAR AIX LPAR Linux LPAR Hypervisor firmware HMC intranet remote admin

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) POWER Hypervisor  System firmware introduced w/ POWER5 always enabled (think “NO BASIC MODE”)  Key features Dynamic LPAR (DLPAR)  allows dynamic reallocation of resources ... but only if the OS supports it! shared processor pool (Micro-partitioning)  AIX 5.3 & Linux only (no i5/OS support)  allows fractional physical CPU entitlement virtual device support (VLAN, SCSI, consoles)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) POWER Hypervisor source: Advanced POWER Virtualization on IBM System p5 – SG

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Virtual IO Server  a software appliance that runs in a dedicated LPAR  as the name implies, it provides virtual IO services to other (“client”) LPARs virtual disks virtual Ethernet adapters shared Ethernet adapters (SEA)  Integrated Virtualization Manager (IVM) feature added w/ VIO Server 1.2 provides a browser-based subset of HMC function main restriction – all resources are owned by VIO, no dedicated resources

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Hardware Management Console (HMC)  hardware appliance to manage one or more managed systems (CECs)  includes partition management and many other management functions  provides optional web interface for remote access

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Web-based System Manager (WebSM)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Processor allocations

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) SMT and PURR source: Advanced POWER Virtualization on IBM System p5 – SG

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Physical, virtual, and logical processors source: Advanced POWER Virtualization on IBM System p5 – SG

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Focus of this research:  What can we learn about LPAR level activity from an AIX Point of View?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Let’s talk Instrumentation

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Old Faithfuls  iostat, vmstat, sar and time: Use PURR based statistics whenever SMT is enabled %user, %system, %iowait and %idle.  When executing on a shared-processor partition: pc or %physc represent % physical processor consumed. ec or %entc represent % of entitled capacity consumed.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) iostat example  What does %user, %sys, %idle, %iowait really represent in a shared-processor environment?  % of what? Isn’t it a moving target?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Old Faithfuls  Sar –p ALL: Use PURR based statistics whenever SMT is enabled %usr, %sys, %wio and %idle.  When executing on a shared-processor partition: physc represent % physical processor consumed. %entc represent % of entitled capacity consumed.  lcpu = shows number of logical CPUs  ent = shows entitlement

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Sar –P ALL example  What does that ‘U’ record represent?  ‘-’ normally shows the total

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) sar –P ALL  U row is reported if running in shared processor pool.  It shows the UNUSED system-wide capacity. System-wide = LPAR.  If in uncapped mode: The individual processor rows (phyc) are calculated against physical consumption  Unused is computed against entitlement and not physical consumption. Source: man sar

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Old Faithfuls  mpstat -s: Displays both virtual and logical processors  proc0 – represents a Virtual CPU  cpu0 and cpu1 represent logical CPUs

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Other Commands  Mpstat –a Provides a lot of detail info about processor activity, well beyond the scope of this presentation. cpu0 and cpu1 represent logical CPUs

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) lparstat - New  lparstat: Shows configuration and performance information for the partition in which it is run.  4 Modes: lparstat :monitoring mode lparstat –h:hypervisor summary lparstat –H:hypervisor calls lparstat –i:Configuration info

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) lparstat Psize = processors in shared pool Lbusy = percent occupation of the logical CPUs at the system and user level Vcsw = virtual context switches Phint = phantom interrupts that the partition received

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) lparstat -h Adds: %hypv – the amount of time spent in the Power hypervisor. hcalls – the number of calls AIX made to the Power hypervisor (per sec) Note: requires root access

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) lparstat -H Note: requires root access

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) lparstat -i

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)  So far all the instrumentation discussed has been related to the current LPAR  Is there a way to see other LPARs?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) topas - cecdisp  topas: Is a real time monitor with significant capability -cecdisp has been added for LPAR support

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) topas -cecdisp

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) topas –cecdisp - Another

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Understanding the Instrumentation

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Experiment

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) stress –c 1 –t 60 & One thread pushes a physical cpu to 100% and we’ve hit 100% of our entitlement. Note: physc is shown in processors, not a percentage. Note the lbusy is only 50%, meaning only 1 (of the 2) logical cpus is being used. Note that SMT mode doesn’t limit the ability to fully consume a physical CPU. App = the available physical processors in the shared pool.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) stress –c 2 –t 60 & 2 threads still produces physical CPU = 100% and we’ve still hit 100% of our entitlement. Note the lbusy is now 100%, meaning both logical CPUs are now saturated. SMT is allowing 2 logical CPUs to run. In an uncapped environment, why didn’t it get another physical processor involved?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Let’s give this LPAR another VCPU

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Rerun stress –c 1 –t 60 Note we now have 4 logical CPUs. Entitlement of 1.00 has not changed. But we now have 2 more logical CPUs (smt=On) When we run stress with 1 thread we still get 100% of a physical CPU Entitled capacity is also at 100% which. But note lbusy is now at 25%.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Rerun stress –c 2 –t 60 For the first time we have 2 physical processors saturated. Entitlement has finally exceeded 100%! lbusy is 50% What will happen when we go to 3 threads?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Rerun stress –c 3 –t 60 Still have 2 physical processors saturated. Entitlement still 200. lbusy is now 75. What’s holding us back?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Let’s add yet another VCPU…

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Test 3 Threads again… Want to make a prediction? Physical consumption goes to 3 physical processors. lbusy is now 50. Note we now have 6 logical CPUs. What will happen when we go to 4 threads?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Stress –c 4 –t 60 Physical consumption and entitlement do not change. At 3 threads we were getting all the resource that was available. With 4 threads we are simply slicing the same resources across an additional thread, so no more work can actually get done.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Putting it into practice

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Before PowerVM support oprah AIX 5.3 harpo AIX 5.3 karl AIX 5.3 marx VIO Server groucho Linux (RedHat) gummo Linux (SuSE) System Viewed

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) CPU utilization If this was running on a MicroPartition, what does 100% represent?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Process Utilization Same Question

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Before PowerVM Support oprah AIX 5.3 harpo AIX 5.3 Karl AIX 5.3 marx VIO Server groucho Linux (RedHat) gummo Linux (SuSE)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) With PowerVM Support oprah AIX CPU Dedicated harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared Shared Pool

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Demo Environment oprah AIX CPU Dedicated harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared Shared Pool System Viewed

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) LPAR Configuration Info

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared oprah AIX CPU Dedicated Shared Pool What can we see about the CEC? Physical view

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Physical CPU utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Shared pool utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) oprah AIX CPU Dedicated harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared Shared Pool Shared processor pool What about LPARs competing for the Shared Pool?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Now we see a very different view of harpo’s real CPU usage! Shared pool utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Memory allocation

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Context switches

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) oprah AIX CPU Dedicated harpo AIX 5.3 Shared karl AIX 5.3 Shared marx VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared Shared Pool Dedicated LPARs Can I see Dedicated LPARs?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Dedicated CPU utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) oprah AIX CPU Dedicated harpo AIX 5.3 Shared karl AIX 5.3 Shared marx AIX 5.3 VIO Server Shared groucho Linux (RedHat) Shared gummo Linux (SuSE) Shared Shared Pool LPAR details What about the LPAR that is the PerfMan managed system?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Physical CPU utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) Virtual CPU

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) In Summary  PowerVM is a sophisticated virtualization platform modeled after zSeries partitioning.  Respectable instrumentation is available from an AIX guest, but it requires a knowledgeable consumer.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM) References  Advanced POWER Virtualization on IBM System p5: Introduction and Configuration (IBM Redbook SG )  IBM System p Advanced POWER Virtualization Best Practices (IBM Redpaper)  System p Logical Partitioning Guide (SA )