1. IBM PowerVP - Overview and Demo
Joe Searcy
Sr. Systems Engineer, Trading Systems
2014

2. Agenda Overview Using PowerVP What is PowerVP GUI Overview
How do I obtain PowerVP
What are the requirements
PowerVP Architecture
PowerVP Security
Setup & Install
Install GUI
Install Agents
Start & Stop Agents
Updates
Using PowerVP
GUI Overview
Hardware Drill Down
Partition Drill Down
Recording
Closing
Links
Q&A

3. OVERVIEW

4. Overview: What is PowerVP?
SERVER
AGENT
GUI
PowerVP is a tool consisting of software based agents that are installed within LPAR’s to feed data to a Java based utility that graphically shows various performance related details of a single PowerSystems server ecosystem.
Highlights
Optimize your virtualized IBM® Power Systems™
Understand current workloads performance from virtual machine to physical hardware
Simple real-time graphical performance information highlighting overcommitted resources
Ability to replay saved performance data to analyze historical performance data
Accelerate the resolution of performance issues
Real-time continuous monitoring user interface
Proactively address performance for virtualized workloads

5. Overview: How do I obtain PowerVP
Steps for downloading updates:
1. Navigate to
2. Click Find product.
3. Begin typing PowerVP in the Product selector field.
4. Select PowerVP from the list. For Installed version, select All.
5. Identify and select the fix that is required.
6. Download the fix.
7. Open the download document and follow the link in the "Download Package" section.
Latest version to date: Service Pack 1

6. Overview: What does PowerVP require
PowerVP is included in the PowerVM Enterprise Edition, current and new PowerVM Enterprise Edition customers are entitled for PowerVP. PowerVM Standard Edition clients can purchase PowerVP Standard Edition.
Prequisites
Hardware
Power 7
Power 7+
Power 8
Firmware
On Power7 systems the firmware level of 770 and 780 or later is required
Operating System
Supported Release Levels
AIX 6.1 & 7.1
VIOS – All versions
IBMi 7.1 TR6 and 7.2 (both with PTF S152700, MF58478, MF58480)
RHEL 6.4 and later
SLES 11 SP3 and later
*Supported models as of
Java
The GUI requires Java JRE version
HMC
The “Allow Performance Information Collection” must be enabled for a logical partition that is running System Level Agent
Network
port for non-SSL and for SSL traffic by default. The port numbers can be customized to fit the security needs of the customer. The requirement is for the System Level Agent and the Partition Level Agent to use the same port numbers.
PowerVP Version 1.12 is the minimum version support
The firmware levels below 770 lack the new instrumentation required for Power
File system Requirements
PowerVP has a small footprint and is installed in /opt/ibm/powervpon UNIX systems, on IBM it is installed in QLSE library. Configuration files are installed in /etc/opt/ibm/powervp on UNIX and on IBM i the configuration files are installed in /QIBM/UserData/PowerVP. It is necessary to ensure that the file systems are not full.The disk space required for GUI client is dependent on the usage of background performance data collection feature of PowerVP.VP on POWER8 systems

7. Overview: PowerVP Architecture
System Level Agent
PowerVP requires a designated partition on a Power System to be utilized for the collection of system wide resource utilization data across a frame or central electronics complex. For the designated partition set the LPAR properties to allow performance information collection
Partition Level Agent
In order to view the partition specific information, PowerVP requires an agent to be installed on the partitions, this agent is called a Partition Level Agent. The Partition Level Agent registers with System Level Agent, hence the System Level Agent must be up and running before installing the Partition Level Agent.
Graphical User Interface
The GUI client is a Java based application using Java Swing. The client is available for multiple platforms including, Windows, AIX, Linux, MacOSX and Solaris or any platforms that can run Java applications. The GUI client connects to the System Level Agent, the client must be able to connect to the port on which the System Level Agent is listening (default is for SSL and for non-SSL).
Background Recorder
For recording system level data without using GUI a Java jar file named powervp.jar is available. The jar file can be launched from any server that can connect to the System Level Agent. Currently only system level data can be recorded using the command line option. Partition Level Agent recording can only be done using the client GUI.
The System Level Agent uses kernel extension module on UNIX systems to make privileged hcalls (hypervisor call) to the hypervisor. On IBM i, the API by the way of XPF and SLIC (System Licensed Internal Code) interfaces with the hypervisor. After the System Level Agent makes this connection with the hypervisor, the agent requests performance counters from the hypervisor as needed. There can be multiple System Level Agents installed on a Frame, it is advisable to use two System Level Agents installed for redundancy.
The Partition Level Agents use partition APIs to retrieve Ethernet and Disk I/O utilization data and also access Performance Monitoring Unit (PMU) to get hardware thread information. On AIX and VIOS the PMU is accessed using Performance Monitor API (pmapi). On IBM i PEX is used to get PMU data. On Linux systems Partition Level Agent uses perf stat commands to obtain performance information.
The System Level Agent will also act as partition agent for the partition it is running on. There is no need for additional partition level agent.
The GUI can be installed on multiple clients and multiple clients can connect to view the data simultaneously; however, multiple client connections will add load to the system as they all will be requesting performance data every second (default refresh time).

8. Overview: PowerVP Security
The PowerVP GUI will connect to the System Level Agent using a valid user name and password on the partition. When accessing partition information using drill down, or when using the Monitor LPAR option, the client GUI is connecting to the Partition Level Agent to access the data.
There are two options to authenticate to the Partition Level Agent:
1. Using the System Level Agent to authenticate the connection to the partition. This is the default option, after the client GUI is authenticated by the System Level Agent, the user will be able to drill down into the partition. In this method there is no need to provide authentication for signing on to the Partition Level Agent
2. The second method is to provide valid user and password details for the GUI client to connect to the Partition Level Agent
PowerVP provides both secure and non-secure options for connection between the agents and the client GUI. Even though user and password information is not transmitted in plain text in non-secure mode, the rest of the data is transmitted in plain text. The secure option uses SSL protocol for communications between the PowerVP components. To use this option, each PowerVP agent will need a SSL certificate.

9. SETUP & INSTALL

10. Setup & Install: Install GUI
The PowerVP installer is a GUI with an easy to follow menu-driven dialog. The PowerVP GUI client is installed by default on the system where the installer is run. The PowerVP installer and the client can be on Windows, Linux or AIX systems
To install PowerVP, mount the install media and change the appropriate directory for your operating system. The two executable files in the directory are PowerVP and PoweVP_IBMi_Agent. The PowerVP executable must be used to perform the install. It is advised to run the install from the media itself as there are other directories on the install media that are required by the installer (example: license directory).

11. Setup & Install: Install GUI
When the PowerVP Installer is executed, after language selection and acceptance of license agreement, you are prompted to choose the type of install to perform. There are three install options:
Typical
This option will install PowerVP client and the agents.
PowerVP Client GUI
This option installs only the client GUI, useful for users who have PowerVP agents already installed.
PowerVP Server Agents
This option installs the PowerVP agents only, useful for users who have the PowerVP client GUI installed and are adding new LPARs to the PowerVP monitoring.
Typical – Where you want to start. Install GUI and agents
PowerVP Client GUI – Secondary workstations after you have the agents installed and running
PowerVP Server Agents – Useful to grab the agents to work with automating installation on LPAR’s
For AIX, VIO Server, and Linux systems the required PowerVP packages are extracted locally.
These packages have to be uploaded to the logical partitions in binary mode and
manual install has to be performed.
Agents are installed to “~/IBM/PowerVP/PowerVP_Installation/PowerVP_Agent_Installation_Instructions/AIX/powervp bff” on Linux
The next window prompts for destination directory for PowerVP install and the following window gives options for the GUI client, make the appropriate choices and continue.

12. Setup & Install: Install Agents
The PowerVP GUI install will extract the installp filesets for the agent locally. The filesets are copied into the install directory on the system where the GUI install was run.
Upload the filesets into a directory on the target AIX or VIO server where installation is intended
The PowerVP Agent installer will drop additional filesets (GSKit8 SSL) in “/tmp/gskit8” for version prior to In newer releases the GSKit filesets were included in the initial GUI install and need to be copied over separate from the agent fileset.
To Uninstall:
From the directory where powervp.x.x.x.x.bff resides (must be root to do this): installp -u powervp.rte
####################################

13. Setup & Install: Install Agents
The installed PowerVP agent can be configured using the iconfig program or by editing the config file manually. The iconfig program automatically updates the PowerVP configuration file. Depending on the type of agent, the iconfig options change.
Edit the Agent config file manually
Edit the Agent config with iconfig
There is only one Agent install package, but two “modes” the agent can run as: System Level and Partition Level.
The Partition Level Agent must be configured to point to a server (IP/DNS) that has a System Level Agent installed.
The configuration file is located at /etc/opt/ibm/powervp/powervp.conf. This file can be edited manually or with the iconfig program. run '/opt/ibm/powervp/iconfig config-keyword=config-value config-keyword=config-value ...' If config-value contains blanks, it must be enclosed in double quotes, e.g. Listen="* ssl" For example, to set the SSL port in the configuration file: run '/opt/ibm/powervp/iconfig Listen="* ssl"‘
The log file is /var/log/powervp.log
No configuration changes are needed if installing as a System Lavel Agent
It’s a good idea to have 2 system level agents per frame for redundancy. VIOS’s work great for hosting the system level agents. Each Partition Level Agent can be configured with 2 System Level agents in case of failure.

14. Setup & Install: Stop & Start Agents
Start the Agent
Run the launcher
$ /etc/rc.d/rc2.d/SPowerVP
Stop the Agent
Locate the PowerVP syslet PID and kill the process
$ ps –eaf | grep syslet

15. Setup & Install: Updates
Minor Updates
So far IBM has released a few minor updates which can be applied “on top of” the existing agent.
Major Updates
There is no upgrade path from PowerVP Version to PowerVP Version To migrate to version 1.1.2, uninstall the previous agent and then install the newer PowerVP agent. You cannot register a PowerVP Version Partition Level Agent to a PowerVP Version System Level Agent.
Upgrade Steps
Stop the PowerVP agent
Un-install PowerVP packages
Install PowerVP agent
Configure the newly installed agent and start
Verify the agent is up and running and the GUI can connect
There is a GA Fix with that is required to be installed, especially for POWER8
systems.

16. USING PowerVP

17. Using PowerVP: GUI Overview
The PowerVP client GUI is installed as part of PowerVP agent install. It can also be installed separate from the agents using the PowerVP installer. You must first connect the PowerVP GUI a System Level Agent before it can be used for monitoring performance on a Power System. To connect, click File -> New Connection. A new connection window appears as shown
In the Hostname field enter the IP address or host name of the partition where the System Level Agent is installed. The Port and SSL Port fields are pre- populated with the default port numbers of and respectively, if different ports are to be used change accordingly. In connection type click Secure or Non-secure based on your System Level Agent configuration. In the login credentials section enter valid User name and Password details and click OK to connect.

18. Using PowerVP: GUI Overview
The main window of PowerVP is displayed when the connection to the System Level Agent is successful. The main window has six different focus areas as shown
System Usage
The system usage section provides overall usage of the system including Cycles Per Instruction (CPI) and a running graph of system overall core utilization. The CPI gives an overall efficiency of the Power System. Normally, a lower CPI is better than a higher CPI.
It also provides measured frequency of the core from the hypervisor.
The system overall core utilization graph shows usage for 10 seconds by default. The 10 second interval can be increased by clicking the graph and zooming out.
Left click the system usage graph to get detailed core usage per LPAR.
Host Information
The host information section displays the partitions that the GUI is connected to and is actively receiving real time data. It does not show all of the partitions with the Partition Level Agent installed, only those partitions that are being drilled down to get the LPAR data.
There are three columns, the first column is hostname or IP address of the partition. The second column is the type of agent running on the partition, system or partition (LPAR) agent. The third column shows if the performance metrics is being recorded.
Playback
The playback area is the interface to record and playback the server performance metrics on your workstation. To view the recordings you will need to start a new instance of the PowerVP GUI (you cannot replay the recording if the PowerVP GUI is already connected to the system).
System Information
The system information area displays the summary of the Power System that is being monitored. It displays information related to POWER7 or POWER8 processor-based systems, total cores on the system, total processor modules, and the platform of the partition.
Partition List
The top panel on the right hand side of the main window contains a list of all the partition that are on the Power System.
There are six columns (LPAR, LPAR ID, State, Cores Entitled, Cores Assigned, Utilization), the table has a sortable header, you can click an individual column heading to sort the table.
System Display
The large section is a graphical representation of the Power System in the system display. Depending on the Power System type and model there will single or multiple Nodes represented by boxes.
Inside these Node boxes are the boxes that account for the processor modules on the Node. The lines between the processor module boxes represent buses. The lines between Nodes are the buses connecting Nodes with each other. The color and the numbers on processor module boxes indicate the usage
Playback
To load a recording, click File -> Load, choose the partition folder and drill down to choose the file. The files with .slt extension are recordings of the System Level Agent and the files with .prt extension are recordings of the Partition Level Agent. When viewing the recording standard playback controls like fast-forward, rewind, pause and stop are available. There is also a slide bar showing the time, it can be used to view performance data at a specific time.
To start recording, under Host Information section click the LPAR that needs recording and click red record button
– The first column is LPAR column, this column can be used to identify what partitions have agents installed. A blank indicates that no agent is installed.
– The second column is LPAR ID, it shows the lpar identification number of the partition and matches HMC profile data.
– The third column is State, it shows if the partition is shared or dedicated.
– The fourth column is Cores Entitled, it displays configured cores of the partition.
– The fifth column is Cores Assigned, it displays cores that are currently used by the partition.
– The last column Utilization, shows a moving bar that indicates the percent of CPU utilization of the partition. This column can be used to easily identify the partitions that are overcommitted.

19. Using PowerVP: Hardware Drill Down
The PowerVP hardware drill down display appears when you click the Node boxes in the
system display panel of the main PowerVP window. The hardware drill down shows the detail
hardware structure of the Node. The processor modules are shown in larger boxes, and
within the boxes are columns representing the cores in each module. The core utilization is
depicted on the cores by bars, and the colors of the columns change based on core utilization
thresholds configured.
Memory Usage
This image depicts the memory footprint and affinity for each LPAR. Each LPAR is depicted by an assigned color. LPAR ID’s and percentages of utilization are available in a context menu when you hover over this area
Memory Controller Utilization
This image depicts the utilization of a memory controller. Memory controllers will not be depicted if they have not been populated with DIMM’s (v )
I/O Controllers & Buses
This image depicts the utilization of an I/O controller. On POWER7 systems this will be a GX controller and on POWER8 it will be a PHB (PCI Host Bridge) controller. This will be applicable for all network adapters, HBA’s, etc.
CPU Core & Bus Utilization
This image depicts the utilization for each core on each chip as well as each bus used for chip to chip communication (intra and inter node)
CPU Affinity
When CPU affinity is enabled the core columns utilization information is replaced with partition affinity. The columns will take the colors of the LPARs that are assigned to the cores. If a mouse is hovered over the columns, a tooltip will show the LPAR ids and the number of virtual CPUs that are assigned to the core.
The lines between processor modules represent the buses between the modules. If the lines extend beyond the display, they are the lines depicting the buses between the Nodes.
Bus lines may appear dotted if they aren’t currently being utilized.
# CPU Core & Bus Utilization
- A breakdown of CPU’s per LPAR can be shown by selecting the partition(s) in the top section. Cores will be highlighted with the color for the given LPAR. Cores within a single shared CPU pool will be grouped together.
# Power 7
3 intra node connections per chip (XYZ)
2 inter node connections per chip (AB)
HighEng (770/780/795): Max 3 hop interconnect
# Power 8
3 inter node connections per chip (ABC)
HighEng (????): Max 2 hop interconnect

20. Using PowerVP: Partition Drill Down
To get details for a partition, a Partition Level Agent has to be installed and running. After the PowerVP GUI connects to the System Level Agent, it builds the list of partitions that are actively running a Partition Level Agent.
The partition drill down view can be opened by right clicking the host name or IP address in partition list and select Open LPAR or by double-clicking on the LPAR name. The partition view opens a new window, there is an icon on the tab to revert to the main window if needed.
LPAR Information
The LPAR Information pane gives static information for the LPAR such as host name, LPAR ID number, platform, type, processor type and mode, number of disk drives and ethernet adapters. Along with static data, it provides real time core, CPI, and the MIPs data.
If the first column in the LPAR list; labeled LPAR, has either the host name or IP address of the partition, then PowerVP GUI can connect and provide a partition view.
# LPAR Stats
CPU - The CPU column shows the CPU utilization of the partition. It is displayed as a percentage of entitled processors.
Disk Transfer Rate - The disk transfer rate shows the total bytes read and written to disk. The column is clickable to get the individual transfer rates of the disks. This is useful when working with VIO and VIO clients using virtual SCSI disk. Currently, there is no support for Fibre Channel or Virtual Fibre Channel adapters and storage area network (SAN) disks.
Ethernet Adapters - The Ethernet adapters column represents that total transfer rate of the all the ethernet adapters on the server that have an IP address configured. The drill down will provide individual adapter transfer rates based on bytes sent and received. The data can be used to see ethernet adapter utilization of LPARs from one central location.
LSU CPI - The LSU CPI column represents the cycles spent on the Load/Store unit resources of a LPAR. When the column is clicked a break down of LSU CPI appears.
The LSU CPI reflects the cycles consumed for accessing data from L1 cache, L2 cache, L3 cache and memory. In the LSU CPI detail view there are columns representing the multiple memory/cache access cycle times. The L2 CPI and L3 CPI columns show the percent of utilization and the cycles took to access L2 and L3 cache access. The Local memory CPI gives the percent of usage and the cycles taken to access local memory controller of the processor module. The L2.1 and L3.1 CPI columns represent cache access across the cores within the same chip. The column Remote L2/L3 CPI displays the usage and cycles spent for accessing L2/L3 cache cross the chip boundaries within the same node.
The Distant L2/L3 CPI column represents the usage and cycles spend accessing the L2 and L3 cache across the node boundaries. The Remote memory CPI column depicts the usage of memory across the chip boundaries within a node. The Distant Memory CPI column illustrates the memory usage across the node boundary and cycles spent accessing it.
The LSU details are useful in debugging performance related issues on OLTP applications.
POWER8 processor has L4 cache. the LSU detail will have an additional column.
The LSU CPI on AIX Power8 partition does not work with base version 1.12, the GA fixpack fixes the issue. We advise you install the GA fixpack.
FXU CPI - The FXU CPI represents the cycles spent on floating point execution in a partition. (Fixed Point Unit??????)
GCT CPI - The GCT CPI represents the cycles spent waiting Global Completion Table (GCT) for the partition.
The Global completion table (GCT) is a table representing a group of instructions currently being processed by the processor. It stores the instruction, the logical program order of instructions, and the completion order of instructions in the group. The empty GCT means there is nothing to do in the pipeline.
LPAR Stats
The LPAR Stats area contains columns labeled CPU, Disk Transfer Rate, Ethernet Adapters, LSU CPI, FXU CPI, and GCT CPI.

21. Using PowerVP: Recording
PowerVP provides a feature that can record and playback the performance data of a Power System and its partitions on the workstation where the GUI is running. The recording can be started and stopped by a simple click of a button using the GUI. Before recording the performance metrics the GUI should be connected to the appropriate PowerVP agent. The recording on the GUI is a client side recording and not server side, the connection to the agent is important. The location of recorded data is configurable using Edit -> Preferences, the log directory is the path where PowerVP will save the recorded performance data.
Recording of performance data is useful in debugging performance related problems. It is useful to compare performance impact of a change when adding processors or memory to the partitions as an example. Performance improvements due to DPO or impact of DLPAR on affinity of the system etc. By recording performance metrics before the change and comparing it with performance metrics after the change it is an easy way to understand the impact of a change.
The PowerVP can record large amounts of data when recording is enabled, it is important to ensure that there is sufficient space to store recorded data. It is advised to increase the sample rate from the default of 1 second. This can be changed by editing the “/etc/opt/ibm/powervp/powervp.conf” file and by changing SampleInterval stanza. The sample interval must be changed for the System Level Agent only, the Partition Level Agents pick up the sample interval from the System Level Agent. To be effective, the PowerVP agent needs to be restarted after the sample rate is changed.
Before recording, first ensure the GUI is connected to the PowerVP agent. The System Level Agent is connected by default, but for the Partition Level Agent you need to drill down on the partition list. Once the PowerVP GUI is connected to the agents, it will be displayed in the “Host Information” section of the main window. To start recording, click the host in the “Host Information” section and press the record button. The recording column in the “Host Information” section will show “Yes”. The same step can be repeated for any partition to be recorded. To stop the recordings, press the square stop button.

22. Using PowerVP: Recording
PowerVP provides a way to record the System Level Agent data from a command line. This is a useful method of recording performance metrics from any workstation without using the GUI. PowerVP installs a powervp.jar file in the install directory that can be used to perform recording of performance metrics.
Currently the powervp.jar file does not have a port option to provide ports from the command line. The workaround is to update the GUI client (the “powervp.properties” file)with appropriate port number and then kick off the command line recording.
To view the recording, a new instance of PowerVP GUI must be started. If you are connected to a live system, the PowerVP GUI will not load the recorded file.
To load a recorded file select File  Load from the menu, and choose the partition that you want to playback and click Open. If multiple partitions are recorded simultaneously, you can load all of them together and PowerVP will synchronize the files and align the time lines with the system level data and partition level data. The playback controls like fast forward, rewind, pause and stop can be used while replaying the metrics.

23. Closing

24. Links PowerVP Site PowerVP Data Sheet PowerVP Demo Video
PowerVP Data Sheet
PowerVP Demo Video
PowerVP Installation and User Guide
PowerVP Introduction and Technical Overview
Installing PowerVP Service Pack 1

25. Q&A