IBM PowerVP - Overview and Demo Joe Searcy joseph.searcy@theice.com Sr. Systems Engineer, Trading Systems 2014

Agenda Overview Using PowerVP What is PowerVP GUI Overview How do I obtain PowerVP What are the requirements PowerVP Architecture PowerVP Security Setup & Install Install GUI Install Agents Start & Stop Agents Updates Using PowerVP GUI Overview Hardware Drill Down Partition Drill Down Recording Closing Links Q&A

OVERVIEW

Overview: What is PowerVP? SERVER AGENT GUI PowerVP is a tool consisting of software based agents that are installed within LPAR’s to feed data to a Java based utility that graphically shows various performance related details of a single PowerSystems server ecosystem. Highlights Optimize your virtualized IBM® Power Systems™ Understand current workloads performance from virtual machine to physical hardware Simple real-time graphical performance information highlighting overcommitted resources Ability to replay saved performance data to analyze historical performance data Accelerate the resolution of performance issues Real-time continuous monitoring user interface Proactively address performance for virtualized workloads

Overview: How do I obtain PowerVP Steps for downloading updates: 1. Navigate to http://www.ibm.com/support/fixcentral 2. Click Find product. 3. Begin typing PowerVP in the Product selector field. 4. Select PowerVP from the list. For Installed version, select All. 5. Identify and select the fix that is required. 6. Download the fix. 7. Open the download document and follow the link in the "Download Package" section. Latest version to date: 1.1.2 Service Pack 1

Overview: What does PowerVP require PowerVP is included in the PowerVM Enterprise Edition, current and new PowerVM Enterprise Edition customers are entitled for PowerVP. PowerVM Standard Edition clients can purchase PowerVP Standard Edition. Prequisites Hardware Power 7 Power 7+ Power 8 Firmware On Power7 systems the firmware level of 770 and 780 or later is required Operating System Supported Release Levels AIX 6.1 & 7.1 VIOS – All versions IBMi 7.1 TR6 and 7.2 (both with PTF S152700, MF58478, MF58480) RHEL 6.4 and later SLES 11 SP3 and later *Supported models as of 06-10-2014 Java The GUI requires Java JRE 1.6.0 version HMC The “Allow Performance Information Collection” must be enabled for a logical partition that is running System Level Agent Network port 13000 for non-SSL and 13001 for SSL traffic by default. The port numbers can be customized to fit the security needs of the customer. The requirement is for the System Level Agent and the Partition Level Agent to use the same port numbers. PowerVP Version 1.12 is the minimum version support The firmware levels below 770 lack the new instrumentation required for Power File system Requirements PowerVP has a small footprint and is installed in /opt/ibm/powervpon UNIX systems, on IBM it is installed in QLSE library. Configuration files are installed in /etc/opt/ibm/powervp on UNIX and on IBM i the configuration files are installed in /QIBM/UserData/PowerVP. It is necessary to ensure that the file systems are not full.The disk space required for GUI client is dependent on the usage of background performance data collection feature of PowerVP.VP on POWER8 systems

Overview: PowerVP Architecture System Level Agent PowerVP requires a designated partition on a Power System to be utilized for the collection of system wide resource utilization data across a frame or central electronics complex. For the designated partition set the LPAR properties to allow performance information collection Partition Level Agent In order to view the partition specific information, PowerVP requires an agent to be installed on the partitions, this agent is called a Partition Level Agent. The Partition Level Agent registers with System Level Agent, hence the System Level Agent must be up and running before installing the Partition Level Agent. Graphical User Interface The GUI client is a Java based application using Java Swing. The client is available for multiple platforms including, Windows, AIX, Linux, MacOSX and Solaris or any platforms that can run Java applications. The GUI client connects to the System Level Agent, the client must be able to connect to the port on which the System Level Agent is listening (default is 13001 for SSL and 13000 for non-SSL). Background Recorder For recording system level data without using GUI a Java jar file named powervp.jar is available. The jar file can be launched from any server that can connect to the System Level Agent. Currently only system level data can be recorded using the command line option. Partition Level Agent recording can only be done using the client GUI. The System Level Agent uses kernel extension module on UNIX systems to make privileged hcalls (hypervisor call) to the hypervisor. On IBM i, the API by the way of XPF and SLIC (System Licensed Internal Code) interfaces with the hypervisor. After the System Level Agent makes this connection with the hypervisor, the agent requests performance counters from the hypervisor as needed. There can be multiple System Level Agents installed on a Frame, it is advisable to use two System Level Agents installed for redundancy. The Partition Level Agents use partition APIs to retrieve Ethernet and Disk I/O utilization data and also access Performance Monitoring Unit (PMU) to get hardware thread information. On AIX and VIOS the PMU is accessed using Performance Monitor API (pmapi). On IBM i PEX is used to get PMU data. On Linux systems Partition Level Agent uses perf stat commands to obtain performance information. The System Level Agent will also act as partition agent for the partition it is running on. There is no need for additional partition level agent. The GUI can be installed on multiple clients and multiple clients can connect to view the data simultaneously; however, multiple client connections will add load to the system as they all will be requesting performance data every second (default refresh time).

Overview: PowerVP Security The PowerVP GUI will connect to the System Level Agent using a valid user name and password on the partition. When accessing partition information using drill down, or when using the Monitor LPAR option, the client GUI is connecting to the Partition Level Agent to access the data. There are two options to authenticate to the Partition Level Agent: 1. Using the System Level Agent to authenticate the connection to the partition. This is the default option, after the client GUI is authenticated by the System Level Agent, the user will be able to drill down into the partition. In this method there is no need to provide authentication for signing on to the Partition Level Agent 2. The second method is to provide valid user and password details for the GUI client to connect to the Partition Level Agent PowerVP provides both secure and non-secure options for connection between the agents and the client GUI. Even though user and password information is not transmitted in plain text in non-secure mode, the rest of the data is transmitted in plain text. The secure option uses SSL protocol for communications between the PowerVP components. To use this option, each PowerVP agent will need a SSL certificate.

SETUP & INSTALL

Setup & Install: Install GUI The PowerVP installer is a GUI with an easy to follow menu-driven dialog. The PowerVP GUI client is installed by default on the system where the installer is run. The PowerVP installer and the client can be on Windows, Linux or AIX systems To install PowerVP, mount the install media and change the appropriate directory for your operating system. The two executable files in the directory are PowerVP and PoweVP_IBMi_Agent. The PowerVP executable must be used to perform the install. It is advised to run the install from the media itself as there are other directories on the install media that are required by the installer (example: license directory).

Setup & Install: Install GUI When the PowerVP Installer is executed, after language selection and acceptance of license agreement, you are prompted to choose the type of install to perform. There are three install options: Typical This option will install PowerVP client and the agents. PowerVP Client GUI This option installs only the client GUI, useful for users who have PowerVP agents already installed. PowerVP Server Agents This option installs the PowerVP agents only, useful for users who have the PowerVP client GUI installed and are adding new LPARs to the PowerVP monitoring. Typical – Where you want to start. Install GUI and agents PowerVP Client GUI – Secondary workstations after you have the agents installed and running PowerVP Server Agents – Useful to grab the agents to work with automating installation on LPAR’s For AIX, VIO Server, and Linux systems the required PowerVP packages are extracted locally. These packages have to be uploaded to the logical partitions in binary mode and manual install has to be performed. Agents are installed to “~/IBM/PowerVP/PowerVP_Installation/PowerVP_Agent_Installation_Instructions/AIX/powervp.1.1.0.3.bff” on Linux The next window prompts for destination directory for PowerVP install and the following window gives options for the GUI client, make the appropriate choices and continue.

Setup & Install: Install Agents The PowerVP GUI install will extract the installp filesets for the agent locally. The filesets are copied into the install directory on the system where the GUI install was run. Upload the filesets into a directory on the target AIX or VIO server where installation is intended The PowerVP Agent installer will drop additional filesets (GSKit8 SSL) in “/tmp/gskit8” for version prior to 1.1.2. In newer releases the GSKit filesets were included in the initial GUI install and need to be copied over separate from the agent fileset. To Uninstall: From the directory where powervp.x.x.x.x.bff resides (must be root to do this): installp -u powervp.rte ####################################

Setup & Install: Install Agents The installed PowerVP agent can be configured using the iconfig program or by editing the config file manually. The iconfig program automatically updates the PowerVP configuration file. Depending on the type of agent, the iconfig options change. Edit the Agent config file manually Edit the Agent config with iconfig There is only one Agent install package, but two “modes” the agent can run as: System Level and Partition Level. The Partition Level Agent must be configured to point to a server (IP/DNS) that has a System Level Agent installed. The configuration file is located at /etc/opt/ibm/powervp/powervp.conf. This file can be edited manually or with the iconfig program. run '/opt/ibm/powervp/iconfig config-keyword=config-value config-keyword=config-value ...' If config-value contains blanks, it must be enclosed in double quotes, e.g. Listen="* 13001 ssl" For example, to set the SSL port in the configuration file: run '/opt/ibm/powervp/iconfig Listen="* 13001 ssl"‘ The log file is /var/log/powervp.log No configuration changes are needed if installing as a System Lavel Agent It’s a good idea to have 2 system level agents per frame for redundancy. VIOS’s work great for hosting the system level agents. Each Partition Level Agent can be configured with 2 System Level agents in case of failure.

Setup & Install: Stop & Start Agents Start the Agent Run the launcher $ /etc/rc.d/rc2.d/SPowerVP Stop the Agent Locate the PowerVP syslet PID and kill the process $ ps –eaf | grep syslet

Setup & Install: Updates Minor Updates So far IBM has released a few minor updates which can be applied “on top of” the existing agent. Major Updates There is no upgrade path from PowerVP Version 1.1.0 to PowerVP Version 1.1.2. To migrate to version 1.1.2, uninstall the previous agent and then install the newer PowerVP agent. You cannot register a PowerVP Version 1.1.0 Partition Level Agent to a PowerVP Version 1.1.2 System Level Agent. Upgrade Steps Stop the PowerVP agent Un-install PowerVP 1.1.0 packages Install PowerVP 1.1.2 agent Configure the newly installed agent and start Verify the agent is up and running and the GUI can connect There is a GA Fix with 1.1.2 that is required to be installed, especially for POWER8 systems.

USING PowerVP

Using PowerVP: GUI Overview The PowerVP client GUI is installed as part of PowerVP agent install. It can also be installed separate from the agents using the PowerVP installer. You must first connect the PowerVP GUI a System Level Agent before it can be used for monitoring performance on a Power System. To connect, click File -> New Connection. A new connection window appears as shown In the Hostname field enter the IP address or host name of the partition where the System Level Agent is installed. The Port and SSL Port fields are pre- populated with the default port numbers of 13000 and 13001 respectively, if different ports are to be used change accordingly. In connection type click Secure or Non-secure based on your System Level Agent configuration. In the login credentials section enter valid User name and Password details and click OK to connect.

Using PowerVP: GUI Overview The main window of PowerVP is displayed when the connection to the System Level Agent is successful. The main window has six different focus areas as shown System Usage The system usage section provides overall usage of the system including Cycles Per Instruction (CPI) and a running graph of system overall core utilization. The CPI gives an overall efficiency of the Power System. Normally, a lower CPI is better than a higher CPI. It also provides measured frequency of the core from the hypervisor. The system overall core utilization graph shows usage for 10 seconds by default. The 10 second interval can be increased by clicking the graph and zooming out. Left click the system usage graph to get detailed core usage per LPAR. Host Information The host information section displays the partitions that the GUI is connected to and is actively receiving real time data. It does not show all of the partitions with the Partition Level Agent installed, only those partitions that are being drilled down to get the LPAR data. There are three columns, the first column is hostname or IP address of the partition. The second column is the type of agent running on the partition, system or partition (LPAR) agent. The third column shows if the performance metrics is being recorded. Playback The playback area is the interface to record and playback the server performance metrics on your workstation. To view the recordings you will need to start a new instance of the PowerVP GUI (you cannot replay the recording if the PowerVP GUI is already connected to the system). System Information The system information area displays the summary of the Power System that is being monitored. It displays information related to POWER7 or POWER8 processor-based systems, total cores on the system, total processor modules, and the platform of the partition. Partition List The top panel on the right hand side of the main window contains a list of all the partition that are on the Power System. There are six columns (LPAR, LPAR ID, State, Cores Entitled, Cores Assigned, Utilization), the table has a sortable header, you can click an individual column heading to sort the table. System Display The large section is a graphical representation of the Power System in the system display. Depending on the Power System type and model there will single or multiple Nodes represented by boxes. Inside these Node boxes are the boxes that account for the processor modules on the Node. The lines between the processor module boxes represent buses. The lines between Nodes are the buses connecting Nodes with each other. The color and the numbers on processor module boxes indicate the usage Playback To load a recording, click File -> Load, choose the partition folder and drill down to choose the file. The files with .slt extension are recordings of the System Level Agent and the files with .prt extension are recordings of the Partition Level Agent. When viewing the recording standard playback controls like fast-forward, rewind, pause and stop are available. There is also a slide bar showing the time, it can be used to view performance data at a specific time. To start recording, under Host Information section click the LPAR that needs recording and click red record button – The first column is LPAR column, this column can be used to identify what partitions have agents installed. A blank indicates that no agent is installed. – The second column is LPAR ID, it shows the lpar identification number of the partition and matches HMC profile data. – The third column is State, it shows if the partition is shared or dedicated. – The fourth column is Cores Entitled, it displays configured cores of the partition. – The fifth column is Cores Assigned, it displays cores that are currently used by the partition. – The last column Utilization, shows a moving bar that indicates the percent of CPU utilization of the partition. This column can be used to easily identify the partitions that are overcommitted.

Using PowerVP: Hardware Drill Down The PowerVP hardware drill down display appears when you click the Node boxes in the system display panel of the main PowerVP window. The hardware drill down shows the detail hardware structure of the Node. The processor modules are shown in larger boxes, and within the boxes are columns representing the cores in each module. The core utilization is depicted on the cores by bars, and the colors of the columns change based on core utilization thresholds configured. Memory Usage This image depicts the memory footprint and affinity for each LPAR. Each LPAR is depicted by an assigned color. LPAR ID’s and percentages of utilization are available in a context menu when you hover over this area Memory Controller Utilization This image depicts the utilization of a memory controller. Memory controllers will not be depicted if they have not been populated with DIMM’s (v1.1.2.0+) I/O Controllers & Buses This image depicts the utilization of an I/O controller. On POWER7 systems this will be a GX controller and on POWER8 it will be a PHB (PCI Host Bridge) controller. This will be applicable for all network adapters, HBA’s, etc. CPU Core & Bus Utilization This image depicts the utilization for each core on each chip as well as each bus used for chip to chip communication (intra and inter node) CPU Affinity When CPU affinity is enabled the core columns utilization information is replaced with partition affinity. The columns will take the colors of the LPARs that are assigned to the cores. If a mouse is hovered over the columns, a tooltip will show the LPAR ids and the number of virtual CPUs that are assigned to the core. The lines between processor modules represent the buses between the modules. If the lines extend beyond the display, they are the lines depicting the buses between the Nodes. Bus lines may appear dotted if they aren’t currently being utilized. # CPU Core & Bus Utilization - A breakdown of CPU’s per LPAR can be shown by selecting the partition(s) in the top section. Cores will be highlighted with the color for the given LPAR. Cores within a single shared CPU pool will be grouped together. # Power 7 3 intra node connections per chip (XYZ) 2 inter node connections per chip (AB) HighEng (770/780/795): Max 3 hop interconnect # Power 8 3 inter node connections per chip (ABC) HighEng (????): Max 2 hop interconnect

Using PowerVP: Partition Drill Down To get details for a partition, a Partition Level Agent has to be installed and running. After the PowerVP GUI connects to the System Level Agent, it builds the list of partitions that are actively running a Partition Level Agent. The partition drill down view can be opened by right clicking the host name or IP address in partition list and select Open LPAR or by double-clicking on the LPAR name. The partition view opens a new window, there is an icon on the tab to revert to the main window if needed. LPAR Information The LPAR Information pane gives static information for the LPAR such as host name, LPAR ID number, platform, type, processor type and mode, number of disk drives and ethernet adapters. Along with static data, it provides real time core, CPI, and the MIPs data. If the first column in the LPAR list; labeled LPAR, has either the host name or IP address of the partition, then PowerVP GUI can connect and provide a partition view. # LPAR Stats CPU - The CPU column shows the CPU utilization of the partition. It is displayed as a percentage of entitled processors. Disk Transfer Rate - The disk transfer rate shows the total bytes read and written to disk. The column is clickable to get the individual transfer rates of the disks. This is useful when working with VIO and VIO clients using virtual SCSI disk. Currently, there is no support for Fibre Channel or Virtual Fibre Channel adapters and storage area network (SAN) disks. Ethernet Adapters - The Ethernet adapters column represents that total transfer rate of the all the ethernet adapters on the server that have an IP address configured. The drill down will provide individual adapter transfer rates based on bytes sent and received. The data can be used to see ethernet adapter utilization of LPARs from one central location. LSU CPI - The LSU CPI column represents the cycles spent on the Load/Store unit resources of a LPAR. When the column is clicked a break down of LSU CPI appears. The LSU CPI reflects the cycles consumed for accessing data from L1 cache, L2 cache, L3 cache and memory. In the LSU CPI detail view there are columns representing the multiple memory/cache access cycle times. The L2 CPI and L3 CPI columns show the percent of utilization and the cycles took to access L2 and L3 cache access. The Local memory CPI gives the percent of usage and the cycles taken to access local memory controller of the processor module. The L2.1 and L3.1 CPI columns represent cache access across the cores within the same chip. The column Remote L2/L3 CPI displays the usage and cycles spent for accessing L2/L3 cache cross the chip boundaries within the same node. The Distant L2/L3 CPI column represents the usage and cycles spend accessing the L2 and L3 cache across the node boundaries. The Remote memory CPI column depicts the usage of memory across the chip boundaries within a node. The Distant Memory CPI column illustrates the memory usage across the node boundary and cycles spent accessing it. The LSU details are useful in debugging performance related issues on OLTP applications. POWER8 processor has L4 cache. the LSU detail will have an additional column. The LSU CPI on AIX Power8 partition does not work with base version 1.12, the GA fixpack fixes the issue. We advise you install the GA fixpack. FXU CPI - The FXU CPI represents the cycles spent on floating point execution in a partition. (Fixed Point Unit??????) GCT CPI - The GCT CPI represents the cycles spent waiting Global Completion Table (GCT) for the partition. The Global completion table (GCT) is a table representing a group of instructions currently being processed by the processor. It stores the instruction, the logical program order of instructions, and the completion order of instructions in the group. The empty GCT means there is nothing to do in the pipeline. LPAR Stats The LPAR Stats area contains columns labeled CPU, Disk Transfer Rate, Ethernet Adapters, LSU CPI, FXU CPI, and GCT CPI.

Using PowerVP: Recording PowerVP provides a feature that can record and playback the performance data of a Power System and its partitions on the workstation where the GUI is running. The recording can be started and stopped by a simple click of a button using the GUI. Before recording the performance metrics the GUI should be connected to the appropriate PowerVP agent. The recording on the GUI is a client side recording and not server side, the connection to the agent is important. The location of recorded data is configurable using Edit -> Preferences, the log directory is the path where PowerVP will save the recorded performance data. Recording of performance data is useful in debugging performance related problems. It is useful to compare performance impact of a change when adding processors or memory to the partitions as an example. Performance improvements due to DPO or impact of DLPAR on affinity of the system etc. By recording performance metrics before the change and comparing it with performance metrics after the change it is an easy way to understand the impact of a change. The PowerVP can record large amounts of data when recording is enabled, it is important to ensure that there is sufficient space to store recorded data. It is advised to increase the sample rate from the default of 1 second. This can be changed by editing the “/etc/opt/ibm/powervp/powervp.conf” file and by changing SampleInterval stanza. The sample interval must be changed for the System Level Agent only, the Partition Level Agents pick up the sample interval from the System Level Agent. To be effective, the PowerVP agent needs to be restarted after the sample rate is changed. Before recording, first ensure the GUI is connected to the PowerVP agent. The System Level Agent is connected by default, but for the Partition Level Agent you need to drill down on the partition list. Once the PowerVP GUI is connected to the agents, it will be displayed in the “Host Information” section of the main window. To start recording, click the host in the “Host Information” section and press the record button. The recording column in the “Host Information” section will show “Yes”. The same step can be repeated for any partition to be recorded. To stop the recordings, press the square stop button.

Using PowerVP: Recording PowerVP provides a way to record the System Level Agent data from a command line. This is a useful method of recording performance metrics from any workstation without using the GUI. PowerVP installs a powervp.jar file in the install directory that can be used to perform recording of performance metrics. Currently the powervp.jar file does not have a port option to provide ports from the command line. The workaround is to update the GUI client (the “powervp.properties” file)with appropriate port number and then kick off the command line recording. To view the recording, a new instance of PowerVP GUI must be started. If you are connected to a live system, the PowerVP GUI will not load the recorded file. To load a recorded file select File  Load from the menu, and choose the partition that you want to playback and click Open. If multiple partitions are recorded simultaneously, you can load all of them together and PowerVP will synchronize the files and align the time lines with the system level data and partition level data. The playback controls like fast forward, rewind, pause and stop can be used while replaying the metrics.

Closing

Links PowerVP Site PowerVP Data Sheet PowerVP Demo Video http://www-03.ibm.com/systems/power/software/performance/ PowerVP Data Sheet http://public.dhe.ibm.com/common/ssi/ecm/en/pod03082usen/POD03082USEN.PDF PowerVP Demo Video http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ST&infotype=SA&appname=STGE_PO_PO_USEN&htmlfid=POJ03245USEN PowerVP Installation and User Guide http://public.dhe.ibm.com/partnerworld/pub/misc/power_vp_installation_and_user_guide.pdf PowerVP Introduction and Technical Overview http://www.redbooks.ibm.com/redbooks.nsf/RedpieceAbstracts/redp5112.html?Open Installing PowerVP Service Pack 1 https://www.ibm.com/developerworks/community/blogs/cgaix/entry/installing_powervp_service_pack_1?lang=en

Q&A