1. The Pros and Cons of Collecting Performance Data using Agentless Technology Dima Seliverstov John Tavares Tianxiang Zhang BMC Software, Inc.

2. © Copyright BMC Software, Inc. Why Agentless › Installing and administrating agents is a pain –Agents have to be distributed across the enterprise –When agents break they have to be patched –Each agent you install has its own file system and network port requirements –Agents are not always built to network or security standards

3. © Copyright BMC Software, Inc. Blue Sky Model › Wouldn’t it be great if operating systems came with a standards based agent installed and configured for monitoring and capacity planning?

4. © Copyright BMC Software, Inc. Weather Report › Identify which operating systems come with performance collection installed › Identify the metrics available and how often to collect them › Network and security issue › Identify an alternate way of getting the system performance data without having to install an agent

5. © Copyright BMC Software, Inc. Overview of the Presentation › Agentless data collection is a powerful technology which has its advantages and disadvantages. –System performance metrics availability –Agentless monitoring components installed as part of the operating system –Security and network issue › Present our experience with Windows agentless technology

6. © Copyright BMC Software, Inc. Agentless Implementations Considered › SNMP (Simple Network Management Protocol) › WMI (Windows Management Interface) › Windows Remote Registry › WBEM (Web Based Enterprise Management) › Agentless monitoring by sending system commands over the network

7. © Copyright BMC Software, Inc. WBEM (Web Based Enterprise Management) Introduction › Developed by DMTF (Distributed Management Task Force) › Built on top of the Common Information Model (CIM) –Hierarchical object oriented representation of management information such as computer system, network devices and applications –Compiled textual representation known as MOF (Managed Object Format)

8. © Copyright BMC Software, Inc. WBEM Communication Mechanism › WBEM Client –Issues CIM Operation requests and receives and processes CIM Operation responses › WBEM Server –Uses CIMOM (CIM Object Manager) to communicate with clients –Receives and processes CIM Operation requests and issues CIM Operation responses –Uses CIM Repository as persistent store containing CIM data in a namespace –The root of the namespace is root\cimv2 › WBEM Provider –Process CIM Operations on one or more managed resources and maps the resource specific interface to a CIM interface

9. © Copyright BMC Software, Inc. WBEM Security › Windows implementation of WBEM –WMI (Windows Management Instrumentation) – Encryption and authentication on per call or a per packet basis › UNIX WBEM –Encryption and authentication support via public key certificates via https

10. © Copyright BMC Software, Inc. Windows Remote Registry › In addition to WMI, Microsoft provides a Remote Registry interface to getting system information from the performance registry remotely › Available since Windows NT › Unfortunately, there have been several security warning regarding the use of remote registry service › The protocol for the remote registry is RPC (Remote Procedures Call) and introduces challenges in terms of network connectivity and platform independent client tools

11. © Copyright BMC Software, Inc. WBEM Availability › Windows –WMI installed as part of the core operating system since Windows 2000 › Solaris –Solaris WBEM Services has been installed since Solaris 2.9 › HP-UX –HP WBEM Services has been installed since HPUX 11.0 › AIX –WBEM is supported for via the AIX 5L Expansion Pack and Web Download Pack for AIX 5.2 and higher › Linux –RedHat and SUSE support Open Pegasus package Linux install packages

12. © Copyright BMC Software, Inc. SNMP Overview › Standard primarily used for network management › SNMP stores elements in a Management Information Base (MIB ) –Extensible –MIB stores information in a tree format –Extended for system performance data RFC 2790 Host Resource MIB

13. © Copyright BMC Software, Inc. SNMP Communication Mechanism › Management information is maintained by the SNMP agent and queried by a SNMP manager › SNMP agent and the SNMP manager must have access to the Management Information Base › The SNMP agent and SNMP manager communicate via UDP (User Datagram Protocol) port 161 › The SNMP manager gets the data from the agent via the get-request and get-next-request

14. © Copyright BMC Software, Inc. SNMP Security › There are 3 major versions of SNMP –SNMP v1 Unencrypted traffic IP address-based access lists and community strings –SNMP v2 Encryption –SNMP v3 Encryption Authentication

15. © Copyright BMC Software, Inc. SNMP Agent Availability › Usually not installed as part of the operating system › Open source SNMP agent called net-snmp is available to gather the data for the Host Resource MIB › A Host Resource MIBs provides a limited set of data –The data availability will be shown in the slides to follow

16. © Copyright BMC Software, Inc. Metric Availability › Windows provides the best data set › UNIX WBEM implementations provide a good data set with a limited number of exceptions › SNMP provides a limited amount of performance configuration data and almost no statistical data

17. © Copyright BMC Software, Inc. CPU

18. © Copyright BMC Software, Inc. Memory

19. © Copyright BMC Software, Inc. Disk and File System

20. © Copyright BMC Software, Inc. Network

21. © Copyright BMC Software, Inc. Process

22. © Copyright BMC Software, Inc. UNIX Agentless Monitoring by Sending System Commands over the Network › Log in remotely to multiple agentless computers and collect the necessary performance metrics › Use Secure Shell –Supports certificate based encryption and authentication Metric TypeCommand CPUsar,uptime Memorysar, vmstat Processps I/Onetstat,iostat,vmstat,netstat,nfsstat Diskdf,iostat

23. © Copyright BMC Software, Inc. Downside the Approach › The commands vary from operating system to operating system › The output of the command must be parsed specifically for each operating system › Secure shell may not be installed on all computers of interest › Metrics are usually instantaneous and not cumulative and must be sampled frequently for a reliable system estimates

24. © Copyright BMC Software, Inc. Issues with Agentless Data Collection › Network Ports › Network Bandwidth › Data Sampling › Patching

25. © Copyright BMC Software, Inc. Network Ports › Windows WMI and Remote Registry use dynamic ports –This wide range of ports used, can play havoc with firewall rules InfrastructurePorts Used SNMP161 WMI Remote Registry 135,137,139,445 1024-65536 WBEM5988,5989

26. © Copyright BMC Software, Inc. Network Bandwidth › SNMP agents use considerably/exponentially less bandwidth then WBEM –WBEM uses an XML based protocol › If you lose network connectivity from the client to the agent-less server, you will lose data

27. © Copyright BMC Software, Inc. Network Bandwidth continued › No batch transfer –Data is collected through out the day –It’s not possible to schedule batch data transfer at a particular time when the network utilization is low › No summarization –Data is not summarized at the agentless computer –Every data point requested is transmitted over the network

28. © Copyright BMC Software, Inc. Data Sampling › Cumulative Counters and Instantaneous Gauges –Cumulative Counters need to be sampled less frequently because they represent the difference in a system between two points in time › WMI and WBEM provide many metrics that are cumulative › SNMP and sending system commands over the network approach will provide mostly instantaneous metrics › SNMP and some UNIX WBEM implementations CPU and Process Metrics are provided as an average over a 1 minute time interval › In general, to achieve good representation of system activity for capacity planning a sample rate should be 1 – 2 minutes for instantaneous values

29. © Copyright BMC Software, Inc. Patching agentless components › Isn’t the whole point of going to agentless data collection to avoid patching? › Agentless really means that someone already installed an agent for you › Agentless components do break and need to be patched › It can be challenging to identify and address a problem due to a variety of tools and debugging techniques for each platform

30. © Copyright BMC Software, Inc. Experience with Windows Agentless Technology › WMI requires 'Enable Account' and 'Remote Enable' permissions for the Root/CIMV2 for the WMI namespace › Configuring user accounts on individual computers can be challenging –Use domain administrator account –Use regular user domain account to configure agent-less computers with a regular user with 'Enable Account' and 'Remote Enable' permissions for the Root/CIMV2 for the WMI namespace You will need to create the permission on each individual computer › Windows WMI encountered frequent security events when running using a non-administrator user account on Windows 2003 machines › Network administrators were concerned about the constant network traffic during the day

31. © Copyright BMC Software, Inc. Final Thoughts › WBEM is extremely promising › Software vendors including Hewlett-Packard, IBM, Microsoft, RedHat and SUSE are beginning to roll out their products with WBEM support enabled › WBEM provides a common and secure framework for collecting system performance data › A strong development, open source community and a standards organization are supporting a variety of implementations for clients, servers and providers