1. © JBoss Inc. 2005 Application and System Monitoring Using JBoss Application Server Nicholas Whitehead Robert Quinn Patrick Urban March 1, 2005

2. 2 Agenda Holistic Application and System Performance Monitoring Challenges In Collecting and Analyzing Performance Data A Proposed Architecture for a Performance Management Data Platform An Implementation of the Platform Questions (and tentative answers)

3. 3 Performance Monitoring & Analysis

4. 4 Heterogeneous Monitoring Targets Network (LAN, WAN, Routers, Switches) Hosts (CPU, Memory, Disk, Threads) Various Operating Systems Application Server (JMX, Instrumentation, Pooling) Middleware (Message Throughput) JVM (Memory, GC) Database (Exec Plans, Memory, Concurrency, Locks) Files (ad nauseum) Applicaton Specific Structures

5. 5 Challenges Phone Call: “System is slow” Why and what do you do about it ? Prevention: Where are your weak spots ? Planning: How many servers do I need to get us through the next n months ? What is the relationship between system load and: Number of users Day of the week Month of the year My astrological sign ?

6. 6 So Many Monitor Points, So Little Time

7. 7 Techniques For Performance Monitoring Commercial Enterprise Frameworks Expensive Not Always Complete Manual Data Collection & Merging No Real Time Vizualization Inflxexible High Maintenance Adhoc Monitoring No Cross Referencing Multiple Monitors Load Monitored System

8. 8 A Cascading Performance Emergency Users Report System is slow. Where’s Waldo ?

9. 9 Anatomy of a Perfomance Hit There He Is !

10. 10 A Consolidated Platform For Peformance Data Acquisition

11. 11 Key Attributes Central, globally available clearing house for performance and system status data. Scale to large numbers of data sources and client dashboards. Supply dashboards, reports and status summaries to dashboards. Provide platform for post processing acquired performance and status data. Monitor Proxy: Multiplex multiple monitor clients to one set of predictable and controllable data acquisition streams. Support Correlation of Heterogeneous Metrics

12. 12 JMX – Perfect For The Job Extensible and Standardized Container for In-Memory State Robust, Flexible API Rich and Dynamic Meta Data Organized Domain Structure Built In Monitors Attribute Change and other Notifications Wide Support for 3 rd Party Tools Rich Lifecycle Control

13. 13 Services Simple State: Holds the last reading for a specific metric. e.g. The last CPU utilization reading on webserver1. Data History Buffer: The last n readings of a specific metric. Shows flat, upward or downward trend. Persistence: Store performance data for reporting, historical analysis and forecasting. Charting: Real time data visualization. Alerts: Alerting services for specific conditions, monitor triggers or failed readings. Boundary Tracking: Track high, low and average values for performance statistics. Data Summarization: Aggregation of large quantities of data into understandble summarized reportlets.

14. 14 Simple State & State History Buffer JMX Attributes Contain the Latest Scanned Value of a Performance Metrics History Buffer Show the Last n Readings.

15. 15 Persistence & Reporting Store performance data for: Reporting Forecasting Capacity Planning Trend Analysis

16. 16 Charting Visualization of real time performance data. Side by side or large and rich displays containing many graphs for visual correlation.

17. 17 Alerts Alerting by email and pager when established threshholds are crossed. Nagios Monitoring System Service Status Screen www.nagios.org

18. 18 Boundary Tracking Track boundaries of selected metrics High Low Average

19. 19 Summarization & Aggregation Large quantities of performance data can be aggregated and summarized to an understandable reportlet. \\IISWEBSERVER01\Thread(inetinfo/0)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/0)\Thread Wait Reason: 0 \\IISWEBSERVER01\Thread(inetinfo/1)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/1)\Thread Wait Reason:6 \\IISWEBSERVER01\Thread(inetinfo/2)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/2)\Thread Wait Reason: 0 \\IISWEBSERVER01\Thread(inetinfo/3)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/3)\Thread Wait Reason:6 \\IISWEBSERVER01\Thread(inetinfo/4)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/4)\Thread Wait Reason: 0 \\IISWEBSERVER01\Thread(inetinfo/5)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/5)\Thread Wait Reason:6 \\IISWEBSERVER01\Thread(inetinfo/6)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/6)\Thread Wait Reason: 0 \\IISWEBSERVER01\Thread(inetinfo/7)\Thread State: 5 \\IISWEBSERVER01\Thread(inetinfo/7)\Thread Wait Reason:6

20. 20 Helios – An Implementation An Open Source Implementation Powered by JBoss Application Server Performance Data Service & Clearing House Heterogeneous Data Sources Extensible Monitors and Post Processing Customizable and Easy to Configure Dashboards

21. 21 Evolution of Helios Original State of Affairs Multi-tool data collection Manual Data Merge No Proxy. All monitor clients put load on servers.

22. 22 Evolution of Helios Simple Automation Automated multi-tool data collection Daily merge Manual Reports and Charts

23. 23 Evolution of Helios Simple Dashboard Numeric and character data only. Web page console. Limited data.

24. 24 Evolution of Helios Evolving Architecture Implementation of MBean Proxies. MBean generated graphics. Single console still requires rebuild and redeploy to change.

25. 25 Helios Version 0.1 New Architecture Dynamic Dashboards that can be saved and shared. Dashboard display widgets can be resized and moved. Configurable, extensible and hot deplyable data source monitors. Extensible post- processing using plugins. External client API for push data. More…..

26. 26 Helios Architecture Extensible and Robust JMX (Remote Notification Really Work) Hibernate Managed Persistence (& HARs !!) Scalable and Clusterable Services Multi-Protocol Remoting Great Platform To Work On Free Platform To Work On

27. 27 Monitors - Overview Implemented as a JMX MBean Acts as a proxy to a performance data source Performance data is available as attributes Configured using XML in jboss-service.xml Can be configured to pass results to a collection of plugins Broadcasts notification of new scans and attribute changes Active Monitors scan a performance data source (Pull) Passive Monitors receive performance data from an External Client Agent (Push)

28. 28 Monitors - Types Currently Available JMXMonitor - Monitors JMX Attributes WindowsMonitor - Monitors Windows Performance Manager Stats SQLMonitor - Monitors JDBC Data Sources YahooFinancialMonitor - Monitors Yahoo Finance Securities PassiveMonitor – Creates attributes for holding results from a remote agent JRat Java Byte Code Instrumentation – Acquires Java Method Elapsed Times In Progress LinuxMonitor: Monitors Linux /proc file performance data JSR77 Autoconfiguring Monitor Text Stream Reader File Scraper (Log Analyzer) Telnet Data Acquisition SSH Data Acquisition Planned OpenArm Helios Mediator: Application Response Measurement Data Acquisition GkRellm Data Acquisition NetPerf Data Acquisition SNMP Data Acquisition PING Monitor

29. 29 Monitors – Sample Configuration

30. 30 Plugins - Overview Configured for each Monitor Receives a copy of each scan result Extends a monitor’s functionality Configured using XML in jboss-service.xml Can create a new result and register it an attribute in the Monitor (back register)

31. 31 Plugins - Types Currently Available PersistScalar – persists scalar results PersistSimpleArray – persists simple arrays RollingHistory – Tracks ‘n’ previous versions of a scan VariaGraphPlugin – Creates line graphs (JPG, PNG or SVG) TrackAverageNumber- Computes averages BasicSQLHTMLFormatter – Formats ResultSets as HTML J2EEStatsHTMLFormatter – Formats JSR77 results as HTML TrackHighNumber – Maintains a maximum value TrackLowNumber – Maintains a minimum value TrackAverageNumber – Maintains the average value In Progress Graphing Library Nagios External Event Updates File Logging Event Planned Customizable Event Broadcaster For Integrating to 3 rd Parties Customizable HTML Renderer

32. 32 Plugins – Sample Configuration

33. 33 Subscription Service - Overview Allows a client to subscribe to and receive updates for Monitor attributes Implemented as an MBean HTTP Façaded provide for web clients Clients establish a “session” Server Side Proxy manages client’s subscriptions Subscription listen on the notification bus and filter attributes notifications based on client’s subscription Client can subscribe/unsubscribe to attributes Client “polls” and receives all available updates

34. 34 Subscription Service - Components

35. 35 Subscription Service – Basic Use Cases

36. 36 Visualizing The Data Dashboard Manager A WYSIWYG tool for managing and sharing views of your information real-time via “Dashboards”.

37. 37 The Goals Of Helios Dashboard Manager Easy Creation Easy Viewing Easy Sharing

38. 38 Easy Creation All available “Monitors” displayed for selection in a tree style view called the Dictionary

39. 39 Easy Creation Cont. Drag and Drop style dashboard creator Supports resizing and overlapping

40. 40 Easy Viewing View dashboards using a browser Currently supports IE 5.5+ and Firefox 1.0 Supports full-screen viewing and slide show style for multiple dashboards. Live Data Updates. Supported output type include… Single Value Array Rich HTML Image.gif,.jpg,.png SVG View on any platform via Firefox No additional client installs or plugins required Clean presentation ready look

41. 41 Easy Sharing Dashboards support custom HTML content via “Labels” that allow the creator to annotate a dashboard as necessary. Underlying XML descriptor allows for dashboard definitions to be imported and exported between Helios server instances.

42. 42 Future Plans Library of Client-Side Rendered Graphs via SVG More Dashboard Customization Options Look and Feel Behavior Security/Dashboard Ownership Integration

43. 43 Client API Remote API support for: Creating New Passive Monitors. Creating New Monitor Attributes Updating Monitor Attributes Protocols: RMI JMS Simple XML over HTTP XML over HTTP allows non-Java scripting languages such as Perl and TCL to perform remote data acquisition for Helios.

44. 44 Thanks ! Helios: Greek God Of The Sun. He sees everything that happens and is often called upon to shed light on events. Helios. See All. Hear All. Understand All. https://helios.dev.java.net/index.html