SCMG’12 Igor Trubin, PhD IBM, Sr. IT Architect SEDS-Lite: Using Open Source Tools (R, BIRT and MySQL) to Report and Analyze Performance Data

SCMG’12 2 A genda oSEDS-Lite project: Open source based implementation of Statistical Exception Detection System - SEDS-Lite Introduction SEDS-Lite Introduction oOpen Source and “free” tools for Capacity Management – review oControl Charts as a good test of using different tools for Capacity management - Power of Control Charts and IT-Chart Concept (Part 1) Power of Control Charts and IT-Chart Concept (Part 1) oR-system scriping to built Control Charts ( System Management by Exception: Power of Control Charts Mar 25, 2009) System Management by Exception: Power of Control Charts oHow To Build IT-Control Chart - Use the Excel Pivot Table!How To Build IT-Control Chart - Use the Excel Pivot Table! oBIRT based Control Chart BIRT based Control Chart oBIRT Data Cubes Usage for Performance Data Analysis BIRT Data Cubes Usage for Performance Data Analysis oBuilding IT-Control Chart by BIRT against Data from the MySQL Database Building IT-Control Chart by BIRT against Data from the MySQL Database oR-Script to Aggregate (ETL to MySQL) Actual data with Base-line data for IT-Control ChartsR-Script to Aggregate (ETL to MySQL) Actual data with Base-line data for IT-Control Charts oEV-Control Chart (to build using BIRT)EV-Control Chart oSimplified Exception Detection using BIRT: UCL=LCL : How many standard deviations do we use for Control Charting? Use ZERO! UCL=LCL : How many standard deviations do we use for Control Charting? Use ZERO! oNon-open source tool usage example - COGNOSNon-open source tool usage example - COGNOS oBenefits and Values of implementing SEDS

SCMG’12 SEDS-Lite Open source based implementation of Statistical Exception Detection System - SEDS-Lite IntroductionSEDS-Lite Introduction

SCMG’12 4 Open Source and “free” tools for Capacity Management Previous reviews: oMeasureIT - Issue Open Source Capacity & Performance Management Tools for Windows & Unix Systems ( Brian Johnson )MeasureIT - Issue Open Source Capacity & Performance oCapacity Planning and Performance Monitoring with Free Tools (for mspcmg 2011! - Adrian Cockcroft)Capacity Planning and Performance Monitoring with Free Tools Adrian Cockcroft The missed point is BI and Generic DB open source tools usage for Performance Analysis and Reporting oBIRT (Business Intelligence and Reporting Tools)BIRT oMySQLMySQL oNoSQLNoSQL oPostgreSQLPostgreSQL

SCMG’12 5 Control Charts as a good test of using different tools for Capacity management What is the Control Chart? o Power of Control Charts and IT-Chart Concept (CMG’09 -’10) Power of Control Charts and IT-Chart Concept

SCMG’12 6 R-system to built Control Charts The input data is Unix File Space Utilization: R-script R-script (published on my blog):published on my blog (FYI: qcc: An R package for quality control charting :qcc: An R package for quality control charting project.org/web/packages/qcc/index.html project.org/web/packages/qcc/index.html ) oEXAMPLE3: Monthly Profile R download: CMG’09 workshop “ Power of Control Charts “, SCMG - Mar 25, 2009 Power of Control Charts

SCMG’12 7 R-system to built Control Charts CMG’09 workshop “ Power of Control Charts “, SCMG - Mar 25, 2009 Power of Control Charts

SCMG’12 BIRT based Control Chart What is BIRT – see BIRT Report Designer Tutorial DemoBIRT Report Designer Tutorial Demo There is a tool that generates control charts against performance data using BIRT:a tool The IT-Control Chart would give much more value for analyzing time stamped historical data. Is that possible to build using BIRT?IT-Control Chart 8

SCMG’12 9 The case where the input data source is CSV file: BIRT based IT-Control Chart: Live demo But how to calculate that UCL and LCL numbers!?

SCMG’12 10 Weekhour calculation: How to calculate that UCL and LCL numbers!? Use EXCEL Pivot Table Pivot Table creation (IT-Chart Builder)IT-Chart Builder Pivot Chart built

SCMG’12 11 Data behind IT Control chart is a Cube: How to calculate that UCL and LCL numbers!? BIRT Data Cubes Usage for Performance Data Analysis Input data is raw stamped measures The result (in the form of BIRT report designer preview) is on the picture at left

SCMG’12 12 (1) The Reference set or base-line is based on the input raw data with some filtering and computed columns (weekday and weekhour) How to calculate that UCL and LCL numbers!? BIRT Data Cubes Usage for Performance Data Analysis (2) the Actual data set is the same but having the different filter: (raw[“date”] Greater “ ”)

SCMG’12 13 (3) To combine both data sets for comparing base-line vs. actual, the “Data Set1” is built as a “Joint Data Set” by the following BIRT Query builder: How to calculate that UCL and LCL numbers!? BIRT Data Cubes Usage for Performance Data Analysis (4) Then the Data Cube was built in the BIRT Data Cube Builder with the structure shown on the following screen: (5) The next step is building report starting with Cross Table (which is picked as an object from BIRT Report designer “Pallete”): The picture above shows also what fields are chosen from Cube to Cross table.

SCMG’12 14 (6) The final step is dropping “Chart” object from “Palette” and adding UCL calculation using Expression Builder for additional Value (Y) Series: How to calculate that UCL and LCL numbers!? BIRT Data Cubes Usage for Performance Data Analysis

SCMG’12 15 To see the result one needs just to run the report or to use a "preview' tab on the report designer window: BIRT Data Cubes Usage for Performance Data Analysis

SCMG’12 16 How to calculate that UCL and LCL numbers!? SQL script for MySQL Database (upload) (1) That raw data (the same) needs to be uploaded to some table (CPUutil) in the MySQL schema (ServerMetric) by using the following script (sqlScriptToUploadCSVforSEDS.sql):

SCMG’12 17 (2) Then the output (result) data (ActualVsHistoric table) is built using the following script (sqlScriptToControlChartforSEDS.sql): How to calculate that UCL and LCL numbers!? SQL script for MySQL Database (Calculate)

SCMG’12 How to calculate that UCL and LCL numbers!? R script against MySQL Database 18 The same SQL script transformed to the RODBC based R-Script:SQL script The result of the script run is the same "ActualVsHistorical" table in the servermentrics database on MySQL The result data is identical with the data used for plotting IT- Control Chart. The data itself can be seen by just typing the data frame name in the R-Console window:

SCMG’12 19 (3) In the BIRT, the connection to MySQL database is established (to MySQLti with schema ServerMetrics to table ActualVsHistorical): Building IT-Control Chart by BIRT against Data from the MySQL Database

SCMG’12 20 (4) the chart is developed the same way like that was done in BIRT based Control Chart post (slide 14):BIRT based Control Chart Building IT-Control Chart by BIRT against Data from the MySQL Database

SCMG’12 21 (5) In BIRT you can specify report parameters, that could be then a part of any constants including for filtering (to change a baseline or to provide server or metric names). Building IT-Control Chart by BIRT against Data from the MySQL Database (6) Finally the report should be run to get the following result, which is almost identical with the one built for BIRT based Control Chart post (slide 15)BIRT based Control Chart Lastly: R-system can be used to run the same SQL script against MySQL by using Package 'RMySQL‘!!!! Package 'RMySQL‘

SCMG’12 22 EV meta-metric introduced in 2001 as a measure of anomaly severity. ( The Exception Value Concept to Measure Magnitude of Systems Behavior Anomalies) The Exception Value Concept to Measure Magnitude of Systems Behavior Anomalies EV-Control Chart (to build using BIRT) It is the difference (integral) between actual data and control limits. In CMG’08 paper “Exception Based Modeling and Forecasting” That metric was plotted using Excel to explain how it could be used for a new trend starting point recognition: “Exception Based Modeling and Forecasting

SCMG’12 23 Plotting that meta-metric and/or two their components (EV+ and EV-) over time gives a valuable picture of system behavior. EV-Control Chart (to build using BIRT) The BIRT example of the EV-chart against the same sample data: EV calculation

SCMG’12 24 EV-Control Chart (to build using BIRT) The EXCEL example of the EV-chart against the same sample data:

SCMG’12 25 It is based on real script developed and successfully tested against real data for BIRT tool to get exceptional servers list: BIRT tool SEDS-Lite: Exception Detection (EV-based) using BIRT Below is example of DB2- like SQL script to implement that. Lite version: (UCL=LCL)

SCMG’12 SEDS elements to build by non-open source tools oTivoli Common Reporting portal to run report and get result: COGNOS example: oReport Studio to write reports: 26 o Framework Manager to build meta data model with some preliminary calculation like: WeekDay=(DAYOFWEEK([ODBC].[cpuutil].[date])-1)*24 +([ODBC].[cpuutil].[hour]) oSimple Exception Detector (EV top list)

SCMG’12 Benefits and Values of implementing SEDS and/or SETDS (Statistical Exception and Trend Detection) FeatureBenefit Automate the IT- and EV- Control Charts generation SEDS-liteHelps to reduce system defects and severe incidents; achieves a faster time to value for analysis; reduce manual work Implement Anomaly detection (as a part of SETDS) SEDS-liteReduce system severe incidents related to Performance and Capacity. Provides smart alerting based on system/apps behavior learning and will be used for truly proactive Capacity Management. Implement Automatic trend recognition (Statistical pattern recognition based) SETDSIncreases time for upgrade planning. Reduce manual work to build trend-forecast. A repeatable process allows the trend recognition to be applied to more metrics, increasing the field of view for potential issues and prediction accuracy. Automate Demand forecasts and trend- forecast reporting with Out-Layers free base- lines SETDSIncreases time for upgrade planning. Eliminate manual work to do correlation analysis and increases time for analysis and decision making. VALUE: Provide deeper insights into the use of technology platform resources and more time to react based on the insights, by correlating types of forecast data to resulting, sometimes subtle changes in technology platform utilization and capacity, and displaying them in a consistent, accurate, repeatable manner. 27

SCMG’12 28 References Global and Application Levels Exception Detection System, Based on MASF Technique Global and Application Levels Exception Detection System, Based on MASF Technique Igor Trubin: “ Global and Application Levels Exception Detection System, Based on MASF Technique ”, Proceedings of the Computer Measurement Group, ( Global and Application Levels Exception Detection System, Based on MASF Technique ( Disk Subsystem Capacity Management Based on Business Drivers I/O Performance Metrics and MASF Disk Subsystem Capacity Management Based on Business Drivers I/O Performance Metrics and MASF Linwood Merritt, Igor Trubin: “ Disk Subsystem Capacity Management Based on Business Drivers I/O Performance Metrics and MASF ”, Proceedings of the Computer Measurement Group, ( Disk Subsystem Capacity Management Based on Business Drivers I/O Performance Metrics and MASF ( Mainframe Global and Workload Level Statistical Exception Detection System, Based on MASF Mainframe Global and Workload Level Statistical Exception Detection System, Based on MASF Linwood Merritt, Igor Trubin: : “ Mainframe Global and Workload Level Statistical Exception Detection System, Based on MASF ”, Proceedings of the Computer Measurement Group, ( Mainframe Global and Workload Level Statistical Exception Detection System, Based on MASFhttp:// Capturing Workload Pathology by Statistical Exception Detection System Capturing Workload Pathology by Statistical Exception Detection System Igor Trubin: “ Capturing Workload Pathology by Statistical Exception Detection System ”, Proceedings of the Computer Measurement Group, ( Capturing Workload Pathology by Statistical Exception Detection System ( System Management by Exception, Part 6 System Management by Exception, Part 6 Igor Trubin: “ System Management by Exception, Part 6 ”, Proceedings of the Computer Measurement Group, ( System Management by Exception, Part 6 ( System Management by Exception, Part Final System Management by Exception, Part Final Igor Trubin: “ System Management by Exception, Part Final ”, Proceedings of the Computer Measurement Group, 2007.( ) System Management by Exception, Part Final Exception Based Modeling and Forecasting Exception Based Modeling and Forecasting Igor Trubin: “ Exception Based Modeling and Forecasting ”, Proceedings of the Computer Measurement Group, ( ) Exception Based Modeling and Forecasting IT-Control Chart IT-Control Chart Igor Trubin: “ IT-Control Chart ”, Proceedings of the Computer Measurement Group, ( ) IT-Control Charthttp://itrubin.blogspot.com/2010/07/my-new-cmg10-paper-it-control-charts.html

SCMG’12 29 SCMG’12 Questions? Everything is ready to build SEDS-Lite: Using Open Source Tools (R, BIRT and MySQL) Igor Trubin, PhD BLOG: