1. Adding Richness to Measurement A Case for Developing and Using Complex Measures

2. Data is Not Information The Search for Meaning in Measures Meaning and Methodology - the Medium is the Message Meaning and Methodology - the Medium is the Message Multiple Users/Stakeholders Multiple Users/Stakeholders Reporting Versus Quantitative Analysis Reporting Versus Quantitative Analysis Measuring Complex Outcomes Measuring Complex Outcomes Enterprise-Level Activities Enterprise-Level Activities

3. Complexity - Multiple Stakeholders The Public The Public Other Agencies - Entities Other Agencies - Entities Budgeting Budgeting Program Funding Outcomes Program Funding Outcomes Policy Decision Outcomes Policy Decision Outcomes Evaluating Agency - Vendor Performance Evaluating Agency - Vendor Performance

4. Complex Outcomes Multiple Players ( in a Stovepipe System ) – Enterprise Level Activities Multiple Players ( in a Stovepipe System ) – Enterprise Level Activities Significant Number/Scope of Independent Variables ( Limited Control & Influence over Many Primary Outcomes ) Significant Number/Scope of Independent Variables ( Limited Control & Influence over Many Primary Outcomes ) Non-linear Processes ( starts, stops, shifts, drops, etc.) Non-linear Processes ( starts, stops, shifts, drops, etc.) Hypothetical Nature of Many Public Sector Activities Hypothetical Nature of Many Public Sector Activities

5. What does a typical KPM data chart really communicate? Standard format is a column chart with a target Line overlay Standard format is a column chart with a target Line overlay Expressions are most often a yearly raw Mean Expressions are most often a yearly raw Mean The format often implies variation in performance when differences are just normal process variation The format often implies variation in performance when differences are just normal process variation

6. What Lies Beneath …

7. And … in case you think I am making this up … real data from a real agency

8. And you find out things about your process you didnt know before …

9. Aggregate Measures – Selling Points Primary expression is a single expression dashboard indicator ( Easy to understand – Easy to track ) Primary expression is a single expression dashboard indicator ( Easy to understand – Easy to track ) Statistically based ( mathematically verifiable – easy to audit ) – immediately useful for process improvement purposes Statistically based ( mathematically verifiable – easy to audit ) – immediately useful for process improvement purposes Properly constructed indexes can be de- aggregated to provide increasingly granular detail back to the original raw datasets Properly constructed indexes can be de- aggregated to provide increasingly granular detail back to the original raw datasets Can combine different types of data into the same measure Can combine different types of data into the same measure

10. Aggregate Measures – Selling Points Provides a powerful analytic – process improvement tool Provides a powerful analytic – process improvement tool Provides more complete, compelling and valid data for budget support Provides more complete, compelling and valid data for budget support Organizations can use a combination of related operational measures to create a single outcome index ( fewer measures, and little need for multiple part measures in the system ) Common Indices ( Organizational Health, Timeliness of Process, Process Improvement, Customer Service, etc.) Organizations can use a combination of related operational measures to create a single outcome index ( fewer measures, and little need for multiple part measures in the system ) Common Indices ( Organizational Health, Timeliness of Process, Process Improvement, Customer Service, etc.) Allows for updating and adjusting measure components without the need for a formal delete/replace (?) Allows for updating and adjusting measure components without the need for a formal delete/replace (?)

11. Constructing Aggregate Measures What is the Outcome? What is the Outcome? What are the Primary Components of the Outcome? What are the Primary Components of the Outcome? What are the Critical Measures of the Components? What are the Critical Measures of the Components? Normalizing Data – ( removing outliers and translating data into a common unit of expression ) Normalizing Data – ( removing outliers and translating data into a common unit of expression ) Weighting Components Weighting Components

12. Outcomes in the Public Sector Change in Status Change in Status Change in Capability Change in Capability Client/Customer Satisfaction Client/Customer Satisfaction Process Outcomes – Efficiency/Effectiveness 1. Timeliness 2. Defects (errors, rework) 3. Cost Reduction (savings, avoidance) Process Outcomes – Efficiency/Effectiveness 1. Timeliness 2. Defects (errors, rework) 3. Cost Reduction (savings, avoidance) DEFINED Outcomes DEFINED Outcomes

13. Normalizing Data Distribution Analysis Data shape (distribution) Removing outliers – Special Causes of Variation = (Mean +/- 2 Standard Deviations) Upward and Downward Process Control Limits Baseline-ing Distribution Analysis Data shape (distribution) Removing outliers – Special Causes of Variation = (Mean +/- 2 Standard Deviations) Upward and Downward Process Control Limits Baseline-ing Combining Unlike Data Converting to a common expression - % of target Combining Unlike Data Converting to a common expression - % of target

14. Weighting Criteria Contribution to Outcome ( High, Moderate, Low ) Contribution to Outcome ( High, Moderate, Low ) Criticality ( Death, Dismemberment, Skin Rash ) Criticality ( Death, Dismemberment, Skin Rash ) Frequency ( Constantly, Sometimes, Rarely ) Frequency ( Constantly, Sometimes, Rarely ) Data Reliability (. 99999, OK, Flip a Coin ) Data Reliability (. 99999, OK, Flip a Coin )

15. Examples BOLI ( Bureau of Labor and Industries ) Composite Timeliness Measure ( Wage and Hour, Civil Rights ) BOLI ( Bureau of Labor and Industries ) Composite Timeliness Measure ( Wage and Hour, Civil Rights ) Department of Revenue Taxpayer Assistance Department of Revenue Taxpayer Assistance DHS-Courts-CCF Shared Permanency of Placement DHS-Courts-CCF Shared Permanency of Placement

16. Civil Rights Division Timeliness Index CRD MeanMedianSTDTarget % of Initial Mean SME Weighting Component Targets Component Actuals CRD Phase One2.121.31.785851 100 CRD Phase One-B64.9663958.41900.7567.575 CRD Phase Two423524.237.8900.54550 CRD Phase Three171.2130119.2162.64952190200 387.5425Totals 109.68% % of Target

17. Putting it all Together Index Components Effective Discovery – Disclosure of Legal Records