1. Measuring Development Impact: Beyond Satisfaction Deloitte Services LP Janelle HughesJohn DeVille Development LeaderTalent Analytics Manager

2. 2Copyright © 2015 Deloitte Development LLC. All rights reserved. 4 Leaders Are Asking Difficult-To-Answer Questions Are we getting good value from our investment in training? Are employees applying learning on the job and achi evi ng positi ve resul ts? How is training affecting key business metrics? What additional training or support would help people perform better? We have multiple programsaddressing similar learning needs. Which one(s) are most effecti ve? How can we improve current learning and development programs? How can I use data to provide feedback and coaching to my team members? There is a burning platform for organizations to demonstrate the value of the L&D function and impact on business performance, and to further align with the business strategy.

3. 3Copyright © 2015 Deloitte Development LLC. All rights reserved. We introduced three key changes to the way we measure impact Metrics Aligned to Impact Dimensions Expanded from satisfaction questions to include multiple categories based on Bersin’s Impact Measurement Framework Interactive Reports and Dashboards Self-service, highly graphical and interactive format for visualizing data Learner and Manager Follow-Up Surveys Available 30-90 days after course end to measure on- the-job application and business impact Our Journey Towards Analytics Maturity Started with an Enhanced Measurement Strategy

4. 4Copyright © 2015 Deloitte Development LLC. All rights reserved. Most organizations frequently use satisfaction metrics to measure training effectiveness instead of job impact metrics are most valuable Metrics Are Not Aligned with Desired Outcomes Copyright © 2015 Deloitte Development LLC. All rights reserved. 6 What are the most valuable m easures? (% of organizations to rated the following m easures Extremely Valuable) 26% 31% 33% 36% 35% 39% 41% 44% 43% 47% 65% 72% 76% 0%20%40%60%80% Job Impact Business Impact Business Metrics Total Cost Manager Satisfaction Learning Satisfaction ROI Compliance Cost to Deliver Cost to Develop Cost per Hour Completions Enrollments Hours Delivered What is frequently measured? (% of organizations that routinely m easure the following areas) 8% 5% 10% 14% 19% 31% 35% 37% 46% 63% 81% 88% 86% 0%20%40%60%80% 100% Completions Enrollments Satisfaction Student Hours Compliance Total Cost Scores Cost to Deliver Cost to Develop Cost per Student Manager Satisfaction Job Impact Business Impact Actual Business Impact ROI Source: High-Impact Talent Analytics, Bersin by Deloitte, October 2013 Based on self reported data as part of research conducted in measurement and analytics

5. 5Copyright © 2015 Deloitte Development LLC. All rights reserved. Developing a High-Impact Measurement Strategy Copyright © 2015 Deloitte Development LLC. All rights reserved. 9 The most valuable metrics to business leaders are: Impact Focused Actionable Strategic Strategic – scalable and support the long term needs of the organization Actionable – create ability to “take action” and address or make a change Impact Focused – support decisions that will affect business decisions Information resulting from analysis of these metrics can be used to support future development initiatives, identify trends in learner behavior, and demonstrate impact on business priorities.

6. 6Copyright © 2015 Deloitte Development LLC. All rights reserved. Evaluation questions fall into four primary dimensions that measure the impact of the program Learning Impact Dimensions Satisfaction Utility Performance Learning Did learners like the course, materials, environment, etc.? Are learners using what they learned on the job? Did learners really learn something? Did the course improve a learner’s performance? We are looking beyond satisfaction measures to consider the dimensions that drive impact By expanding what we measure, we are creating a fuller picture to assess the impacts of our programs on key business outcomes The more actionable and telling data will allow us to better use it to deliver value What does this mean?

7. 7Copyright © 2015 Deloitte Development LLC. All rights reserved. In small groups, answer these questions for a single dimension: 1.How could data from this dimension be used to demonstrate impact related to business priorities? 2.What questions would result in the most valuable data in this dimension? What Evaluation Metrics Are the Most Valuable Utility Are learners using what they learned on the job? 1) 2) 3) Performance Did the course improve a learner’s performance Did learners really learn something? Learning Did learners like the course, materials, environment, etc.? Satisfaction 1) 2) 3) 1) 2) 3) 1) 2) 3)

8. 8Copyright © 2015 Deloitte Development LLC. All rights reserved. Approach Enabled a Shift From a Focus on Course Satisfaction to Several Key Drivers of Impact Follow-up surveys use multi-rater input to assess on- the-job application We previously focused chiefly on satisfaction as our primary outcome measure An expanded set of metrics broadens our ability to capture impact

9. 9Copyright © 2015 Deloitte Development LLC. All rights reserved. We Continue to Build Upon the Measurement Strategy to Drive Further Impacts Integrated Analytics: Job/Project Contribution Client Satisfaction Retention Engagement Capability/ Competency Growth Learning Utility Adoption Performance Attainment Alignment Predictive/Causal Analytics Learning Program(s) vs. corporate outcomes (market growth, revenue, profitability) Learning Program(s) vs. business outcomes (client satisfaction, profitability) Learning Program(s) vs. business unit, regional outcomes Standardized course evaluations Manager feedback Integrated talent outcome analytics Integrated business outcome analytics Short-termMid-termLonger-term

10. 10Copyright © 2015 Deloitte Development LLC. All rights reserved. How to Employ an Enhanced Course Evaluation at Your Organization Adopt a strategic measurement framework Use a framework to guide your efforts. Include everything that’s important (not just what you can currently measure) Use a logic model for course design Think about measurement before program design and development, not after delivery. Explicitly describe the business problem  learning objectives  on-the-job behaviors  job-level results  organization-level results. Use it with your clients. Design evaluation questions from the top down Define the dimensions you want to measure, then create questions to measure those dimensions Analyze, report, and act on data by dimension The best is the enemy of the good Start small. What do you need to know? What can you measure now? Address the gaps over time Hard measures are great, but indicators will do. Self-reported outcomes are directionally correct and easy to get