1. Measuring (and Driving) the Value of Training Bruce Winner, Los Rios CCD – Government Training Academy Bruce blogs to the training community at - www.GetResultsFromTraining.com

2. *Find out the difference between a simple evaluation and truly measuring the value of training at the five generally accepted levels. *Discover what measures are most important to management, and how to capture that information. *Stop guessing about the appropriate level of measurement and see what the experts and empirical evidence reveals is optimal. *Explore with your colleagues how to create the systems and practices necessary (before, during, and after training), to ensure that your hard efforts result in real value, measured at an appropriate level. Objectives

3. Is it Worth the Effort? What would being able to measure and prove the value of training --- do for YOU?

4. ROI Methodology vs Kirkpatrick Four Levels Not mutually exclusive Lots of overlap - ROI methodology built on the Kirkpatrick Model ROI methodology vs ROI (the ratio) I will use BOTH going forward, but…

5. Over 50 ROI Methodology books and case studies in print

7. ROI or Kirkpatrick Levels of Measurement LevelMeasurement Focus 0. Input Measures input such $ spent, people trained, $ per trainee 1. Reaction and Relevance Measures participant reaction to the program, did they find it relevant, and could capture planned action(s) 2. LearningMeasures changes in knowledge, skills, and attitudes 3. Application & Implementation Measures what is applied on the job, changes in job behaviors (more of x, less of y) 4. Business Impact (Results) Captures changes in business impact measures (Q.C.O.T) 5. ROI Compares program benefits to program costs (simple ration or benefits minus cost / cost)

8. Intangible Benefits Adaptability Brand awareness Career minded Caring Collaboration Communication Conflicts Cooperation Corporate social responsibility Creativity 8 Confidence (story) Culture Decisiveness Employee complaints Engagement Execution Image Innovation Job Satisfaction Leadership Networking Organizational climate Organizational commitment Partnering Reputation Resilience Stress Talent Teamwork

9. 9 Level 5: 5-10% Level 1: 90-100% Level 2: 40-60% Level 3: 30% Level 4: 10-20% Evaluation Targets

10. The Executive View of Metrics / RESULTS MeasureWe Currently Measure This We Should Measure this in the Future My ranking of the importance of this measure 1. Inputs: “Last year, 78,000 employees received formal learning.” 94% 2. Efficiency: “Formal learning costs $2.15 per hour of learning consumed.” 78% 3. Reaction: “Employees rated our training very high, averaging 4.2 out of 5.” 53% 4. Learning: “92% of participants increased knowledge and skills” 32% 5. Application: “At least 78% of employees are using the skills on the job” 11% 6. Impact: “Our programs are driving our top 5 business measures in the organization.” 8% 7. ROI: “Five ROI studies were conducted on major programs yielding an average of 68% ROI.” 4% 8. Awards: “Our learning and development program won an award from American Society for Training and Development 40%

11. The Executive View of Metrics / RESULTS MeasureWe Currently Measure This We Should Measure this in the Future My ranking of the importance of this measure 1. Inputs: “Last year, 78,000 employees received formal learning.” 94% 85%6 2. Efficiency: “Formal learning costs $2.15 per hour of learning consumed.” 78% 82%7 3. Reaction: “Employees rated our training very high, averaging 4.2 out of 5.” 53% 22%8 4. Learning: “92% of participants increased knowledge and skills” 32% 28%5 5. Application: “At least 78% of employees are using the skills on the job” 11% 61%4 6. Impact: “Our programs are driving our top 5 business measures in the organization.” 8% 96%1 7. ROI: “Five ROI studies were conducted on major programs yielding an average of 68% ROI.” 4% 74%2 8. Awards: “Our learning and development program won an award from American Society for Training and Development 40% 44%3

12. Improving Level 1s Level 1-2 2 - 2.5 2 - 3 3 - 4

14. 14 The Alignment Process Start Here End Here 5 ROI Objectives 5 4 Impact Objectives 4 3 Application Objectives 3 2 Learning Objectives 2 1 Reaction Objectives 1 Project Payoff Needs Business Needs Job Performance Needs Learning Needs Preference Needs ROI Impact Application Learning Reaction Initial Analysis Measurement and Evaluation Business Alignment and ForecastingThe ROI Process Model

15. Matching Evaluation Levels To Objectives Table exercise – Short exercise to reinforce differences in LEVELS of Objectives

16. Let’s look at five training objectives (below) and see if we can determine their “level” (level 1-5) Perceive the new work schedule system introduced during training as important to agency success. Level ____ Achieve a post-test score increase of 30% over pre-test. Level ____ Use acceptable coaching skills in 90% of situations where work habits are unacceptable.Level ____ Decrease the time from testing to hiring of new analysts from 40 days to 15 days.Level ____ Achieve a 50% return on investment with the new contact center training program for customer service representatives. Level ____

17. Now it’s your turn! Go to page 6 of your handout. Instructions: For each objective listed below, indicate the level of evaluation at which the objective is aimed. This is more fun as a table-top exercise. Work as a group and try to match an evaluation level to EACH objective in as short a time as possible…

18. DANGER DANGER DANGER

19. 19

21. ParticipantTrainerSupervisor / Manager BEFORE Training Remember our earlier example from today? … Trainer creates objectives at four or five levels… even though they may NOT be measured DURING Training Remember our earlier example from today? … Participants submit new feedback via a Redesigned Level ONE Evaluation AFTER Training A Supervisor Example …Example – The supervisor works with the training participant weekly for 8 weeks after the training to make sure application takes place…. Some Transfer of Training Examples for an Upcoming Exercise The Transfer of Training MATRIX

22. Transfer of Training - Exercise Directions 1) Do the exercise as a table group 2) Each Table has a number (1-9) that corresponds to one of nine squares on the T of T Matrix (see below) THEN – Use the oversized Post-IT Notes to: 3) Each table creates at least one “intervention” or “means to increase transfer of training” at a specific point in time (before, during, or after) with one specific group (participants, trainers, or supervisors) - Your number 1-9 is “your box” 4) Then produce 1 or 2 additional interventions for ANY BOX in the matrix 5) Post ONLY two to the wall chart (one required and one extra) ParticipantTrainerSupervisor / Manager BEFORE Training 123 DURING Training 456 AFTER Training 789

23. Thank you! Please call me if you have any questions or want to talk about “Measuring (or Driving) the Value of Training”. Bruce Winner 916.563.3232 winnerb@losrios.edu And please check out my blog… www.GetResultsFromTraining.comwww.GetResultsFromTraining.com (An unread blog is a terrible thing!)