1. Phillips Associates 1 Boost Training Transfer Using Predictive Learning Analytics™ Presented by: Ken Phillips Phillips Associates May 22, 2016 Session: SU 217 1:30-2:30 PM

2. Phillips Associates 2 Agenda 1. Discover the impact “scrap learning” has on wasted organization resources & lost credibility with business executives 2. Analyze how to build a PLA model that identifies those learners who are most & least likely to apply what they learned 3. Examine the 7 benefits of using PLA in your organization & the 3 phases & 9 steps for getting started

3. Phillips Associates 3 Scrap learning – What is it?

4. Phillips Associates 4 Term coined by KnowledgeAdvisors, a CEB company, that describes the difference between learning that’s delivered and learning that’s applied back on the job Scrap Learning

5. Phillips Associates 5 How big is the problem?

6. Phillips Associates 6 Benchmark Study 1 45% Source: Confronting Scrap Learning CEB White Paper, 2014

7. Phillips Associates 7 Benchmark Study 2 < 20% Applied new skills back on the job > 15% Didn’t try to apply new skills back on the job 65% Tried applying new skills back on the job but reverted back Source: Robert Brinkerhoff, 2004

8. Phillips Associates 8 View from individual organization level

9. Phillips Associates 9 According to ATD 2015 “State of the Industry Report” Average per employee training expenditure Average number of training hours consumed per employee $1229 32.4 = =

10. Phillips Associates 10 Calculating Scrap Learning at individual organization level $1229 X 45% = 32.4 hours X 45% = $1229 X 80% = 32.4 hours X 80% = $553 14.6 $983 25.9

11. Phillips Associates 11 Houston, we have a problem! Source: James Lovell, Apollo 13 flight

12. Phillips Associates 12 The solution: Predictive Learning Analytics ™

13. Phillips Associates 13 is a methodology for peering into the future, at the conclusion of a learning program, and forecasting learner outcomes and actions, with the intent of changing those outcomes and actions for the better. Predictive Learning Analytics™(PLA) Source: Ken Phillips

14. Phillips Associates 14 PLA vs. Traditional Measurement & Evaluation

15. Phillips Associates 15 Kirkpatrick / Phillips Evaluation Model Levels of EvaluationMeasurement FocusTime Frame Level 1: Reaction Participant favorable reaction to a learning program Conclusion of learning program Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes Conclusion of learning program or within 6 to 8 weeks after Level 3: Behavior Degree to which participants applied back-on-the-job what was learned 2 to 12 months Level 4: Results Degree to which targeted business outcomes were achieved 9 to 18 months Level 5: ROI Degree to which monetary program benefits exceed program costs 9 to 18 months

16. Phillips Associates 16 PLA vs. Traditional Learning M&E High Low PastFuture Value Time Traditional Learning M&E Predictive Learning Analytics Traditional Learning M&E Focuses on programs or cohorts Describes what has happened Predictive Learning Analytics Focuses on individual learners Predicts future likelihood of certain behaviors and actions

17. Phillips Associates 17 Algorithm consisting of 10 factors that are known to contribute to training transfer & research shows have a strong positive correlation with either Level 2 learning, Level 3 behavior or Level 4 results Heart of PLA:

18. Phillips Associates 18 Instructions 1. Form a group of 3, 4 or 5 persons 2. Brainstorm a list of things we know to be “truths” about training transfer 3. Be prepared to share your ideas with the whole group Example: Training transfer increases when learners have an immediate opportunity to apply what they learned back on the job

19. Phillips Associates 19 Level 2 Training Transfer Factors 1. Acquire new information – knowledge, skills or attitudes 2. See a program as relevant to themselves and their job 3. See a program as an important investment in their career development Learners need to:

20. Phillips Associates 20 Level 3 Training Transfer Factors 4. Be personally motivated to apply what was learned 5. Have confidence in their ability to apply what was learned 6. Reflect on key lessons learned & how they can help improve their performance Learners need to:

21. Phillips Associates 21 Level 3 Training Transfer Factors (Cont.) 7. Be actively engaged by their manager post-program regarding what was learned 8. Be supported by work colleagues, post- program, to apply what was learned 9. Have an immediate opportunity to apply what was learned Learners need to:

22. Phillips Associates 22 Level 4 Training Transfer Factor 10. See a likely improvement in a key business metric tracked by their department if new information learned is applied Learners need to:

23. Phillips Associates 23 Using the Algorithm

24. Phillips Associates 24 Using The Algorithm Step 1: Convert factors into survey items and include as part of a Level 1 evaluation or administer as a separate survey Step 2: Collect data from Calibration Cohort – first 60-70 participants to attend PLA target training program

25. Phillips Associates 25 Using The Algorithm Step 4: Target participants who are at risk & least likely to apply what they learned for follow-up & reinforcement activities in order to increase training transfer Step 3: Calculate Learner Application Index™ score for each Calibration Cohort program participant

26. Phillips Associates 26 LAI Individual Scores

27. Phillips Associates 27 LAI Factor Scores

28. Phillips Associates 28 Additional Calculations

29. Phillips Associates 29 Additional Calculations Manager Training Support Index Manager Training Support Index™ Sort learners into groups arranged by manager Compute Manager Training Support Index™ score for each manager & identify managers who do a good & poor job of supporting learning Work with those managers who need help doing a better job

30. Phillips Associates 30 Manager Training Support Index™ Score

31. Phillips Associates 31 Additional Calculations Compute Overall Program Quality score & compare the quality of one program with another Target programs not delivering value for either revision or elimination Overall Program Quality

32. Phillips Associates 32 Benefits of Using Predictive Learning Analytics

33. Phillips Associates 33 1. Less money & time wasted on learning that is delivered but not applied back on the job – scrap learning 2. Benefits Of Using PLA 3. Increased personal credibility in eyes of business executive stakeholders More effective & efficient use of follow-up activities by targeting participants who are at risk & least likely to apply what they learned in a program back on the job

34. Phillips Associates 34 5. 4. Benefits Of Using PLA (cont.) Objective way to identify managers who do a poor job of supporting learning so that their approach can be improved Objective way to evaluate the effectiveness of a learning program design

35. Phillips Associates 35 6. Objective way to compare the overall quality of one learning program with another using a single number 7. Benefits Of Using PLA (cont.) Enhanced reputation among L&D colleagues

36. Phillips Associates 36 3 Phases & 9 Steps for Implementing PLA

37. Phillips Associates 37 PLA Phases & Steps

38. Phillips Associates 38 Summary The issue of scrap learning has been around forever. But, what’s different today is that with Predictive Learning Analytics™ there now is a way to manage it. Source: Ken Phillips

39. Phillips Associates 39

40. Phillips Associates 40 Phillips Associates ken@phillipsassociates.com (847) 231-6068 www.phillipsassociates.com 34137 N. Wooded Glen Drive Grayslake, Illinois 60030 Ken Phillips

41. Phillips Associates 41 Ken Phillips, PhD, CPLP He regularly speaks to Association for Talent Development (ATD) groups, university classes and corporate learning and development departments. Since 2008, he has spoken at the ATD International Conference on topics related to measurement and evaluation of learning. In addition, in 2015 he was invited to present at the ATD Middle East North Africa (MENA) Conference and the ATD China Summit on topics related to measurement and evaluation of learning. Prior to pursuing a Ph.D. in the combined fields of organization behavior and educational administration at Northwestern University, Ken held management positions with two colleges and two national corporations. In addition, he has written articles that have appeared in td magazine, Training Today and HR.com, and is a contributing author to four books in the L&D field. Ken earned the Certified Professional in Learning and Performance (CPLP ® ) credential from ATD in 2006 as a pilot pioneer and was recertified in 2009, 2012 and again in 2015. Ken is founder and CEO of Phillips Associates, a consulting and publishing company with expertise in measurement and evaluation of learning and performance management. He has more than 30 years experience designing learning instruments and assessments and has authored more than a dozen published learning instruments. LinkedIn: www.linkedin.com/in/ken-phillips-3420b11www.linkedin.com/in/ken-phillips-3420b11

42. Phillips Associates 42 Select a single, high profile, and costly learning program for your PLA project and initially “stay under the radar” Implementation Guidelines 2. Phase 1: Setting the stage 1. Build your Predictive Learning Analytics™ algorithm

43. Phillips Associates 43 Collect data & calculate individual Learner Application Index™ score for each program participant Implementation Guidelines 3. Phase 1: Setting the stage (cont.) Calculate “scrap learning” percentage baseline score associated with program 4.

44. Phillips Associates 44 Implementation Guidelines Phase 2: Implementing the methodology 5. Identify learners at risk & least likely to apply what was learned back on job & target for follow-up activities 6. Conduct Level 2 & Level 3 evaluations in order to validate accuracy of PLA algorithm 7. Recalculate scrap learning percentage following implementation of follow-up activities

45. Phillips Associates 45 Implementation Guidelines Phase 3: Sharing your success Report results to business executive stakeholders 8. 9. Enhance accuracy of PLA algorithm by including additional data from company LMS or HRIS, if available

46. Phillips Associates 46 The goal of training: “Helping learners achieve great results, not providing great training!” Source: Robert Brinkerhoff