2. 1 1 Using Level 5 ROI to Ensure Training Success Breakout Session # Name: Wayne R. Brantley, MS Ed., PMP, CRP, ITIL, CPLP Senior Director of Professional Education Villanova University Date: July 12 th 2011 Time: 2:30 PM

3. 2 Professional Education Online Six Sigma Project Management Contract Management IS Security Business Analysis Software Testing Human Resources ROI Methodology Leadership For more information, please call 1-800-571-4953 or visit www.VillanovaU.com

4. 3 3 Agenda Introduce you to the ROI Methodology Understand the benefits of implementing the ROI Methodology in your organization Connect ROI to contract management initiatives Explain the ROI process

5. 4 4 What is ROI? ROI= Net Monetary Benefits Program Costs X 100

6. 5 5 WRONG! Okay, right if you’re an MBA It is a story of how you collected the data Where the data comes from Shows all the numbers Shows who deserves what credit Ties the training to the business strategically Reports intangibles!!! It is a methodology

7. 6 6 Global Trends in Measurement and Evaluation Investment is increasing to 3-5% of the budget Organizations are maturing in their evaluation strategies Increased focus driven by clients and sponsors ROI is the fastest growing metric Evaluation data is used to drive improvement and secure funding Processes are systematic and methodical, often designed into the delivery and implementation processes

8. 7 7 Basic Elements An Evaluation Framework Case Applications and Practice Implementation A Process Model Operating Standards and Philosophy

9. 8 8 Basic Elements An Evaluation Framework

10. 9 9 LevelMeasurement Focus 1.Reaction & Planned Action Measures participant satisfaction with the program and captures planned actions, if appropriate. 2.LearningMeasures changes in knowledge, skills, and attitudes related to the program. 3.ApplicationMeasures changes in on-the-job behavior or actions as the program is applied, implemented, or utilized. 4.Business ImpactMeasures changes in business impact variables. 5.Return on InvestmentCompares program benefits to the costs. ROI is reported in the five-level evaluation framework

11. 10 Basic Elements A Process Model

12. 11 Text

13. 12 Basic Elements Operating Standards and Philosophy

14. 13 Operating Standards and Philosophy – Guiding Principles 1. When a higher level evaluation is conducted, data must be collected at lower levels. 2. When an evaluation is planned for a higher level, the previous level of evaluation does not have to be comprehensive. 3. When collecting and analyzing data, use only the most credible sources. 4. When analyzing data, choose the most conservative among alternatives. 5. At least one method must be used to isolate the effects of the project/initiative. 6. If no improvement data are available for a population or from a specific source, it is assumed that little or no improvement has occurred.

15. 14 Operating Standards and Philosophy – Guiding Principles 7. Estimates of improvement should be adjusted for the potential error of the estimate. 8. Extreme data items and unsupported claims should not be used in ROI calculations. 9. Only the first year of benefits (annual) should be used in the ROI analysis of short-term projects/initiatives. 10. Costs of the solution should be fully loaded for ROI analysis. 11. Intangible measures are defined as measures that are purposely not converted to monetary values. 12. The results from the ROI methodology must be communicated to all key stakeholders.

16. 15 Basic Elements Case Applications and Practice

17. 16 Basic Elements Implementation

18. 17 Basic Elements An Evaluation Framework Case Applications and Practice Implementation A Process Model Operating Standards and Philosophy

19. Develop Objectives of Solution(s) Develop Objectives of Solution(s) Develop Evaluation Plans and Baseline Data Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Collect Data During Solution Implementation Collect Data After Solution Implementation Collect Data After Solution Implementation Level 2: Learning Level 4: Business Impact Level 1: Reaction, Satisfaction, and Planned Actions Level 3: Application/ Implementation (1) Evaluation Planning Data Collection THE ROI PROCESS Calculating the Return on Investment of a Business Performance Solution Data Collection Plan ROI Analysis Plan Project Plan

20. 19 Develop Objectives of the Solution Why are you implementing the training? What do you expect the improvement to be? Can you quantify?

22. Evaluation Planning ROI Analysis Plan –Data Items (Usually Level 4 Impact) –Methods for Isolating the Effects of the Program/Process –Methods of converting Data to Monetary Values –Cost Categories –Intangible Benefits –Communication Targets for Final Report –Other Influences/Issues during Application –Comments

23. 22 Evaluation Planning Project Plan Major Milestones Deliverables Timelines Sequence

24. Develop Objectives of Solution (s) Develop Objectives of Solution (s) Develop Evaluation Plans and Baseline Data Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Collect Data During Solution Implementation Collect Data After Solution Implementation Collect Data After Solution Implementation Level 2: Learning Level 4: Business Impact Level 1: Reaction, Satisfaction, and Planned Actions Level 3: Application/ Implementation EvaluationPlanning (2) Data Collection THE ROI PROCESS Calculating the Return on Investment of a Business Performance Solution

25. 24 Level 1 (Reaction) and Level 2 (Learning) Objectives Provide: Feedback on program Feedback on delivery Knowledge and performance skills acquired

26. 25 Data Collection During Program MethodLevel 1Level 2  Surveys    Questionnaires    Observation    Interviews    Focus Groups    Tests 

27. 26 Level 3 (Application) and Level 4 (Impact) Objectives Provide: Ties the training to the initiative Direction to designers and developers Guidance to instructors and facilitators Goals for participants Satisfaction for program sponsors A framework for evaluators Tie to strategic goals for training

28. 27 Data Collection Post Program MethodLevel 1Level 2 Surveys  Questionnaires   Observations on the job  Interviews  Focus Groups  Action planning/improvement plans   Performance contracting   Performance monitoring  

29. Isolate the Effects Isolate the Effects Convert Data to Monetary Value Convert Data to Monetary Value Calculate the Return on Investment Calculate the Return on Investment Tabulate Costs of Solution Tabulate Costs of Solution Identify Intangible Measures Identify Intangible Measures Generate Impact Study Generate Impact Study Level 5: ROI Intangible Measures (3) Data Analysis Reporting

30. 29 Isolating Methods Isolation shows your contribution Techniques used Control groups Trend line analysis Forecasting methods End user/performer’s, supervisor’s, and/or management’s estimate of impact (percent) Use of experts/previous studies Subordinate’s input of program impact Calculate/Estimate the impact of other factors Customer input

31. 95% Use of Trend Line Analysis Shipment Productivity Months Percent of Schedule Shipped J F M A M J J A S O N D J 100 % 90% 85% Team Implementation Actual Average 94.4% Average of Trend Projected 92.3% Pre Program Average 87.3% Trend Projection

32. 31 Example of a Participant’s Estimation

33. 32 Converting Data to Monetary Value Challenging Use data bases Look at profits / savings from output Historical costs / savings Experts input End user input Staff estimation

34. 33 Example of Converting Data Using External Database Cost of one turnover* Middle Manager $70,000 annual salary Cost of turnover150% Total cost of turnover$105,000 * External data - value obtained from industry related study

35. Example of Converting Data Using Historical Records & Expert Input The Cost of a Sexual Harassment Complaint Actual Costs from Records Legal Fees, Settlements, Losses, Material, Direct Expenses 35 Complaints Estimated Additional from Staff EEO/AA Staff Time, Management Time $852,000 Annually $852,000 35 Cost Per Complaint = $24,343

36. Isolate the Effects Isolate the Effects Convert Data to Monetary Value Convert Data to Monetary Value Calculate the Return on Investment Calculate the Return on Investment Tabulate Costs of Solution Tabulate Costs of Solution Identify Intangible Measures Identify Intangible Measures Generate Impact Study Generate Impact Study Level 5: ROI Intangible Measures Data Analysis (4) Reporting

37. Intangible Benefits Complaints Conflicts Stress Job Satisfaction Commitment Teamwork Customer Service Engagement

38. 37 Calculating Costs Cost of study Travel Materials Facilities Salaries Hard costs Soft costs

39. 38 Calculating ROI Benefits/Cost Ratio ROI Monetary Benefits Program Costs Net Monetary Benefits Program Costs = = X 100

40. 39 ROI Example Costs for project$80,000 Benefits from project $240,000 $240,000 $80,000 $160,000 $80,000 BCR= ROI== = 200% 3.0 100X=

41. 40 ROI Impact Reports Documented process Traceable Publish and tell your story Make into a part of your culture

42. 41 Criteria for Selecting Programs for Levels 4 (Impact) & 5 (ROI) Expected life cycle of the project The importance of contract management in meeting the organization’s goals Cost of the project Visibility of the project The size of the target audience Extent of management interest

43. 42 ROI Target Options 1.Set the value as with other investments, e.g. 15% 2.Set slightly above other investments, e.g. 25% 3.Set at break even - 0% 4.Set at customer expectations ROI

44. 43 To what types of programs are organizations applying the ROI Methodology? Contract Management Apprenticeship Programs Sales Training Leadership Development Competency Systems Diversity Programs E-Learning Executive Coaching Technical Training Gain Sharing Programs Project Management Retention Solutions Organization Development Orientation Systems Recruiting Strategies Safety & Health Programs Self-Directed Teams Skill-Based/Knowledge- Based Compensation Technology Implementation Quality Management Volunteer Training

45. 44 The ROI Process Takes A Balanced View by Measuring and Reporting: Reaction to program Learning and attitudes Application on the job Impact in work unit Impact on the customer The financial results Intangible benefits Nature and source of problems and opportunities FinancialIntangible Employee Processes Customer Employee Processes Customer

46. 45 Next Steps Show management your value Identify what you want an ROI study done on Determine if you are doing this internal or external Learn the ROI methodology Villanova University Online certification

47. 46 Using Level 5 ROI to Ensure Training Success Are there any questions? Thank you! For more information: www.VillanovaU.com800-571-4953www.VillanovaU.com