1. HOW TO ADDRESS THE THREE MOST CHALLENGING ROI ISSUES Jack J. Phillips, Ph.D. Chairman, ROI Institute 1

2. Objectives After completing this session, participants should perceive the content to be: Relevant to their work Immediately applicable Participants will be able to: Explain the benefits of using return on investment (ROI). Describe the steps in the ROI Methodology. Explain the three most challenging areas in an ROI study. Identify the principal uses of ROI data. 2

3. 3 Show Me! Collect Impact Data Show Me the Money! And Convert Data to Money Show Me the Real Money! And Isolate the Effects of the Project Show Me the Real Money, And Make me Believe it! And Compare the Money to the Cost of the Project Term Issue The “Show Me” Evolution From Executives

4. ROI is the ultimate measure ROI has been the elusive measure ROI has a rich history of application Operating managers understand and relate to ROI ROI builds excitement among stakeholders ROI is a top executive requirement Why ROI Now? 4

5. The ROI Methodology Captures Six Types of Measures Reaction and Planned Action Learning Application Business Impact Return on Investment Intangible Measures ……..and includes a technique to isolate the effects of the program or solution 5

6. Status of Measurement LevelMeasurement Category Current Status* Goal Comments About Status 0 Inputs/Indicators Measures the number of programs, participants, audience, costs, and efficiencies 100% This is being accomplished now 100% † 1 Reaction and Planned Action Measures reaction to, and satisfaction with, the experience, contents, and value of program 100% Need more focus on content and perceived value 79% † *Percent of programs evaluated at this level † Best Practice benchmarking (user for 5 plus years) >> Add your numbers in each box 6

7. Status of Measurement LevelMeasurement Category Current Status* Goal Comments About Status 2 Learning Measures what participants learned in the program – information, knowledge, skills, and contacts (takes-away from the program) 30 – 40% 80 – 90% Must use simple learning measures 54% † 3 Application Measures progress after the program – the use of information, knowledge, skills, and contacts 10%30% Need more follow-up 31% † *Percent of programs evaluated at this level † Best Practice benchmarking (user for 5 plus years) >> Add your numbers in each box 7

8. Status of Measurement LevelMeasurement Category Current Status* Goal Comments About Status 4 Business Impact Measures changes in business impact variables such as output, quality, time, and cost-linked to the program 5%10% This is the connection to business impact 14.4% † 5 ROI Compares the monetary benefits of the business impact measures to the costs of the program. 1%5% The ultimate level of evaluation 4.3% † *Percent of programs evaluated at this level † Best Practice benchmarking (user for 5 plus years) >> Add your numbers in each box 8

9. 9 Level 0 Input and Indicators  Number of programs  Participants  Hours  Requests  Efficiencies  Costs  Time to Deliver Level 1 Reaction and Planned Action  Relevance*  Importance*  Usefulness  Appeal  Emotion  Brevity  Uniqueness  Concreteness  New Information*  Motivation  Appropriateness  Intent to Use* Five Levels of Measurement - Examples * Correlates with Application

10. 10 Level 2 Learning  Information  Knowledge  Understanding  Capability  Contacts  Confidence  Perceptions  Skills Level 3 Application  Use of Information  Use of Knowledge  Use of Skill  Completion of Actions  Completion of Tasks  Implementation of Ideas  Following the Policy  Use of Procedure  Use of Regulation  Success with Application  Barriers  Enablers Five Levels of Measurement - Examples

11. 11 Level 4 Business Impact  Productivity  Quality  Sales  Errors  Incidents  Re-Work  Efficiency  Compliance Discrepancies  Citizen Complaints  Costs Employee Engagement  Employee Retention  Service Delivery  Cycle Time  Customer Satisfaction Level 5 Return on Investment Intangibles …. includes a technique to isolate the effects of the project.  ROI (%)  Benefit/Cost Ratio  Payback Period Five Levels of Measurement - Examples

12. Develop/ Review Objective of Program Develop Evaluation Plans and Baseline Data Collect Data During Program Implementation Collect Data After Program Implementation Evaluation Planning Data Collection LEVEL 1: REACTION AND PLANNED ACTION LEVEL 2: LEARNING LEVEL 3: APPLICATION LEVEL 4: BUSINESS IMPACT The ROI Methodology 12

13. Convert Data To Monetary Value Data AnalysisReporting LEVEL 5: ROI Intangible Benefits Capture Costs of Program Identify Intangible Measures 13 Isolate the Effects of Program Calculate the Return On Investment Generate Impact Study

14. 14 Net Project Benefits Project Costs ROI = Cost of project $230,000 Benefits of project (1 st year) $430,000 ROI = $430,000 - $230,000 $230,000 = 0.87 x 100 = 87% ROI Calculation

15. 15 LevelMeasurement Focus 1. Reaction and Planned Action Measures participant reaction to the program and captures planned action 2. LearningMeasures changes in knowledge and skills 3. ApplicationMeasures implementation, actions, and changes in behavior on the job 4. Business ImpactMeasures changes in business impact variables 5. Return on InvestmentCompares monetary benefits of the impact of the program. Evaluation Framework

16. 16 1.When a higher level evaluation is conducted, data must be collected at lower levels. 2.When an evaluation is planned for a higher level, the previous level of evaluation does not have to be comprehensive. 3.When collecting and analyzing data, use only the most credible sources. 4.When analyzing data, choose the most conservative among alternatives. 5.At least one method must be used to isolate the effects of the program or project. 6.If no improvement data are available, it is assumed that little or no improvement has occurred. Guiding Principles

17. 17 7.Estimates of improvement should be adjusted for the potential error of the estimate. 8.Extreme data items and unsupported claims should not be used in ROI calculations. 9.Only the first year of benefits should be used in the ROI analysis of short-term projects. 10.Fully loaded all costs of the solution, project of program when analyzing ROI. 11.Intangible measures are defined as measures that are purposely not converted to monetary value. 12.The results from the ROI Methodology must be communicated to all key stakeholders.

18. An evaluation system must have five elements. An Evaluation Framework Applications and Practice A Process Model Operating Standards and Philosophy Implementation 18

19. Criteria for Selecting Programs for Level 4 and Level 5 Evaluation Life cycle of the program Linkage of program to operational goals and issues Importance of program to strategic objectives Executive interest in the evaluation Cost of the program Visibility of the program Size of target audience Investment of time required 19

20. 20 Start Here End Here 5 ROI Objectives 5 4 Impact Objectives 4 3 Application Objectives 3 2 Learning Objectives 2 1 Reaction Objectives 1 Project Business Needs Performance Needs Learning Needs Preference Needs Impact Learning Reaction Initial Analysis Measurement and Evaluation Application Business Alignment and ForecastingThe ROI Process Model Alignment Model Payoff Needs ROI

21. Challenge #1 21

22. Matching Exercise A.Survey B.Test C.Questionnaire D.Interview E.Focus Groups F.Observation G.Performance Records 22

23. Increasing Questionnaire Response Rates Provide advance communication about the questionnaire. Clearly communicate the reason for the questionnaire. Indicate who will see the results of the questionnaire. Show how the data will be integrated with other data. Keep the questionnaire simple and as brief as possible. 23

24. Increasing Questionnaire Response Rates Keep questionnaire responses anonymous – or at least confidential. Make it easy to respond; include a self-addressed, stamped envelope/e-mail. Use the local manager to distribute the questionnaires, show support, and encourage response. If appropriate, let the target audience know that they are part of a carefully selected sample. Use one or two follow-up reminders. 24

25. Increasing Questionnaire Response Rates Have the introduction letter signed by a top executive. Enclose a giveaway item with the questionnaire (pen, money, etc.). Provide an incentive (or chance of incentive) for quick response. Send a summary of results to target audience. Distribute questionnaire to a captive audience. Consider an alternative distribution channel, such as e-mail. 25

26. Increasing Questionnaire Response Rates Have a third party gather and analyze data. Communicate the time limit for submitting responses. Consider paying for the time it takes to complete the questionnaire. Review the questionnaire at the end of the formal session. Carefully select the survey sample. Allow completion of the survey during work hours. 26

27. Increasing Questionnaire Response Rates Add emotional appeal. Design questionnaire to attract attention, with a professional format. Let participants know what actions will be taken with the data. Provide options to respond (paper, email, web-site). Use a local coordinator to help distribute and collect questionnaires. Frame questions so participants can respond appropriately and make the questions relevant. 27

28. Improving Post Program Data Collection Plan early: before program is conducted Communicate expectations Secure commitment to provide Build in the data collection tools Use at least 12 techniques in Share results Use the data 28

29. Challenge #2 29

30. Isolating the Effects of the Program Benchmarking Data Method Control Groups Trend Line Analysis Forecasting Methods Participant Estimates Manager Estimates Sr. Management Estimates Expert Input Customer Input * Survey of Users, N = 235 30

31. Control Group Design Control Group Experimental Group M1 M2 Program 31

32. Use of Control Groups Example Retention Solution at the Federal Information Agency An opportunity to participate in a master’s degree program at agency expense and agency time One hundred high-potential employees chosen for program for a three-year master’s program in information science Experimental group of one hundred were involved in the program; another one hundred in control group were not involved Observed employee turnover performance of both groups during the same time Neither group is aware of the control group arrangement How would you select the control group? 32

33. Use of Trend Line Analysis 33 J F M A M J J A S O N D J MONTHS Fraud Incident Rates 1.85% Pre Program Six-Month Average Projected Average — Using Pre Data as a Base 14.5% 7% Post Program Six-Month Average Fraud Program Conducted

34. Example of Estimation Performance Improvement: Time to Process Visas  =15 minutes Factor that Influenced Improvement Percent of Improvement Caused By Confidence Expressed as a Percent Adjusted Percent of Improvement Caused By HR Project60%80%48% Technology Change15%70%10.5% Procedure Change25%60%15% Other___% Total100% 34

35. Using Estimates to Isolate the Effects Describe the task and the process. Explain why the information was needed and how it will be used. Ask participants to identify any other factors that may have contributed to the increase. Have participants discuss the linkage between each factor and the specific output measure. 35

36. Using Estimates to Isolate the Effects Provide participants with any additional information needed to estimate the contribution of each factor. Obtain the actual estimate of the contribution of each factor. The total must be 100%. Obtain the confidence level from each employee for the estimate for each factor (100%=certainty; 0%=no confidence). 36

37. The Power of Estimates Research Comparison with other methods Handling objections Management reactions Participant reaction Use as a last resort 37

38. Matching Exercise A.Control Group B.Trend Line Analysis C.Forecasting D.Participant’s estimate E.Use of Customer Input F.Expert’s Estimate 38

39. Challenge #3 39

40. Matching Exercise A.Profit/savings from output (standard value) B.Cost of quality (standard value) C.Employee time as compensation (standard value) D.Historical costs/savings from records E.Expert input F.External database G.Linking with other measures H.Participant estimation I.Management estimation J.Estimation from HR staff 40

41. Typical Intangible Benefits Adaptability Awards Brand awareness Career minded Caring Collaboration Communication Commitment Conflicts Cooperation Corporate social responsibility Creativity Culture Customer complaints Customer response time Customer satisfaction Decisiveness 41

42. Typical Intangible Benefits Employee complaints Engagement Execution Image Innovation Job satisfaction Leadership Networking Organizational climate Organizational commitment Partnering Reputation Resilience Stress Talent Teamwork 42

43. ROI Is Reported In Two Ways Benefits-Cost Ratio (BCR) ROI (%) Program Benefits Program Costs Net Program Benefits Program Costs = = X 100 43

44. ROI Is Reported In Two Ways $500,000 $400,000 $500,000 - $400,000 $400,000 BCR== 1.25 ROI=x 100 =25% 44

45. Setting the ROI Objective What’s a great ROI? It depends on the objectives Set the value as with other investments – e.g.15% Set the value slightly above other investments – e.g. 25% Set at breakeven – e.g.0% Set at client expectations –e.g. 30% 45

46. ROI Red Flags Not using a balanced profile of data Failure to isolate the effects of the program Overstating the monetary benefits stream Leaving out some costs Not being conservative Not being consistent 46

47. ROI Methodology : The Payoff Align projects to business needs Show contributions of selected projects Earn respect of senior management/administrators Build staff morale Justify/defend budgets Improve support for projects 47

48. ROI Methodology : The Payoff Enhance design and implementation processes Identify inefficient projects that need to be redesigned or eliminated Identify successful projects that can be implemented in other areas Earn a “seat at the table” 48

49. 49 Questions? Web: www.roiinstitute.net Email: jack@roiinstitute.netjack@roiinstitute.net Connect with us on: