Return on Investment: Training and Development ROI and Evaluation 1

ROI Contents Purpose of ROI, and Value of training What is ROI Why ROI Methodology Toolkit

Models Benefit/Cost Ratio ROI (%) 3 3 Benefit/Cost Ratio: This is a basic definition of ROI. This is a quantification of the relation between the benefits of a program and its costs. When BCR is greater than one, the benefits outweigh the costs and the program is considered a success. When BCR is less than one, the cost exceeds the benefits and demonstrates that improvements or changes probably need to be made to justify continuation of the program. Another useful and often used definition/formula expresses the ROI as the percentage return on the costs incurred. This has the advantage of speaking to many investors and stakeholders in their language. A result greater than 100% means that the program has a net benefit after accounting for the costs involved in running it. For instance, an ROI% = 150% means that the program yields a 150% return on money invested; i.e., the program yields $1.50 for every dollar that the program costs. A result less than 100% means the program has a net cost. This means that the program does not recoup its cost after accounting for the benefit. When this happens, there may be a “hidden” or social benefit that is not quantifiable, such as an increase in employee morale. In these cases, stakeholders and decision makers need to ascertain whether the scale of loss is justifiable given the money spent. A loss of 3 percent of several thousand dollars may be worth it to realize a happier workplace, but 3 percent of several million may not; there may be easier or less expensive ways to create a happier workplace. This is where ROI really becomes useful. Used properly it can be an objective method to compare the benefits, costs and returns for two or more programs. Note – See Figure 2-2 on p. 37 and throughout the Phillips text for the model. This is a widely adopted and complete model for the process of calculating ROI for training and development. 3 3

The Value of Human Capital We all know Human Capital (and its development) is vital Human Capital is often much undervalued as an investment opportunity ROI can be very high Phillips refers to 800% being achievable regularly Tennessee Valley authority claim 1000%+ A world-class corporate university claimed 5,612%

Why ROI? Pressure on training functions to show demonstrable return Value for money from training activity Value of HRD to the business; e.g. Financial benefits Staff productivity Staff retention Clear alignment with corporate goals In-house vs outsourced training ROI analysis is a tool to support the contention that training is valuable; demonstrate benefits vs cost Pressure to reduce costs must be balanced by visibility of benefit “Warm and fuzzy” is no longer any good! (in most cases) Jay Cross alternative view

ROI as Process Use evaluation techniques to acquire performance data Use Training Needs Analysis to direct the project to the right data Use analytical techniques to calculate ROI from the data Use ROI results to review training effectiveness Use the reviews to inform future decisions

ROI as Perception Which of these perceptions indicate your client’s view of ROI (which you need to address)? The Board may see a big picture of how the training affects the company’s ability to achieve its corporate goals The finance department may be looking to see how training stacks up financially against other ways to invest the company’s money, and whether the training, as carried out, was financially more effective than alternative forms of development The business unit manager may be solely concerned with the impact on performance and productivity in achieving the goals of their department The training and development manager may be concerned with the impact training programmes are having on the credibility and status as the training function within the company and its ability to secure investment in the future to drive further business performance enhancements

What is ROI? Elements of “Total ROI” Use of ROI Role of ROI Reducing costs Increasing benefits/performance/capability Use of ROI ROI of a training project’s contribution to business or project ROI of one delivery method vs. another (e.g. ILT vs. e-learning) Role of ROI Pre-training assessment and expectation setting Post-training feedback Post-experience value measurement

The Process Model 1: Measure reaction &identify planned actions Immediate reaction and response 2: Cognitive learning and retention Skills, knowledge, attitude changes 3: Assessing application of learning to the job Actual change in behaviour 4: Identifying business results Quantify business value of change 5: Calculating ROI Compare monetary value with costs

Getting Started Select your project to measure ROI measurement Plan Select a significant project; align with significant goals Start with a project that has clearly definable metrics ROI measurement Plan Data Collection Plan Identify the “returns” Identify the investment factors Select the survey audiences and sources of data Select your data collection methods Analysis Plan

Investment Factors Costs incurred Payments to suppliers and service providers Time and attention to create and deliver the training Opportunity cost Time and costs involved in the ROI measurement exercise Other Internal costs

Process Summary – Data Collection Plan Data collected at different times to provide: Pre-training baseline Post-training change analysis Post-experience change analysis Data Collection Plan State the objectives of the training / learning State the objectives of each phase of data collection at each evaluation Level Identify any previously used metrics, values or methodologies used by the client, and determine suitability for the current exercise Select the appropriate evaluation methods Identify the audiences who will be surveyed for data Set the timing for the data collection Allocate responsibilities for data collection and analysis

Process Summary – ROI Analysis Plan Continuation of the Data Collection Plan, capturing information on the key items needed to develop the actual ROI calculation. List Significant Data items (usually Level 4 or 3) to be collected Benefit Factors Cost Factors Methods to isolate effects of the learning/training from other influences Methods to convert data to numerical values Intangible benefits Other influences Communication targets

Process Summary – Data Collection Identify the purposes of the evaluation. State clearly what the evaluations are to measure and what the goals of the training are intended to be. Be as specific as possible about the goals of the training Ensure goals address the performance enhancement, business improvement or cost savings expectations. Select the evaluation instruments and methodology. Establish the timing for the data collection. Decide whether pre-training analysis is required, or post training analysis, or both. (e.g. pre-training and multiple post-training assessments may be necessary to effectively identify the skills changes in Levels 2, 3 and 4.) Carry out the data collection at the levels 1-4

Process Summary – Sources of Data Organisational Performance Records, showing outputs and measurements taken as part of the business’ normal reporting process Testing and certification assessment records Participant feedback Instructor feedback Feedback from participants’ supervisors/managers Feedback from participants’ subordinates Team/group peer feedback Feedback from other internal or external groups (eg HR training departments)

Process Summary – Evaluation Methods Identify how the data will be collected and analysed Surveys On the job Observation Interviews Focus groups Action plans (or Performance contracts) and Program assignments Performance data monitoring

Process Summary – Isolate effects of training Separate training (personal productivity) component of performance change, from new software/systems/processes Essential for credibility Potential methods Use control groups Impact assessments and estimates by participants, managers, peers Trend lines Discount/adjust for over-estimates Apply “inflation adjustment” for estimates and assessments

Process Summary – Convert to money Convert data to monetary value Specific costs and time incurred Costs and time saved Quality increase, reduced waste Improved customer service and satisfaction “Intangible” benefits (retention, commitment, fewer complaints, reduced conflicts etc)

Process Summary - ROI Calculate ROI Benefit-cost ratio= Program Benefits Program Costs ROI % = Net Program Benefits Program Costs x 100 Break-even time = Investment Benefit x Period in months

Evaluation Levels 1. Reaction and Planned Action 2. Learning 3. Application and Implementation 4. Business Impact 5. Return on Investment Kirkpatrick’s Four Levels of Evaluation; Phillips added ROI as the fifth.   Reaction and Planned Action – Frequency: each learning event. Was the customer (trainee) satisfied with the experience? What did they like? What did they learn? Was there anything missing? Consider using a Likert rating scale for feedback. Was the facilitator knowledgeable? Was the subject interesting? Were facilities adequate for the training? Was the training scheduled at a good time? Additional comments? Learning – Frequency: pre- and post-training. Was there a change in attitude, skills and/or knowledge? Assess learning before training, during training and after training to accurately assess learning. Application and Implementation – Frequency: pre- and post-training and particular periods after training is complete (e.g., three months, six months, one year). This speaks to behavior. Are the trainees doing things differently at work by using the trained knowledge and skills? Pre-/post-test, observation, interview. Allow time for the change to occur. Ask employee, supervisor, subordinates for their perception of change in attitude or performance. Business Impact – Frequency: regular intervals over the calendar or fiscal year – monthly or quarterly is typical. This speaks to overall change for the business as a result of the training program—the results. Is there improved quality, improved production, decreased costs, increased job satisfaction, reduced problems or accidents, increased sales? ROI – Frequency: With each new training event or when significant changes are made to existing events. Consider the costs of training versus the benefits of training. How did the bottom line change? Were the benefits greater than the cost? 20 20

Writing Objectives Easy as A, B, C, D Audience: Who? Behavior: What do “they” do? Condition: What is the setting and method of evaluation? Degree: Measurement to be met. Objectives are very specific measures for an organization or an individual. In a training environment, we need to keep the organizational goals and the learning goals in mind as we write objectives for any single training session. Objectives guide the learning. ABCD of writing objectives, using observable verbs: Audience The group of learners the objective is written for; “the learner…” or “the student…” Behavior The action or observable verb which describes what the learner will be able to do after completing the instruction (e.g., describe, compare/contrast, demonstrate). For a more complete list of verbs, please see the verb list in the readings. Condition The tools, resources, setting the students will have and the assessment method to be used. Degree The standard or degree of accuracy to be considered proficient. This can be based on a normative scale, measured against a standard of performance. 21 21

Example Objectives Course objective: Learners will be able to make 15 entries in a customer database in 15 minutes with no more than 1 error. Application objective: Learners will be able to reduce the data entry error rate by 50 percent over the next 6 months. Impact objective: Employee time spent correcting database errors is reduced by 25 percent from last year’s rate. Increasing Scope 22 22

Costs, Budgets, Accounting Quantifying ROI means accounting for all the costs of the program. Fixed costs: independent of the number of participants. Variable costs: Dependent on the number of participants. There are costs at every step – make sure to account for them all. Costs, Budgets, Accounting (PowerPoint slides) - Fixed, variable, hidden, calculated, measured Input measures – see budget sheet – give examples. Fixed Costs vs. Variable Costs Fixed costs are those that remain the same no matter how many individuals participate in the training. Examples include marketing and information distribution about the training, trainer’s time, employees’ time away from job for training, Variable cost are those costs that change based on number of participants. Examples include training manuals and materials, meals (if provided), difference in cost between large room for 50 participants and small room for 15 (if applicable). 23 23

Benefits and Soft Skills Change in: Attitude, work climate, leadership, teamwork. We desire these changes because they ultimately effect productivity. Allow time for change in attitude or behavior, then measure these changes and report qualitatively. Allow time for change in productivity, then measure for data and report quantitatively. This is the soft data. Typically, soft skills can be tied directly to hard data. For example, a better work climate reduces the amount of sick leave and turnover, and increases productivity; better leadership enhances new ideas, new revenue and output. 24

Create a Data Collection Plan What? New information that needs to be recorded? Who? When? How? Ask students to brainstorm: “What data do you need? What new information needs to be recorded? Who will record it? When will it be recorded? How will it be collected?” Return to the course objectives, application objectives and impact objectives to ensure you know what information you will need to measure the effect of training. Some of it may not be readily available and may require planning or coordination with other people to collect. From whom will you collect the feedback/information? Participants; Participants’ managers; Participants’ co-workers; Participants’ supervisees; Clients/Customers; Other When will you collect the information? During training; Right after training; After time has passed How do you collect the information? Surveys/Questionnaires; Interviews; Test performance (norm or criterion references) ; Simulations; Observations 25

Trend Line Example

Stakeholders: Review Customer Employee Supervisor Subordinate President Board of Directors Stockholders Review Then, review who your stakeholders are – who needs or wants the information you have collected. 27

Thank You.. Any Questions: Dr. Pratik Surana +91-9890653263 pratiks@qicpl.com