Return On Investment Integrated Monitoring and Evaluation Framework

Integrating a Framework  Many organisations follow a disjointed staged approach to their monitoring and evaluation following a framework which is responsive rather than proactive. Contract Reporting Internal progress updates Retrospective evaluations Reporting Business Plan Short term response to need Contractual obligation Delivery Monitoring Engagement with services Periodic activity

Integrating a Framework  Introducing an ROI evaluation methodology allows for an integrated framework ensuring better service planning and monitoring together with an evaluation of impact Business Plan Strategic Justification Programme outcomes Staged KPI’s Contractually aligned Delivery Monitoring Engagement with Services Progress towards KPI’s and Outcomes Impact Formative review of delivery Phillips Return on Investment Methodology TM Ongoing Evaluation Contract Reporting Internal progress updates Retrospective evaluations Reporting

Why use ROI methodology? CREDIBILITY More than 2,000 organisations have implemented the ROI Methodology More than 3,000 professionals have attended ROI Certification Workshops More than 15,000 professionals have participated in two-day ROI workshops International and Local ROI Networks 20 books translated into 25 languages Taken from Site content © ABDI ltd 2008

Why use ROI methodology? BOTTOM LINE IMPROVEMENT but can my intervention take the credit? A vital question, with a three-part answer:  Not causality but evidence of a chain of impact.  Impact results are adjusted for the impact of other influences.  Standardisation through twelve guiding principles supports credibility. Taken from Site content © ABDI ltd 2008

Why use ROI methodology? The ROI methodology links the outcome to the intervention through a chain of impact at up to five levels : 1. Participant reaction and planned action. 2. Participant learning. 3. Related personal or operational changes. 4. Related improvements in own environment or business results. 5. Hence return on investment - the net financial gain set against the total cost. Taken from Site content © ABDI ltd 2008

REACTIONLEARNINGAPPLICATION BUSINESS IMPACT ROI Impact Mapping Feedback Forms Surveys Pre/post event test Simulations Self perception Observations Action plans Focus groups Follow ups Self perception Access to performance info Access to financial info Identify costs Identify intangibles Identify influences Relevance of project / event Importance of project / event Recommend to other Learning outcomes achieved Relevant skills gained Maintaining of skills Application of skills Completion of action plans New practices Performance improvements Reduced costs Increased efficiency Isolate effects Control groups Trend analysis Participation estimate Measure intangibles Benefit to Cost ratio Calculate ROI Report analysis to the project deliverers to inform any repeat events or to highlight potential issues Ongoing reporting / feedback to project developers, participants Final Evaluation Report Executive summary Presentation Communication to stakeholders PLANNING ANALYSIS REPORTING THE MODEL Model re-representation by permission © ABDI ltd 2008

Integrating the Model  Business planning  Justify the programme in terms of an opportunity or a need  Specify the long term desired outcomes for the programme  Break the outcome into staged KPI’s  Identify business need for each stage and align with ROI model Business NeedROI Model Return on Investment A case load (cost) reduction for high dependency drug users Business Improvement Increase in self referral to alternative support services Application of skills / personal changes Provide service users with knowledge of community support service through online signposting service Participant Learning Service users to access online signposting and rate relevance Participant Reaction

Integrating the Model  Recording delivery progress  Establish methodology for recording progress to KPI’s / Outputs which includes:  Activity  Risks / Issues  Mitigating measures  Recording delivery Impact  Establish a monitoring environment which utilises existing systems such as client databases, CRM’s etc  Enhance monitoring procedures to ensure the timely and appropriate collection of core data  Develop tools which are standard in style, yet bespoke in content to ensure relevance to the delivery programme

Integrating Model  ROI toolkit  To support the different stages of the ROI model a toolkit should be developed which will include standardised documentation:  Objectives and Target Planning (evaluation plan pt.1)  Data collection Plan (evaluation plan pt.2)  ROI Analysis Plan (evaluation plan pt.3)  Include frequently used tools for measuring the learning and application of skills and business changes. These should be integrated with delivery  Feedback forms / simulations / tests  Action plans / interviews / focus groups  Identify early, and develop methods for full cost calculations

Integrating Model  Reporting Standards Frequent reporting to all stakeholders is key. This should include:  To management / board / funders  Periodic updates on delivery  Headline impact such as engagement and business improvement  To programme deliverers and participants  Success of achieving learning outcomes immediately after delivery  Ongoing reports outlining application and associated changes, leading to…  Full ROI impact assessment

And finally, the12 Guiding Principles 1. When a higher level evaluation is conducted, data must be collected at lower levels 2. When an evaluation is planned at a higher level, the previous level of evaluation does not have to be comprehensive 3. When collecting and analysing data use only the most credible sources 4. When analysing data choose the most conservative alternative for calculations 5. At least one method must be used to isolate the effects of the solution 6. If no improvement data is available, assume no improvement 7. Estimates of improvements should be adjusted for the potential of error 8. Extreme data items or unsupported claims should not be used in ROI calculations 9. Only the first year of benefits should be used in the ROI analysis 10. Cost of the solution should be fully loaded 11. Intangible measures are defined as measures that are purposely not converted to money 12. The results of the ROI Methodology must be communicated to all key stakeholders