QUICK OVERVIEW - REDUCING REOFFENDING EVALUATION PACK INTRODUCTION TO A 4 STEP EVALUATION APPROACH

WHAT IS THE EVALUATION RESOURCE PACK? The evaluation resource pack is a comprehensive and user-friendly slide-show which provides step by step evaluation guidance and resources. It was produced by Justice Analytical Services at the Scottish Government to help evaluators and funders conduct better quality evaluations to assess the value of their interventions. The main purpose of the full version evaluation resource pack is to, Emphasise the importance of using the evidence-base to design interventions. Promote a rigorous 4 STEP approach to evaluation which interventions of ALL SIZES and at ALL STAGES OF DEVELOPMENT can conduct. Help interventions carry out a realistic and rigorous alternative to impact evaluations which are very difficult to do in Scotland Describe how to structure an evaluation report to increase consistency and quality in report writing To provide advice to funders on how to judge the merit of interventions in Scotland.

WHO IS THE EVALUATION RESOURCE PACK FOR? The evaluation resource pack is designed to support anyone commissioning or evaluating criminal justice interventions. In particular it should help, Contractors and practitioners to Evaluate criminal justice interventions using a robust 4 step method Structure an evaluation report Funders to Commission evaluations Judge the strength of evaluation reports Assess the value of interventions

WHY DO WE NEED AN EVALUATION RESOURCE PACK? THE BACKGROUND TO THE 4-STEP METHOD Assessing the impact of funded interventions in Scotland is difficult due to methodological constraints but the pressure to show that interventions are effective has led to some poor and at times dubious evaluations. To see whether your intervention had an impact and made a real difference to users, you need a to compare users with a randomised or matched control group have the same risk of reoffending as the users. You also need large sample sizes and use statistical testing on the outcomes. Very few evaluations are able to use this method so we devised an alternative which mitigates the lack of impact information by emphasising the need to embed robust evaluations from elsewhere and evaluating the extent to which an intervention is evidence-based as part of the evaluation itself. The evaluation then collects data to test whether the intervention was implemented as intended and whether short and medium term outcomes materialised. This pack was devised to support evaluators conduct this type of evaluation which can be described in 4 steps:

THE 4 STEP APPROACH TO EVALUATION Then analyse the data (and collect more if necessary) to find out the extent to which your intervention was evidence-based and if it worked as the logic model predicted it would. Put as much emphasis on describing and evaluating inputs as well as outputs and outcomes Use this logic model to identify indicators for inputs, outputs and outcomes and collect data using relevent methods Draw a logic model describing how your intervention works in practice by describing the links between inputs, outputs and outcomes. The logic model forms the basis for evaluating the whole intervention so this may provide better clues as to why an intervention acheived its outcomes or why it did not. Interventions should be clearly structured and designed using robust evidence so it is important to be familiar with the results from the ‘what works’ and desistance literature. This knowledge should be used to evaluate the extent to which the intervention is grounded in strong and consistent evidence. You could also be explcit about how much it cost and how the funds were spent. Review the evidence Draw a logic model Identify Indicators and collect monitoring data Evaluate logic model

CONTENTS OF THE FULL VERSION The purpose of the pack…………………………………………………………………….. 2 Background: Why a 4 step approach? The challenges of measuring impact in Scotland Impact evaluations – why are they so hard to do? Control and comparison groups – which characteristics need to be similar? So if measuring impact is tricky, what can we do? The 4 step approach to evaluation 5 The 4 step approach…………………………………………………………………………. 4 step approach to evaluation -overview 11 Step 1: Review the evidence What does the evidence say? Evidence summaries An example – an evidence-based justification for a fictitious intervention 13 Step 2: Draw a logic model showing how the service or intervention works……… What are logic models What logic models can do A very simple evidence-based model A logic model template to use Logic model column content– a quick guide The importance of designing a structured intervention An evidence-based logic model – reducing reoffending An example - the Reducing Reoffending evidence model An example -A simple supervised bail logic model 30 Step 3: Identifying indicators and collecting monitoring data……………………….. Use the Logic model of identify indicators Use the Logic model to set evaluation questions and guide the collection of data Example: Indicators for outputs/activities and outcomes Data collection Quantitative and Qualitative data –uses, benefits and limitations. Data capture and analysis An example data collection framework for a criminal justice intervention 40 6

CONTENTS CONTINUED... Step 4: Evaluate logic model………………………………………………………………….. Test the logic model Measuring and reporting outcomes Measuring and reporting impact Caveats to measuring impact Subjective measures of impact A note on Cost Benefit analysis 54 Evaluation Report Structure Structure and content 61 Judging the worth of an intervention…………………………………………………………. Assessing an evaluation report Example judging criteria matrix for a reducing reoffending intervention Features and advantages of a scoring system 65 Advantages and disadvantages of the 4 step logic model approach to evaluation Helpful Resources………………………………………………………………………………….. 70 /71 7