1. Simon Hearn, ODI, London Part 2 of 8 AEA Coffee Break Webinars 2013 DEFINE what is to be evaluated

2. The Rainbow Framework

3. Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr

4. Three evaluation tasks in DEFINING Develop initial description Develop program theory or logic model Identify potential unintended results

5. Develop initial description 1.

6. Thumbnail description Vision & Mission Peak Experience Aims & Objectives

7. Develop program theory or logic model 2.

8. Options for representing logic models Pipeline / results chainLogical frameworkOutcomes hierarchy / theory of changeRealist Matrix

9. Options for representing logic models Pipeline / results chainLogical frameworkOutcomes hierarchy / theory of changeRealist Matrix

10. INPUTS Apples People at risk of poor health IMMEDIATE RESULTS Apples eaten SHORT-TERM RESULTS Improved nutritional status LONGER-TERM RESULTS Improved health

11. Options for representing logic models Pipeline / results chainLogical frameworkOutcomes hierarchy / theory of changeRealist Matrix

12. RESULTS AREAINDICATOR MEANS OF VERIFICATION ASSUMPTIONS/ RISKS GOAL: Improved health Decreased sick days School and work records Good status largely due to nutrition PURPOSE: Improved nutritional status Adequate levels of Vitamin C Sample blood tests Apples retain Vitamin C OUTPUT: Apples eaten Apples replaced by apple cores Visual inspection Apples eaten on location ACTIVITIES: Deliver apples Required numbers delivered Delivery receipts ACTIVITIES: Buy apples Required numbers bought Purchase receipts Visual inspection

13. Options for representing logic models Pipeline / results chainLogical frameworkOutcomes hierarchy / theory of changeRealist Matrix

14. Outcomes chain showing possible alternative causal paths Improved health Improved vitamin C levels Apples eaten (whole or juice) Apples delivered to schools Improved quercetin levels Red apple skin eaten Decreased Body Mass Index Apples eaten instead of junk food snacks

15. Options for representing logic models Pipeline / results chainLogical frameworkOutcomes hierarchy / theory of changeRealist Matrix

16. ContextMechanismOutcome Healthy people at high risk of gastric cancer Quercetin from red apple skin (or red onion) protect cells from free radicals Reduced risk of cancer Smokers or people with regular exposure to radiation Chemicals in cigarette smoke or radiation damages DNA in healthy cells Continued increased risk of cancer Individuals with vitamin deficiency through malnutrition Sufficient vitamin C absorbed when juice from apples (or oranges) consumed Stronger immune system Individuals with vitamin deficiency through excessive alcohol Alcohol destroys vitamin CNo change in vitamin C level Individuals with obesity due to excessive snacking on high fat, high calorie foods Decreased calorie intake due to substitution of apples (or carrot sticks) Reduced obesity and related conditions Individuals with obesity due to genetic condition Decrease sugar intake through substitution of apples No change in obesity but lower risk of diabetes

17. ContextMechanismOutcome Healthy people at high risk of gastric cancer Quercetin from red apple skin (or red onion) protect cells from free radicals Reduced risk of cancer Smokers or people with regular exposure to radiation Chemicals in cigarette smoke or radiation damages DNA in healthy cells Continued increased risk of cancer Individuals with vitamin deficiency through malnutrition Sufficient vitamin C absorbed when juice from apples (or oranges) consumed Stronger immune system Individuals with vitamin deficiency through excessive alcohol Alcohol destroys vitamin CNo change in vitamin C level Individuals with obesity due to excessive snacking on high fat, high calorie foods Decreased calorie intake due to substitution of apples (or carrot sticks) Reduced obesity and related conditions Individuals with obesity due to genetic condition Decrease sugar intake through substitution of apples No change in obesity but lower risk of diabetes

19. http://betterevaluation.org/resource/example/rubrics-oakden

20. Identify potential unintended or negative results 3. Source: www.smarttoolkit.net

21. Negative program theory Risk assessment Key informant interviews Six Hats Thinking

22. Defining – Framing iteration Initial description Discuss evaluation purpose Refine description Define evaluation questions Develop logic model Agree scope of evaluation DEFINE FRAME

23. Simon Hearn Research Fellow, ODI s.hearn@odi.org.uk http://betterevaluation.org/plan/define facebook.com/betterevaluation @bettereval