1. www.3ieimpact.org Philip Davies How is a Policy Supposed to Work? – Theory of Change Analysis Philip Davies International Initiative for Impact Evaluation [3ie] BCURE Evidence-Informed Decision-Making Capacity Building Workshop 1st and 2nd June 2015 Pretoria, South Africa

2. www.3ieimpact.org Philip Davies Evidence is information or data that supports, or rejects, a conclusion Evidence is any thing that increases the estimate of the probability of the truthfulness of a proposition Evidence is particularistic and generalisable Often disagreement agreement on what counts as ‘evidence’ Evidence is always contestable/contested Not all research is of equal value/sufficient quality Single studies can misrepresent the balance of evidence Hence, the need for systematic reviews/synthesis of evidence Some Features of Evidence

3. www.3ieimpact.org Philip Davies Quantitative Evidence Social Surveys – cross sectional and time series Longitudinal Studies – cohort and panel data Experimental Designs (random allocation) Quasi-Experimental Designs (Matched samples, Interrupted Time Series, Regression analysis) Economic Methods - Cost-Benefit and Cost- Effectiveness Analysis Statistical / Econometric Modelling

4. www.3ieimpact.org Philip Davies Qualitative Evidence In-Depth interviews Focus Groups Other Consultative Designs Observational and Participant-Observational Studies Ethnography Documentary Analysis Oral Histories Case Studies

5. www.3ieimpact.org Philip Davies Theory of Change/Logic Model/Programme Theory How is a policy/programme supposed to work? What activities, mechanisms, people, outputs have to be in place? And in what sequence – what is the causal chain? What resources are required – and are available? What data are required – and are available? Is the policy/programme feasible/achievable?

6. www.3ieimpact.org Philip Davies Building a Theory of Change: From Inputs to Outcomes

7. www.3ieimpact.org Philip Davies Constituent Features of a Theory of Change Assumptions?

8. www.3ieimpact.org Philip Davies Constituent Features of a Theory of Change Data Required Surveys, statistics, demographic data Qualitative data Costs/benefits data Systematic review data Documentary analysis Surveys, statistics, demographic data Qualitative data Costs/benefits data Systematic review data Documentary analysis Performance data Historical data Diversity data Qualitative data Effectiveness data Performance data Historical data Diversity data Qualitative data Effectiveness data Stakeholder data Qualitative data Public opinion data Effectiveness data Stakeholder data Qualitative data Public opinion data Effectiveness data Performance data Effectiveness data Stakeholder data Qualitative data Costs/benefits data Performance data Effectiveness data Stakeholder data Qualitative data Costs/benefits data Administrative data Performance data Costs/benefits data Administrative data Performance data Costs/benefits data Administrative data Performance data Qualitative data Administrative data Performance data Qualitative data Counterfactual data Administrative data Survey data, statistics Cost/benefit data Counterfactual data Administrative data Survey data, statistics Cost/benefit data

9. www.3ieimpact.org Philip Davies

10. www.3ieimpact.org Philip Davies Different Levels of Intervention

11. www.3ieimpact.org Philip Davies Cognition Lifestyle Skills and Behaviours Biological Individual Level Demographi c Coping Ability Obesity/H ealth Eating Family Role Modeling Social Support Friends Peers Social Norms Obesity/H ealth Eating Social Environment Home Access Availability Bars/Pubs Supermarkets Opportunities Obesity/H ealth Eating Physical Environment Government Legislation Tax and Price Industry Health Systems Licensing Advertising (Availability) Obesity/H ealth Eating Macro Level Environment (Policy)

12. www.3ieimpact.org Philip Davies Establishing the Policy Logic/Theory of Change Basic Principles Map out the causal chain Test the underlying assumptions with evidence Understand context and culture Anticipate heterogeneity (variance) across the population Rigorous evaluation of impact using an appropriate counterfactual Use mixed methods of evidence gathering

13. www.3ieimpact.org Philip Davies Group Exercise In groups, take a policy issue of your choice and: Map out how the policy is supposed to work, identifying: i.What are the outputs and outcomes to be achieved? ii.What people/agencies are likely to be involved? iii.What mechanisms (training, planning, zoning, financing, communications, delivery etc) are likely to be involved? iv.What time will be required to achieve the above factors? v.What are the likely costs involved, and where should they fall?

14. www.3ieimpact.org Philip Davies Thank you Philip Davies on Email: pdavies@3ieimpact.orgpdavies@3ieimpact.org +44 (0)207 958 8350 Visit www.3ieimpact.org