1. www.3ieimpact.org Philip Davies Making Evidence Accessible and Relevant for Policy and Practice Philip Davies International Initiative for Impact Evaluation [3ie] Africa Evidence Network Regional Meeting Johannesburg, South Africa, 3 rd June 2015

2. www.3ieimpact.org Philip Davies Helping people make better decisions and achieve better outcomes, by using the best available evidence from research and other sources Knowing what are effective interventions (“what works?”) In achieving which outcomes? For which groups of people? Under what conditions? Over what time span? At what costs?, plus Integrating research with decision makers’ knowledge, skills, experience, expertise and judgement What is Evidence-Based Policy

3. www.3ieimpact.org Philip Davies Evidence Experience & Expertise Judgement Resources Values, Beliefs and Ideology Habits & Bureaucratic Culture Lobbyists & Pressure Groups Pragmatics & Contingencies Factors Other Than Evidence

4. www.3ieimpact.org Philip Davies Understanding the Problem (Conceptualisation) Developing Solutions (Policy Development) Putting Solutions Into Effect (Implementation) Monitoring and Evaluation (M&E) The ‘Classic’ Policy Cycle The ‘ROAMEF’ Policy Cycle Evidence is required across the entire policy cycle

5. www.3ieimpact.org Philip Davies The Kind Of Evidence Decision-Makers Look For Identifying the nature, size and dynamics of the problem Specifying the desired objectives Identifying viable policy options Identifying how the policy is supposed to work Identifying the likely and achieved outcomes/impacts Identifying the social distribution of outcomes/impacts Understanding people’s attitudes, experiences, behaviour Valuing the impacts (cost-benefit/cost-effectiveness) Identifying effective implementation and delivery

6. www.3ieimpact.org Philip Davies Types of Evidence for Policy Making

7. www.3ieimpact.org Philip Davies Theory of Change/Logic Model/Programme Theory How is a policy/programme supposed to work? What activities, mechanisms, people, outputs have to be in place? And in what sequence – what is the causal chain? What resources are required – and are available? What data are required – and are available? Is the policy/programme feasible/achievable?

8. www.3ieimpact.org Philip Davies Constituent Features of a Theory of Change Assumptions?

9. www.3ieimpact.org Philip Davies Constituent Features of a Theory of Change Data Required Surveys, statistics, demographic data Qualitative data Costs/benefits data Systematic review data Documentary analysis Surveys, statistics, demographic data Qualitative data Costs/benefits data Systematic review data Documentary analysis Performance data Historical data Diversity data Qualitative data Effectiveness data Performance data Historical data Diversity data Qualitative data Effectiveness data Stakeholder data Qualitative data Public opinion data Effectiveness data Stakeholder data Qualitative data Public opinion data Effectiveness data Performance data Effectiveness data Stakeholder data Qualitative data Costs/benefits data Performance data Effectiveness data Stakeholder data Qualitative data Costs/benefits data Administrative data Performance data Costs/benefits data Administrative data Performance data Costs/benefits data Administrative data Performance data Qualitative data Administrative data Performance data Qualitative data Counterfactual data Administrative data Survey data, statistics Cost/benefit data Counterfactual data Administrative data Survey data, statistics Cost/benefit data

10. www.3ieimpact.org Philip Davies Types of Systematic Review Statistical Meta-Analyses Narrative Systematic Reviews Qualitative Systematic Reviews Rapid Evidence Assessments Evidence Maps and Gap Maps

11. www.3ieimpact.org Philip Davies  Single studies can:  Misrepresent the balance of research evidence  Illuminate only one part of a policy issue  Be sample-specific, time-specific, context- specific  Often be of poor quality Why Do We Need Systematic Reviews?  Consequently, give a biased view of the overall evidence

12. www.3ieimpact.org Philip Davies Systematic searching for studies Systematic critical appraisal of identified studies – separating the wheat from the chaff Systematic and transparent inclusion/exclusion of studies for final review Systematic and transparent extraction of data Systematic statistical testing and analysis Systematic reporting of findings What Makes a Review Systematic?

13. www.3ieimpact.org Philip Davies Statistical Meta-Analytical Reviews Source: David B. Wilson, 2006, A systematic review of drug court effects on recidivism

14. www.3ieimpact.org Philip Davies Synthesise qualitative and ethnographic evidence In-depth interviews, focus groups, observational studies, documentary analysis, case studies Seek common themes, concepts and principles across different studies Detailed attention to context/contextual specificity And stakeholders’ views Do not seek generalisations Qualitative Systematic Reviews Photo © Albert Gonzalez Farran - UNAMID

15. www.3ieimpact.org Philip Davies Scaled down systematic reviews of existing evidence Timed to meet the needs of policy makers/practitioners (1-3 months) Strategically using the ‘three arms’ of systematic searching, but less exhaustively Rapid Evidence Assessments – What Are They? Critical appraisal of identified studies is included Summary of findings, with caveats and qualifications Photo © Panos East Africa

16. www.3ieimpact.org Philip Davies Rapid Evidence Assessments – How Scaled Down?

17. www.3ieimpact.org Philip Davies 3ie Evidence Gap Maps Maps of the existing evidence base on a policy issue, topic or sector such as maternal health, HIV/AIDS, agriculture, extreme poverty Structured around a framework of interventions and outcomes (intermediate and final) A ways of identifying where there is evidence, and where there is not An indication of the quality of this evidence Links to user-friendly summaries in the 3ie database of systematic reviews.

18. www.3ieimpact.org Philip Davies Rapid Evidence Assessments - Limitations Evidence Gap Maps

19. www.3ieimpact.org Philip Davies

20. www.3ieimpact.org Philip Davies

21. www.3ieimpact.org Philip Davies Some Key Sources of Sythesised Evidence 3ie Impact Evaluations Database (http://www.3ieimpact.org/en/evidence/impact-evaluations/)http://www.3ieimpact.org/en/evidence/impact-evaluations/ 3ie Systematic Reviews Database (http://www.3ieimpact.org/en/evidence/systematic-reviews/)http://www.3ieimpact.org/en/evidence/systematic-reviews/ (http://www.3ieimpact.org/en/evidence/systematic-reviews/)http://www.3ieimpact.org/en/evidence/systematic-reviews/ Best Evidence Encyclopedia (http://www.bestevidence.org/)http://www.bestevidence.org/ Cochrane Collaboration (www.cochrane.org)www.cochrane.org Campbell Collaboration (www.campbellcollaboration.org)www.campbellcollaboration.org Collaboration for Environmental Evaluation (http://www.environmentalevidence.org/)http://www.environmentalevidence.org/ National Institute for Health and Clinical Excellence (http: www.nice.org.uk/)www.nice.org.uk/ NHS Evidence (http://www.evidence.nhs.uk/)http://www.evidence.nhs.uk/ National Guidelines Clearinghouse (USA) (www.guidelines.gov)www.guidelines.gov Prospero: International Prospective Register of Systematic Reviews (http://www.crd.york.ac.uk/prospero/http://www.crd.york.ac.uk/prospero/ Social Care Institute for Excellence (http://www.scie.org.uk/)http://www.scie.org.uk/ Social Programs That Work (http://evidencebasedprograms.org/)

22. www.3ieimpact.org Philip Davies UK Policymakers’ Views of Evidence Focus on the ‘end product’, rather than how the information was either collected or analysed Use of ‘anecdotal’ evidence (“tells a story”) Drawing on such things as ‘real life stories’, ‘fingers in the wind’, ‘local’ and ‘bottom-up’ evidence But: “If we try and move anywhere without having the scientific basis to do so we get fleeced in the House” And: DfID Evidence into Action Team And: BCURE Programme + DPME (South Africa)

23. www.3ieimpact.org Philip Davies Sharks Where Do UK Civil Servants Go For Evidence? Plankton Academic/Evaluation Research?

24. www.3ieimpact.org Philip Davies UK Policymakers’ Views of Research Evidence Too Long Verbose Too Detailed Too Dense Impenetrable Too Much Jargon Too Methodological Untimely Irrelevant for policy Source: Campbell, S., et al; 2007, Analysis for Policy

25. www.3ieimpact.org Philip Davies Establish what research says and does not say Establish the policy messages and policy implications Use a 1:3:25 format Very little mention of methodology Be clear - plain English summary Be persistent and opportunistic Improving Communication of Evidence

26. www.3ieimpact.org Philip Davies Thank you Philip Davies Email: pdavies@3ieimpact.orgpdavies@3ieimpact.org +44 (0)207 958 8350 Visit www.3ieimpact.org