Philip Davies Making Evidence Accessible and Relevant for Policy and Practice Philip Davies International Initiative for Impact Evaluation [3ie] Africa Evidence Network Regional Meeting Johannesburg, South Africa, 3 rd June 2015

Philip Davies Helping people make better decisions and achieve better outcomes, by using the best available evidence from research and other sources Knowing what are effective interventions (“what works?”) In achieving which outcomes? For which groups of people? Under what conditions? Over what time span? At what costs?, plus Integrating research with decision makers’ knowledge, skills, experience, expertise and judgement What is Evidence-Based Policy

Philip Davies Evidence Experience & Expertise Judgement Resources Values, Beliefs and Ideology Habits & Bureaucratic Culture Lobbyists & Pressure Groups Pragmatics & Contingencies Factors Other Than Evidence

Philip Davies Understanding the Problem (Conceptualisation) Developing Solutions (Policy Development) Putting Solutions Into Effect (Implementation) Monitoring and Evaluation (M&E) The ‘Classic’ Policy Cycle The ‘ROAMEF’ Policy Cycle Evidence is required across the entire policy cycle

Philip Davies The Kind Of Evidence Decision-Makers Look For Identifying the nature, size and dynamics of the problem Specifying the desired objectives Identifying viable policy options Identifying how the policy is supposed to work Identifying the likely and achieved outcomes/impacts Identifying the social distribution of outcomes/impacts Understanding people’s attitudes, experiences, behaviour Valuing the impacts (cost-benefit/cost-effectiveness) Identifying effective implementation and delivery

Philip Davies Types of Evidence for Policy Making

Philip Davies Theory of Change/Logic Model/Programme Theory How is a policy/programme supposed to work? What activities, mechanisms, people, outputs have to be in place? And in what sequence – what is the causal chain? What resources are required – and are available? What data are required – and are available? Is the policy/programme feasible/achievable?

Philip Davies Constituent Features of a Theory of Change Assumptions?

Philip Davies Constituent Features of a Theory of Change Data Required Surveys, statistics, demographic data Qualitative data Costs/benefits data Systematic review data Documentary analysis Surveys, statistics, demographic data Qualitative data Costs/benefits data Systematic review data Documentary analysis Performance data Historical data Diversity data Qualitative data Effectiveness data Performance data Historical data Diversity data Qualitative data Effectiveness data Stakeholder data Qualitative data Public opinion data Effectiveness data Stakeholder data Qualitative data Public opinion data Effectiveness data Performance data Effectiveness data Stakeholder data Qualitative data Costs/benefits data Performance data Effectiveness data Stakeholder data Qualitative data Costs/benefits data Administrative data Performance data Costs/benefits data Administrative data Performance data Costs/benefits data Administrative data Performance data Qualitative data Administrative data Performance data Qualitative data Counterfactual data Administrative data Survey data, statistics Cost/benefit data Counterfactual data Administrative data Survey data, statistics Cost/benefit data

Philip Davies Types of Systematic Review Statistical Meta-Analyses Narrative Systematic Reviews Qualitative Systematic Reviews Rapid Evidence Assessments Evidence Maps and Gap Maps

Philip Davies  Single studies can:  Misrepresent the balance of research evidence  Illuminate only one part of a policy issue  Be sample-specific, time-specific, context- specific  Often be of poor quality Why Do We Need Systematic Reviews?  Consequently, give a biased view of the overall evidence

Philip Davies Systematic searching for studies Systematic critical appraisal of identified studies – separating the wheat from the chaff Systematic and transparent inclusion/exclusion of studies for final review Systematic and transparent extraction of data Systematic statistical testing and analysis Systematic reporting of findings What Makes a Review Systematic?

Philip Davies Statistical Meta-Analytical Reviews Source: David B. Wilson, 2006, A systematic review of drug court effects on recidivism

Philip Davies Synthesise qualitative and ethnographic evidence In-depth interviews, focus groups, observational studies, documentary analysis, case studies Seek common themes, concepts and principles across different studies Detailed attention to context/contextual specificity And stakeholders’ views Do not seek generalisations Qualitative Systematic Reviews Photo © Albert Gonzalez Farran - UNAMID

Philip Davies Scaled down systematic reviews of existing evidence Timed to meet the needs of policy makers/practitioners (1-3 months) Strategically using the ‘three arms’ of systematic searching, but less exhaustively Rapid Evidence Assessments – What Are They? Critical appraisal of identified studies is included Summary of findings, with caveats and qualifications Photo © Panos East Africa

Philip Davies Rapid Evidence Assessments – How Scaled Down?

Philip Davies 3ie Evidence Gap Maps Maps of the existing evidence base on a policy issue, topic or sector such as maternal health, HIV/AIDS, agriculture, extreme poverty Structured around a framework of interventions and outcomes (intermediate and final) A ways of identifying where there is evidence, and where there is not An indication of the quality of this evidence Links to user-friendly summaries in the 3ie database of systematic reviews.

Philip Davies Rapid Evidence Assessments - Limitations Evidence Gap Maps

Philip Davies

Philip Davies

Philip Davies Some Key Sources of Sythesised Evidence 3ie Impact Evaluations Database ( 3ie Systematic Reviews Database ( ( Best Evidence Encyclopedia ( Cochrane Collaboration ( Campbell Collaboration ( Collaboration for Environmental Evaluation ( National Institute for Health and Clinical Excellence (http: NHS Evidence ( National Guidelines Clearinghouse (USA) ( Prospero: International Prospective Register of Systematic Reviews ( Social Care Institute for Excellence ( Social Programs That Work (

Philip Davies UK Policymakers’ Views of Evidence Focus on the ‘end product’, rather than how the information was either collected or analysed Use of ‘anecdotal’ evidence (“tells a story”) Drawing on such things as ‘real life stories’, ‘fingers in the wind’, ‘local’ and ‘bottom-up’ evidence But: “If we try and move anywhere without having the scientific basis to do so we get fleeced in the House” And: DfID Evidence into Action Team And: BCURE Programme + DPME (South Africa)

Philip Davies Sharks Where Do UK Civil Servants Go For Evidence? Plankton Academic/Evaluation Research?

Philip Davies UK Policymakers’ Views of Research Evidence Too Long Verbose Too Detailed Too Dense Impenetrable Too Much Jargon Too Methodological Untimely Irrelevant for policy Source: Campbell, S., et al; 2007, Analysis for Policy

Philip Davies Establish what research says and does not say Establish the policy messages and policy implications Use a 1:3:25 format Very little mention of methodology Be clear - plain English summary Be persistent and opportunistic Improving Communication of Evidence

Philip Davies Thank you Philip Davies (0) Visit