1. www.3ieimpact.org Philip Davies What Is Evidence, and How Can It Improve Decision Making? Philip Davies International Initiative for Impact Evaluation [3ie] BCURE Evidence-Informed Decision-Making Capacity Building Workshop 1st and 2nd June 2015 Pretoria, South Africa

2. www.3ieimpact.org Philip Davies Group Exercise What is Evidence?

3. www.3ieimpact.org Philip Davies Evidence is information or data that supports, or rejects, a conclusion Evidence has standards of validity and reporting Evidence is almost always probabilistic Often disagreement on what counts as ‘evidence’ Evidence is rarely self-evident Evidence can be about what is generalisable – and what is particular Hence evidence in both quantitative and qualitative Not all research is of equal value/sufficient quality Single studies can misrepresent the balance of evidence Hence, the need for systematic reviews/synthesis of evidence Some Features of Evidence

4. www.3ieimpact.org Philip Davies Different Notions of Evidence Policy Makers’ Evidence Colloquial (Narrative) Anything that seems reasonable Policy relevant Timely Clear Message Researchers’ Evidence ‘Scientific’ (Generalisable) Proven empirically Theoretically driven As long as it takes Caveats and qualifications Source: J. Lomas et al, 2005 Knowledge Translation and Transfer

5. www.3ieimpact.org Philip Davies UK Policymakers’ Views of Evidence Focus on the ‘end product’, rather than how the information was either collected or analysed Use of ‘anecdotal’ evidence (“tells a story”) Drawing on such things as ‘real life stories’, ‘fingers in the wind’, ‘local’ and ‘bottom-up’ evidence But: “If we try and move anywhere without having the scientific basis to do so we get fleeced in the House” And: DfID Evidence into Action Team And: BCURE Programme + DPME (South Africa)

6. www.3ieimpact.org Philip Davies Evidence Experience & Expertise Judgement Resources Values, Beliefs and Ideology Habits & Bureaucratic Culture Lobbyists & Pressure Groups Pragmatics & Contingencies Factors Other Than Evidence

7. www.3ieimpact.org Philip Davies Challenge for this Workshop How can we integrate the use of evidence with these other influencing factors?

8. www.3ieimpact.org Philip Davies Help people make better decisions and achieve better outcomes Provide a better understanding of the nature, size and dynamics of the issue at hand Compare policy options objectively and rigorously Clarify how a policy is supposed to work (ToC) Calculate empirically the size of likely impacts Provide more precise estimates of risk and bias Calculate empirically the diversity/variance of outcomes and impacts Establish a cumulative evidence base for decision making. How Can Evidence Influence Policy?

9. www.3ieimpact.org Philip Davies Group Exercise: What is Evidence versus Opinion?

10. www.3ieimpact.org Philip Davies Evidence versus Opinion Evidence: Data that meet the standards of internal validity, adequacy of reporting, and external validity Key qualities: independence, objective, verifiable Opinion Statements and claims that do not meet the standards of evidence Key qualities: positional, subjective, partial (selective), hard to verify

11. www.3ieimpact.org Philip Davies Hoodies Maths Teaching Welfare- to-Work Opinion-Based Policy Increasing Pressure/Time Evidence-Based Policy Evidence-Based vs Opinion-Based Policy

12. www.3ieimpact.org Philip Davies Thank you Philip Davies Email: pdavies@3ieimpact.orgpdavies@3ieimpact.org +44 (0)207 958 8350 Visit www.3ieimpact.org