1. www.3ieimpact.org Philip Davies What Are Systematic Reviews, And Why Do We Need Them? Philip Davies International Initiative for Impact Evaluation [3ie] Geneva Evaluation Network Event: Systematic Reviews 7 th May 2015

2. www.3ieimpact.org Philip Davies Group Exercise 1: If someone asks you to prepare a brief in three hours on the effectiveness of some WASH intervention, where would you go for sound evidence?

3. www.3ieimpact.org Philip Davies Group Exercise 2: What is Evidence?

4. www.3ieimpact.org Philip Davies Evidence is information or data that supports, or rejects, a conclusion Evidence is almost always probabilistic Often disagreement on what counts as ‘evidence’ – get over it! Evidence is always contestable/contested Evidence is rarely self-evident Evidence can be about what is generalisable – and what is particular Hence evidence in both quantitative and qualitative Not all research is of equal value/sufficient quality Single studies can misrepresent the balance of evidence Hence, the need for systematic reviews/synthesis of evidence Some Features of Evidence

5. www.3ieimpact.org Philip Davies Why Do We Need Systematic Reviews? Sheer amount of available evidence “Beyond the capacity of the human mind”

6. www.3ieimpact.org Philip Davies What Are Systematic Reviews? And separating higher quality from lower quality evidence A way of establishing the overall balance of empirical evidence on a topic or policy A way of identifying what is generalisable and what is context specific

7. www.3ieimpact.org Philip Davies  Single studies can:  Misrepresent the balance of research evidence  Illuminate only one part of a policy issue  Be sample-specific, time-specific, context- specific  Often be of poor quality Why Do We Need Systematic Reviews?  Consequently, give a biased view of the overall evidence

8. www.3ieimpact.org Philip Davies Systematic searching for studies Systematic critical appraisal of identified studies – separating the wheat from the chaff Systematic and transparent inclusion/exclusion of studies for final review Systematic and transparent extraction of data Systematic statistical testing and analysis Systematic reporting of findings What Makes a Review Systematic?

9. www.3ieimpact.org Philip Davies Searching for Evidence

10. www.3ieimpact.org Philip Davies Key Electronic Databases of Evidence ABI/INFORM Global Applied Social Sciences Index and Abstracts (ASSIA) Business Source Complete EconLit ERIC International Bibliography of the Social Sciences (IBSS) Medline/PubMed PAIS International PolicyFile PsycInfo Sociofile Social Science Citation Index (SSCI) World Wide Political Science Abstract

11. www.3ieimpact.org Philip Davies Types of Systematic Review Statistical Meta-Analyses Narrative Systematic Reviews Rapid Evidence Assessments Qualitative Systematic Reviews Evidence Maps and Gap Maps

12. www.3ieimpact.org Philip Davies Statistical Meta-Analytical Reviews Involves data-pooling and statistical synthesis of independent studies And aggregating/cumulating samples and findings Seeks to measure and control bias Photo © Amos Gumulira Requires included studies to be as similar (homogeneous) as possible

13. www.3ieimpact.org Philip Davies Requirements for Homogeneity Similarity of: Population (or sub-groups) Intervention Comparator Outcome Overlapping confidence intervals

14. www.3ieimpact.org Philip Davies Statistical Meta-Analysis Source: David B. Wilson, 2006, A systematic review of drug court effects on recidivism Cumulative Estimate of Effect

15. www.3ieimpact.org Philip Davies Cumulative Estimate of Effect

16. www.3ieimpact.org Philip Davies Cumulative Estimate of Effect

17. www.3ieimpact.org Philip Davies Provide a narrative account of what the evidence tells us Clarify the nature of the intervention Provide descriptive and inferential statistics But on each included study individually, not cumulatively Provide a summary analysis of what the evidence suggests Provide the ‘Signal’ and ‘Noise’ of evidence Narrative Systematic Reviews

18. www.3ieimpact.org Philip Davies Scaled down systematic reviews of existing evidence Timed to meet the needs of policy makers/practitioners (1-3 months) Strategically using the ‘three arms’ of systematic searching, but less exhaustively Rapid Evidence Assessments – What Are They? Critical appraisal of identified studies is included Summary of findings, with caveats and qualifications Photo © Panos East Africa

19. www.3ieimpact.org Philip Davies Not a comprehensive or exhaustive as systematic reviews; Hence, REAs are more likely to be subject to statistical bias than a full systematic review We must, therefore proceed with greater caution with REAs Rapid Evidence Assessments - Limitations

20. www.3ieimpact.org Philip Davies Synthesise qualitative and ethnographic evidence In-depth interviews, focus groups, observational studies, documentary analysis, case studies Seek common themes, concepts and principles across different studies Detailed attention to context/contextual specificity And stakeholders’ views Do not seek generalisations Qualitative Systematic Reviews Photo © Albert Gonzalez Farran - UNAMID

21. www.3ieimpact.org Philip Davies What Are Evidence Gap Maps? E vidence Gap Maps are thematic collections of evidence in international development – e.g. health, HIV/AIDS Agriculture, WASH, extreme poverty, micro-finance, etc. An overview of existing literature structured around a framework based on key policy relevant interventions and key intermediate and final outcomes An overview of the quality of this evidence Identify where we have strong evidence, and where we do not Provides links to user-friendly summaries in the 3ie register of impact evaluation published studies and the database of systematic reviews.

22. www.3ieimpact.org Philip Davies Evidence Gap Maps

23. www.3ieimpact.org Philip Davies

24. www.3ieimpact.org Philip Davies

25. www.3ieimpact.org Philip Davies 3ie – International Initiative for Impact Evaluation (www.3ieimpact.org)www.3ieimpact.org Best Evidence Encyclopedia (http://www.bestevidence.org/) - Educationhttp://www.bestevidence.org/ Campbell Collaboration (www.campbellcollaboration.org) - Education, Crime and Justice, Social Welfarewww.campbellcollaboration.org Cochrane Collaboration (www.cochrane.org) - Healthwww.cochrane.org Coalition for Evidence-Based Policy (http://coexgov.securesites.net/index.php?keyword=a432fbc34d71c7)http://coexgov.securesites.net/index.php?keyword=a432fbc34d71c7 National Institute for Health and Clinical Excellence (http: www.nice.org.uk/) - Health www.nice.org.uk/ NHS Evidence (http://www.evidence.nhs.uk/) - Healthhttp://www.evidence.nhs.uk/ Social Care Institute for Excellence (http://www.scie.org.uk/)http://www.scie.org.uk/ Social Care Social Programs That Work http://www.evidencebasedprograms.org/ Social Programmeshttp://www.evidencebasedprograms.org/ Sources of Sound Evidence

26. www.3ieimpact.org Philip Davies Thank you Philip Davies Email: pdavies@3ieimpact.orgpdavies@3ieimpact.org +44 (0)207 958 8350 Visit www.3ieimpact.org