1. Is a meta-analysis right for me? Jaime Peters j.peters@exeter.ac.uk June 2014

2. Systematic review Study 1Study 2 Study 3 Study 4Study N …. Meta-analysis What is a meta-analysis? (1) Gene Glass 1976: “Meta-analysis refers to the analysis of analyses” Quantitative synthesis of data extracted from a systematic review

3. What is a meta-analysis? (2) Framework for investigation into –Heterogeneity/differences between study results –Biases associated with the review (publication/reporting) –Assess sensitivity of results A weighted average

4. Why is a meta-analysis useful? Meta-analysis > single study: therefore greater power to detect an effect if one exists Quantify effect sizes and their uncertainty –“average effect is X”, rather than “most studies report a positive effect” Facilitate synthesis of large number of studies –Summarising tens, sometimes hundreds studies Allows exploration of heterogeniety Obtain results that inform evidence-based medicine –E.g. input directly into cost-effectiveness analyses

5. What might affect your decision to do a meta-analysis? Study quality Heterogeneity Reporting or publication biases Incomplete reporting Importance of a protocol

6. Example: Arsenic systematic review

7. Study quality Different study designs can be subject to different types of bias, e.g. Blinding in RCTs; recall bias in case-controls Quality assessment, e.g. Cochrane risk of bias tool But, what to do with this information? –Garbage in, garbage out –Precise meta-analysis results, but how useful? Use a pre-defined cut-off for the meta-analysis? Assess sensitivity of results to low quality studies Down-weight low quality studies?

8. Example: Arsenic systematic review Quality appraisal: http://www.ephpp.ca/PDF/Quality%20Assessment%20T ool_2010_2.pdf http://www.ephpp.ca/PDF/Quality%20Assessment%20T ool_2010_2.pdf 51 studies –Weak –Moderate –Strong Weak n=50 Strong n=1

9. Heterogeneity Populations age sex Setting geography primary/ secondary care Clinical Methodological Statistical Study design RCT, cohort, case-control, etc Outcome measure mean difference, odds ratio, hazard ratio, etc Consequence of clinical and/or methodological heterogeneity Observed effects from each study more different than we would expect by chance ProtocolSub-groupsQ-test, I 2 Meta-regression Work-arounds Average effect Individual patient data

10. Example: Arsenic systematic review Interventions –51 studies –50 interventions! Study design –RCTs –Pre/post –Non-randomised –Cross-over Follow-up times Country Unit of analysis –Samples from multiple sites –Repeat samples from one site –Repeat samples from multiple sites

11. Reporting and publication biases Interesting, positive, statistically significant results more likely to be published and reported Without consideration, meta-analysis just replicates, sometimes exaggerates, this bias Avoidance: comprehensive search of grey literature Methods to assess presence of reporting and/or publication biases Funnel plots, tests, methods for adjustment

12. Example: Arsenic systematic review Comprehensive search, including grey literature Due to heterogeneity, no attempt to investigate presence of reporting or publication bias Mention possibility of reporting bias as limitation Methods for identifying or adjusting for reporting and publication bias are not ideal Require >10? studies and perform poorly in presence of heterogeneity

13. Incomplete reporting Word, table, figure limits Poor reporting Contact authors Individual patient data

14. Example: Arsenic systematic review Poor quality reporting Could not extract necessary outcome data –“effectiveness evidence inconclusive” Did not contact authors – time and resources running out! Major limitation of the systematic review Did we conduct a meta-analysis?

15. Other difficulties Too few studies: testing for heterogeneity and publication/reporting biases, meta-regression and sub- groups Multiple reports of same study Outcomes measured at different time-points Most issues not restricted to meta-analysis Meta-analysis: systematic and transparent framework Protocol considering possibilities very important Cochrane Handbook great resource: handbook-cochrane.org

16. Useful refs AJ Sutton et al (2000) Methods for Meta-analysis in Medical Research, Wiley. [Systematic review of trials and other studies, HTA, 1998 Vol 2 No 19 http://www.journalslibrary.nihr.ac.uk/hta/volume-2/issue-19] http://www.journalslibrary.nihr.ac.uk/hta/volume-2/issue-19 JPT Higgins and S Green (2011) Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 (Cochrane website) M Egger et al (2001) Systematic Reviews in Health Care: Meta-analysis in Context, BMJ. M Borenstein et al (2009) Introduction to Meta-analysis (Statistics in Practice), Wiley. j.peters@exeter.ac.uk Questions? Comments?