Systematic reviews and meta-analyses: when and how to do them Andrew Smith Royal Lancaster Infirmary 18 May 2015

Archie Cochrane

Cochrane’s challenge ‘It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials’

Traditional reviews and the ‘desk drawer’ problem

Systematic review ‘A review in which evidence on a topic has been systematically identified, appraised and summarised according to predetermined criteria’

Systematic review 1. Decide scope and purpose (PICO) 2. Exhaustive search for material 3. Inclusion on pre-defined criteria 4. Quality assessment of relevant studies 5. Data extraction 6. Synthesis and integration 7. Interpretation

Primary and secondary research Reviews are retrospective, observational studies and are subject to both systematic and random error Bias can be reduced by applying the same principles in reviews as primary studies

Bias Primary Recruitment Selection Allocation Detection Attrition Reporting Secondary Selection Publication Language Inclusion Appraisal Interpretation

Funnel plots

Trial quality assessment

Meta-analysis

Heterogeneity: variability in estimates of treatment effects between studies in a meta-analysis Significant heterogeneity suggest that trials are not estimating a single common effect – possibly due to differences in patients, setting, intervention and outcomes Can be clinical, methodological or statistical Can be explored by subgroup and sensitivity analyses Meta-regression can also be applied Heterogeneity

Common ‘faults’ Too little thought Too many outcomes Too many subgroups ‘Lumping’ vs. ‘splitting’ Scoring risk of bias Not doing anything with quality assessment

Challenges in systematic reviews The RCT itself Anaesthesia/critical care Changing methodology New review types: DTA, non-randomised, prognostic, network meta-analysis False certainty

Trial sequential analysis

Challenges in systematic reviews The RCT itself Anaesthesia/critical care Changing methodology New review types: DTA, non-randomised, prognostic, network meta-analysis False certainty Uncertainty: inconclusive conclusions

The limits to knowledge doubtful unknown certain

When When you’re thinking about a research project When you need practical guidance When you really can’t do any primary research!

How Think first – write a protocol! Use PICO to specify the question Register the review Use ‘risk of bias’ tool for quality Get as many comments as possible Be explicit about process and judgements Be realistic about what can be achieved Use PRISMA tool for reporting