1. Systematic reviews and Meta-analyses
Alison Brettle,
Research Fellow (Information)
Salford Centre for Nursing, Midwifery and Collaborative Research
University of Salford

2. Aims
To discuss the role of the systematic reviews and meta-analyses and cover issues involved in their critical appraisal and interpretation

3. Systematic Review
A review of all the literature on a particular topic, which has been systematically identified, appraised and summarised giving a summary answer.

4. What is a systematic review?
An overview of primary research studies conducted according to explicit and reproducible methodology
A rigorous method of summarising research evidence
Shows what we know and don’t know about a topic area
Provides evidence of effectiveness (or not) by summarising and appraising relevant evidence

5. Systematic reviews aim
To find all relevant research studies (published and unpublished)
To assess each study on basis of defined criteria
Synthesise the findings in an unbiased way
Present a balanced and impartial summary of the findings taking any flaws into consideration

6. Advantages of systematic reviews
Summarise evidence, keep people up to date without reading all published research literature
Allow large amounts of data to be assimilated (eg by busy clinicians, policy makers etc)
A clearer picture by collating results of research
Reduce bias – removes reviewers personal opinions, preferences and specialist knowledge
Explicit methods - allow the reader to assess how review has been compiled
More reliable conclusions because of methods used

7. Systematic review models
Medical/Health care
Cochrane Collaboration, NHS Centre for Reviews and Dissemination
Usually includes “high quality” research evidence – RCTs
Often includes meta-analysis (mathematical synthesis of results of 2+ studies that addressed same hypothesis in same way)
Social care/Social Sciences
SCIE, EPPI Centre, Campbell Collaboration
Often include wider range of studies including qualitative
Often narrative synthesis of evidence

8. Systematic review process
Define/focus the question
Develop a protocol
Search the literature (possibly 2 stages scoping and actual searches)
Refine the inclusion/exclusion criteria
Assess the studies (data extraction tools, 2 independent reviewers)
Combine the results of the studies to produce conclusion– can be a qualitative or quantitative (meta-analysis)
Place findings in context – quality and heterogeniety of studies, applicability of findings

9. Methodology for a systematic review of randomised controlled trials1
Greenhalgh, T, BMJ 1997;315:

10. What type of study design?
How effective is paracetamol at reducing pain?
Does smoking increase the risk of oral cancer?

11. STRONG Experimental studies/ clinical trials
Randomised controlled trials
Non-randomised controlled trials
Observational studies
Cohorts
Case-controls
Cross-sectional surveys
Case series
Case reports
WEAK Expert opinion, consensus

12. Experimental studies Randomised controlled trial
Non-randomised controlled clinical trial
Evaluating the effectiveness of an intervention

13. Observational studies
Measuring the incidence of a disease; looking at the causes of disease; determining prognosis
Looking at the causes of disease; identification of risk factors; suitable for examining rare diseases
Measuring the prevalence of a disease; examining the association
Cohort
Case-control
Cross-sectional
survey

14. What is a meta-analysis?
Optional part of a systematic review
Systematic reviews
Meta-analyses

15. Meta-analysis
The process of using statistical methods to combine the results of different studies.
The aim is to integrate the findings, pool the data, and identify the overall trend of results
(Dictionary of Epidemiology, 1995)

16. Systematic Reviews
Understanding the jargon and the blobs!

17. Odds Ratio, Relative Risk Measures of risk
The likelihood of something happening V
The likelihood of something not happening
Methods that describe/work out if something is causing harm (or having an effect)

18. Odds Ratio Graph (Blobbogram)
LEFT E S M O
RIGHT E
Line of no significance 2
more than 1
0.5
less than 1 1

19. Odds Ratio 2 more than 1 0.5 less than 1 1 Best estimate
Confidence Interval
(wobble factor) 2
more than 1
0.5
less than 1 1

20. Odds Ratio (Blobbogram)2
more than 1
0.5
less than 1 1
Less than one is good – causing no harm (having an effect), middle line is no effect and right of line is causing harm/having no effect

21. Confidence Interval
Is the range within which the true size of effect (never exactly known) lies, with a given degree of assurance (95% or 99%).

22. Confidence Intervals (Wobble factor)

23. Confidence Interval (CI)
= the wobble factor, how sure are we about the results?
- the shorter the CI the more certain we are about the results
- if it crosses the line of 1 (no treatment effect) the intervention might not be doing any good and could be doing harm

24. Heterogeneity
Clinical heterogeneity – differences in trial characteristics
Statistical heterogeneity - the variability in the reported effect sizes between studies
how similar are the results?
are the differences among the results of the trials greater than could be expected by chance alone?

25. Number needed to treat (NNT)
the number of people you would need to treat with a specific intervention to see one additional occurrence of a specific outcome
Another way of describing the benefits of treatment

26. The p-value in a nutshell
How often you would see a similar result by chance, when
actually there was no effect by the drug or treatment.
Impossible Certain Absolutely
p= Very unlikely 1 in 1000
p= Fairly unlikely 1 in 20
p= Fairly likely 1 in 2
p= Very likely 3 in 4
The probability of something happening

27. Critical appraisal Is the study valid? What are the results?
Trustworthy
What are the results?
Is it useful in practice?
Relevant?
Generalisable?

28. Evaluating quality of systematic reviews
Is there a clearly defined question?
Thorough and comprehensive search
Was methodological quality assessed and studies weighted accordingly? (Were studies reliable and valid?)
How sensitive are the results to the way the review was done – ie if you changed the inclusion criteria how would this affect results?
Interpretation of numerical results
Things you are looking for to establish if it is valid

29. Further reading
Greenhalgh T (1997) How to read a paper: papers that summarize other papers (systematic reviews and meta-analyses), BMJ, 315:

30. Useful resources Cochrane Collaboration http://www.cochrane.org/
Centre for Reviews and Dissemination
Finding studies for systematic reviews
EPPI-Centre – Stages of a review
SCIE - The conduct of systematic research reviews for SCIE knowledge reviews