1. Systematic Reviews-intro level
Phil Wiffen
Director of Training UK Cochrane Centre
Editor, Cochrane Collaborative Review Group
in Pain, Palliative & Supportive Care

2. The rationale Evidence based medicine What are systematic reviews ?
How do we find, understand and evaluate systematic reviews?
Tools to present data

3. League table of NNTs to produce at least 50% pain relief
over 4-6 hours compared to placebo in pain of
moderate or severe intensity

4. What evidence-based medicine is:
Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.
Sackett (BMJ 1996; 312: 71-2)

5. What evidence-based medicine is:
The practice of EBM requires the integration of
individual clinical expertise
with the
best available external clinical evidence from systematic research.

6. Another definition of EBM
Evidence based medicine is an approach
to health care that promotes the collection,
interpretation and integration of valid,
important and applicable patient reported,
clinician observed and research derived
evidence. The best available evidence,
moderated by patient circumstances and
preferences, is applied to improve the
quality of clinical judgements.
McKibbon KA et al ‘The medical literature as a resource for Evidence Based Care’

7. “...approximately 17000 new biomedical books are published annually.”
“There are perhaps biomedical journals in the world, and they have grown steadily by 7% a year since the seventeenth century.
Yet about 15% of medical interventions are supported by solid scientific evidence...
...only 1% of the articles in medical journals are scientifically sound”
R. Smith quoting Prof. D. Eddy, BMJ 1991; 303:
“...approximately new biomedical books are published annually.”
Lowe and Barnett, JAMA 1994; 271:
More than RCTs have been published in pain relief research since 1950

8. The size of the task How many biomedical papers are there ?
Medline 17 million records, 5000 journals, 80 countries
Embase 14 million records, 7000 journals, 70 countries
CINAHL 1 million records 2900 journals. 13 languages
Others: e.g. LILACS ????
April 2009

9. numbers / inclusion / exclusion
UNBIASED
‘good’ RCT
numbers / inclusion / exclusion
blinding
power
enthusiast
systematic reviews +
open ‘biased’ study
local use
‘expert’
clinical practice

10. Tools not Rules

11. Type & Strength of Evidence
I Strong evidence from at least 1 systematic review of multiple well-designed randomised controlled trials
II Strong evidence from at least 1 properly designed randomised controlled trial of appropriate size
III Evidence from well designed trials without randomisation, single group pre-post, cohort, time series or matched case-controlled studies
IV Evidence from well-designed non experimental studies from more than 1 centre or research group
V Opinions of respected authorities, based on clinical evidence, descriptive studies or reports of expert committees

12. What is a systematic review ?
Filing Cabinets
Friends
Foreigners ?
The world literature on a subject

13. Systematic Reviews
“Clinical review articles should be as scientific as the articles they review”
Haynes, BMJ 1992; 304: 330-1
“ The fundamental difference between a review and a primary study is the unit of analysis, not the scientific principles that apply”
Oxman & Guyatt, CMAJ 1988; 138:

14. qualitative quantitative 0. Frame question 1. Search for trials
narrative review (overview)
systematic review ± meta-analysis
0. Frame question
1. Search for trials
2. Score for quality
3. Validity of trial results
qualitative
quantitative
4. Extract data
4. Vote-counting
5. Meta-analysis

15. Finding systematic reviews
Medline has a filter

16. Finding systematic reviews
Cochrane Library
DARE database on Cochrane Library

17. Understanding systematic reviews
Meta-analysis (forest plot)
NNTs
L’Abbe plots
Assessment of bias in the included studies

18. Ibuprofen 400 mg vs. paracetamol 1000 mg for acute postoperative pain1 2 3 4 5 6 7
-20
-10 10 20 30
Cooper et al, 1984
Cooper, 1984
Cooper et al, 1989
Schachtel al, 1989
Mehlisch et al, 1990
Overall weighted difference
mean differences and 95% CI
( % of the maximum possible TOTPAR value)
favours ibuprofen
favours paracetamol
no difference
difference between the mean effects within the trial and 95% CI

19. A closer look at a forest plot (the meta-analysis)

20. Cochrane Database of Systematic Reviews
Published by John Wiley & Sons, Ltd

21. There is a label to tell
you what the comparison
is and what the outcome
of interest is

22. the labels say – things to
At the bottom there’s
a horizontal line. This
is the scale measuring
the treatment effect.
Here the outcome is ……..
Take care to read what
the labels say – things to
the left do not always mean the treatment is better than the control.
Note that when using RevMan you can change the labels at the bottom of the graph, if need be

23. The vertical line in the
middle is where the
treatment and control
have the same effect –
there is no difference
between the two

24. The data for
each trial are here, divided
into the treatment and control
groups
This is the % weight
given to this study in the
pooled analysis
For each study
there is an ID

25. The label above the graph tells you what statistic has been used
The data shown in
the graph are also
given numerically
The label above the graph
tells you what statistic
has been used
Talk about the blob first, and then the CI, then show them the numerical data to the right
Each study is given a blob, placed where the data measure the effect.
The size of the blob is proportional to the % weight
The horizontal line is called a confidence interval and is a measure of how we think
the result of this study might vary with the play of chance.
The wider the horizontal line is, the less confident we are of the observed effect.

26. ** Note on interpretation **
The pooled analysis is given a diamond shape
where the widest bit in the middle
is located at the calculated
best guess (point estimate),
and the horizontal width is the
confidence interval
** Note on interpretation **
If the confidence interval crosses the line of no effect, this is equivalent to saying that we have found no statistically significant difference in the effects of the two interventions

27. Cochrane Database of Systematic Reviews
Published by John Wiley & Sons, Ltd

28. Numbers needed to treat (NNTs)
The Number of people who have to be treated
for ONE to benefit

29. Number-needed-to-treat (NNT)
Controls
Ncon
Impcon
Actives
Nact
Impact
Number of patients
Improved = Clinical end point 1
NNT =
Impact
Impcon -
Nact
Ncon

30. Number-needed-to-treat (NNT)
NNT is treatment specific -takes into account the
event rate in controls:
may be a placebo effect
may be the effect of another treatment 1
100 -
NNT =

31. Number Needed to Treat (NNT) = 1/AR
active control
improved 80 20 N
Relative Risk (RR) = (Impact/Nact) / (Impcon/Ncon)
Relative Risk Reduction (RRR) = (1-RR) / 100
Absolute Risk (AR) = (Impact/Nact) - (Impcon/Ncon)
Number Needed to Treat (NNT) = 1/AR
RR = 4; AR = 0.6; NNT = 1.7 (best 1.25)

32. L'Abbé plot for treatment
100
Treatment better
than control
equality
Proportion improved
with treatment 75 50
Control better
than treatment 25 25 50 75
100
Proportion improved with control

33. World literature on paracetamol10 20 30 40 50 60 70 80 90
100
200
600/650
500
1000
At least 50% pain relief with paracetamol
At least 50% pain relief with placebo
Moore et al Pain 1997;70:193

34. ‘Risk of bias’ assessment in Cochrane reviews

35. Risk of bias summary
Here ‘Blinding’ and ‘Incomplete outcomes data’ have been assessed for two sets of outcomes

36. Evaluating systematic reviews
Critical appraisal skill programme (CASP)
10 questions to make sense of a review

37. Ten questions to make sense of a review
Adapted from Oxman AD et al Users Guide to the Medical Literature VI How to use an overview. JAMA 1994; 272 (17):
For each question answer : YES, NO or DON’T KNOW
A. Are the results of the review valid ?
1. Did the review address a clearly focused issue ?
e.g. the population , intervention and or outcomes
2. Did the authors look for the appropriate sort of papers ?
Did they deal with the issues and have appropriate study design ?
Is it worth continuing ??
3. Do you think the important relevant studies were included ?
Look for search methods, reference list use, unpublished studies and non English language
4. Did the authors do enough to assess the quality of included studies ?
5. If the results of studies have been combined, was it reasonable to do so ?

38. B. What are the results ?
6. What is the overall result of the review ?
Is there a clear numerical expression ?
7. How precise are the results ?
Confidence intervals ?
C. Will the results help my local situation ?
8. Can the results be applied locally ?
9. Were all important outcomes considered ?
10. Are the benefits worth the harms and costs ?
Phil Wiffen Sep2011

40. Conclusions
Evidence based medicine has emphasised the importance of systematic reviews as evidence for care
Need to know how find systematic reviews
Need to know how to read a meta-analysis