1. Systematic Review Systematic review
Why do we worry about reviews with misleading results?
How does a systematic review protect against misleading results
Understanding inconsistency in systematic reviews

2. Thrombolytic Therapy Textbook/Review Recommendations Cumulative 0.5
1.0
2.0
Year
RCTs
Pts
1960
Experimental
Not Mentioned
Rare/Never
Routine
Specific
1965 21 5
1970 1 10 1 2
P<.01 2 8 7
1980 8 1 12
P<.001 M 8
1985 1 4 M 7 1 3 M
P<.00001 5 2 2 M 1 M 15 8 1
1990 M 6 1
Odds Ratio (Log Scale)
Favours Treatment
Favours Control

3. Prophylactic Lidocaine in MI
Outcome = death
Favors treatment
Favors placebo
Relative risk (CI)
Cumulative
Year # RCTs Subjects
Recommendations
Yes No Not mentioned
9 1 1
8 0 2
5 0 2
This slide shows a cumulative meta-analysis of the effect of prophylactic lidocaine in preventing death from myocardial infarction. As in the previous examples, this slide shows:
Expert opinion differs from available evidence
Expert opinion varies
8 0 3
4 2 1
[Gordon - it was not clear in your original slide when the 1st meta-analysis was published. I have identified Hine et al, Arch Intern Med 1989, is this correct?]
st meta-analysis published

4. What went wrong?

5. Unclear too broad question
Unrepresentative articles
Failure to understand evidence quality
Biased inferences

6. unclear too broad question
unrepresentative articles
Evidence quality poor understanding
biased inferences
explicit eligibility
(PICO, methods)
comprehensive search
RoB assessment
duplicate eligibility, risk of bias
MA (absolute)
Formal quality rating

7. unclear too broad question
unrepresentative articles
Evidence quality poor understanding
biased inferences
explicit eligibility
(PICO, methods)
comprehensive search
RoB assessment
duplicate eligibility, risk of bias
MA (absolute)
Formal quality rating

8. unclear too broad question
unrepresentative articles
Evidence quality poor understanding
biased inferences
explicit eligibility
(PICO, methods)
comprehensive search
RoB assessment
duplicate eligibility, risk of bias
MA (absolute)
Formal quality rating

9. unclear too broad question
unrepresentative articles
Evidence quality poor understanding
biased inferences
explicit eligibility
(PICO, methods)
comprehensive search
RoB assessment
duplicate eligibility, risk of bias
MA (absolute)
Formal quality rating

10. The right question all cancer therapy for all cancers
all antiplatelet agents for all atheroembolic events (heart, head, leg)
all aspirin doses for stroke
30 to 300 mg. for ischemic stroke
How did you decide when ok to pool?

11. What were your criteria?
Across range of
patients
interventions
comparators
outcomes
Effect more or less same
If not
big effect in severe patients, no effect in mild
big effect in high dose, no effect in low
big effect in short term, none in long term

12. Inconsistency
When doing meta-analysis, need to check if assumption is accurate:
effect similar across
patients
interventions
outcomes
methodology

13. Are you happy pooling?

14. Are you happy pooling?

15. What criteria were you using?
similarity of point estimates
less similar, less happy
overlap of confidence intervals
less overlap, less happy

16. Homogenous Ho: RR1 = RR2 = RR3 = RR4
test for heterogeneity what is the p-value?
p=0.99 for heterogeneity

17. -40
-24 -8 8 24 40 56
RRR (95% CI)

18. Heterogeneous test for heterogeneity what is the p-value?
p-value for heterogeneity < 0.001

19. Only a little concerned
I2 Interpretation
100%
Why are we pooling?
Very concerned
Only a little concerned
Getting concerned 0%
No worries

20. Homogenous
What is the I2 ?
p=0.99 for heterogeneity
I2=0%

21. Heterogeneous What is the I2 ? I2=89%
p-value for heterogeneity < 0.001
I2=89%

22. Homogenous
If this result, what next?
p=0.99 for heterogeneity
I2=0%

23. Heterogeneous If this result, what next? I2=89%
p-value for heterogeneity < 0.001
I2=89%

24. Heterogeneity look for explanation patients interventions outcomes
risk of bias
No good explanation? What to do?
Decrease confidence in effect estimates

25. Stroke
p=0.99 for heterogeneity
I2= 0%

26. Total Fractures

27. p=0.04 for heterogeneity

28. p=0.04 for heterogeneity
I2=43%

29. Vitamin D versus placebo/control

30. Vitamin D versus placebo/control
p= 0.07 for heterogeneity

31. Vitamin D versus placebo/control
p= 0.07 for heterogeneity
I2= 53%

33. p= 0.32 for heterogeneity

34. p= 0.32 for heterogeneity
I2= 14%

35. Summary Lots of reasons traditional reviews went wrong
Systematic reviews: strategies to protect against misleading results
Single estimate most useful when same effect across patients, interventions, outcomes, methods
Is there excessive heterogeneity?
estimates too variable, confidence intervals non-overlapping
low heterogeneity p-value, high I2
if so, look for explanation
patients, intervention, outcome, methodology
unexplained rate down for inconsistency
Do you tweet?

36. @EBCPMcMaster
Follow us on Twitter