1. Systematic Reviews and Meta-Analysis -Part 2-
SPICE+B 2017
Systematic Reviews and Meta-Analysis -Part 2-
Farid Foroutan, PhD (c)
Health Research Methods, Evidence, and Impact
McMaster University
July 15, 2019
Ani Orchanian-Cheff, Information Specialist

2. Plan Importance of systematic reviews Process of systematic reviews
Assessment of confidence in evidence (GRADE)

3. Systematic Reviews & Meta-Analysis
Identify ALL studies EVER conducted
Appraise the quality of identified studies (risk of bias)
Meta-Analysis
Combine results of similar studies quantitatively
Produces summary statistics that represents different studies
Summary statistics more precise – combine sample size

4. Thrombolytic Therapy Textbook/Review Recommendations Cumulative 0.5
1.0
2.0
Year
RCTs
Pts
1960
Experimental
Not Mentioned
Rare/Never
Routine
Specific
1965 21 5
1970 1 10 1 2
P<.01 2 8 7
1980 8 1 12
P<.001 M 1 8 4
1985 M 1 7 3 M
P<.00001 5 2 2 M 1 15 8 M 1
1990 M 6 1
Odds Ratio (Log Scale)
Favours Treatment
Favours Control

5. Formulate question question
eligibility criteria / methodologic criteria
a priori hypotheses to explain heterogeneity
conduct search
review titles and abstracts
review full text of possibly eligible articles
assess risk of bias, abstract data
generate pooled estimates and confidence intervals
look for explanations of heterogeneity
GRADE evidence profile

6. Formulate question question
eligibility criteria / methodologic criteria
a priori hypotheses to explain heterogeneity
conduct search
review titles and abstracts
review full text of possibly eligible articles
assess risk of bias, abstract data
generate pooled estimates and confidence intervals
look for explanations of heterogeneity
GRADE evidence profile

7. Formulation of Research Question
Population
Intervention
Comparison
Outcome
Impact of acupuncture in patients with tension-type headache when compared to usual care, placebo or other interventions?
P patients with tension-type headache
I acupuncture
C usual care, placebo, other intervention
O number of headaches, duration
functional limitation, quality of life
adverse effects

8. Eligibility Criteria Study designs to include Unbiased studies
RCTs
RCTs with blinding
RCTs with blinding > 99% fu
Cohort studies > 80% fu
Very high likelihood of bias
Unbiased studies

9. Eligibility Criteria Methods: allocation explicitly randomized
patients followed > 8 weeks after randomization
Language: all languages
Time Frame: no restriction
Publication type: published and unpublished

10. Unpublished studies? To avoid/minimize Publication Bias:
the selective publication of manuscripts based on the magnitude, direction, or statistical significance of the study results
Negative studies are less likely to be published than studies than positive studies

11. Eligibility Criteria

12. Formulate question question
eligibility criteria / methodologic criteria
a priori hypotheses to explain heterogeneity
conduct search
review titles and abstracts
review full text of possibly eligible articles
assess risk of bias, abstract data
generate pooled estimates and confidence intervals
look for explanations of heterogeneity
GRADE evidence profile

13. Vitamin D & Non-vertebral Fractures

14. Possible reasons for inter-study variability
Technical term for inter-study variability = heterogeneity
Possible reasons?
Population characteristics…
Intervention (dose, mode of administration, etc…)
Comparator (placebo, usual practice, etc…)
Outcome (definition, severity level, etc…)
Risk of Bias (high/medium vs. low)

15. Formulate question question
eligibility criteria / methodologic criteria
a priori hypotheses to explain heterogeneity
conduct search
review titles and abstracts
review full text of possibly eligible articles
assess risk of bias, abstract data
generate pooled estimates and confidence intervals
look for explanations of heterogeneity
GRADE evidence profile

16. Conducting search strategy
Hope you were awake for the first half

17. Screening likely be many citations identified through the searches
2 independent reviewers
Need to achieve consensus when the 2 reviewers disagree
discussion or third reviewer

18. Formulate question question
eligibility criteria / methodologic criteria
a priori hypotheses to explain heterogeneity
conduct search
review titles and abstracts
review full text of possibly eligible articles
assess risk of bias, abstract data
generate pooled estimates and confidence intervals
look for explanations of heterogeneity
GRADE evidence profile

19. Full Text Review Detailed review to establish relevance definitively
Apply same criteria (PICO and design)
2 independent reviewers
Need to achieve consensus when the 2 reviewers disagree
discussion or third reviewer

20. Formulate question question
eligibility criteria / methodologic criteria
a priori hypotheses to explain heterogeneity
conduct search
review titles and abstracts
review full text of possibly eligible articles
assess risk of bias, abstract data
generate pooled estimates and confidence intervals
look for explanations of heterogeneity
GRADE evidence profile

21. Data Extraction Data extraction by two independent reviewers
ONLY done for eligible included
Demographic characteristics of populations
Population, intervention, comparison
Risk of bias
Outcomes

22. Risk of Bias (RCTs)
1.) Was allocation sequence adequately generated? 2.) Was allocation adequately concealed? 3.a) Were patients blinded? 3.b) Were healthcare providers blinded? 3.c) Were data collectors blinded? 3.d) Were outcome assessors blinded? 3.e) Were data analysts blinded? 4.) Was loss to follow-up (missing outcome data) infrequent? 5.) Are reports of the study free of suggestion of selective outcome reporting?

23. Risk of Bias (Observational Studies)
cohorts drawn from same population
measurement of exposure
measurement prognostic factors
adjustment for prognostic factors
assessment of outcome
completeness of follow-up

24. Formulate question question
eligibility criteria / methodologic criteria
a priori hypotheses to explain heterogeneity
conduct search
review titles and abstracts
review full text of possibly eligible articles
assess risk of bias, abstract data
generate pooled estimates and confidence intervals
look for explanations of heterogeneity
GRADE evidence profile

25. Meta-analysis Combine results of similar studies quantitatively
Produces summary statistics that represents different studies
Summary statistics more precise – combine sample size

26. Meta-analysis When not to meta-analyze High clinical heterogeneity
Severe vs. mild disease
Different lengths of follow-up
Varied interventions
Methodological heterogeneity is too high
High vs. low risk of bias studies

27. Meta-analysis Things to consider before starting:
A priori hypotheses (sub-grouping of studies)
Types of comparisons
Treatment A vs placebo
Treatment B vs Treatment A
Treatment C vs Treatment A
Types of outcomes
Dichotomous
Continuous

28. Two models for pooling
Random effects vs. Fixed effect

29. Two models for pooling Train 50 teachers Questions one could ask
10 classes each over next year
randomize 5 to old and new curricula
Questions one could ask
among these 50 teachers, what is the relative impact of the two curricula?
among a wider group of teachers of whom these 50 are a random sample, what is the relative impact of the two curricula?
Assumptions one could make
relative impact same in all teachers
relative impact differs across teachers

30. Two models for pooling differences
question: these teachers versus all teachers
assumption: effect same or different
substitute studies for teachers and you have the question and assumption for fixed and random effects

31. Two models for pooling Fixed Random Conceptual
Estimates effect in this sample of studies
Effects same in all studies
Estimates effect in population of which studies are random sample
Effects differ– what is the average effect
Statistical
Error variance only within study
Error variance within and between
Practical
Narrow CI
weight large vs small ↑↑↑
Wider CI
weight large vs small ↑

32. Fixed:
CI same in the face
of small or large variability

33. Random:
CI wider in large variability
than in small variability

34. Lots of variability:
Random CI wider than
fixed CI
Fixed
Random

35. Fixed Random Lots of variability and small
studies have different estimates
than large studies:
Random CI wider than fixed CI and
point estimate of random close to
small studies
Fixed
Random

36. What to choose? Random uncertainty does increase with variability
CIs should capture that
argues for random effect
we always use, unless….
dominating trial(s) with results discrepant with small trials

37. Quality (Strength) of Evidence
The GRADE method
Grading of Recommendations Assessment Development and Evaluation
Classifies evidence as (very low, low, moderate, and high)
Based on classification on 5 required
Quality of studies
Consistency of studies
Directness of evidence to clinical question
Precision of results
Publication bias
And four non-required domains
Dose-response association
Plausible confounding
Strength of association

38. What are we grading? Two components quality of body of evidence
confidence in estimate of effect
high, moderate, low, very low
RCTs start high
observational studies start low
strength of recommendation
strong and weak

39. Determinants of Quality
What can lower quality?
Risk of Bias
Inconsistency
Indirectness
Imprecision
Publication bias

40. Risk of bias

41. Risk of bias

42. Risk of bias

43. Risk of Bias
Statistically significant difference between pooled effect estimate of high and low risk of bias studies:
Consider only reporting on low risk of bias studies
Not significantly different, JUST a RISK of bias.
If all studies high risk of bias, rate down!!

44. Consistency of Results

45. Consistency of Results

46. What Criteria Were You Using?
Similarity of point estimates
less similar, more inclined to say “heterogeneity”
Overlap of confidence intervals
less overlap, more inclined to say “heterogeneity”

47. Homogenous test for heterogeneity what is the p-value?
what is the null hypothesis
for the test for heterogeneity?
Ho: RR1 = RR2 = RR3 = RR4
p=0.99 for heterogeneity

48. Heterogeneous test for heterogeneity what is the p-value?
p-value for heterogeneity < 0.001

49. Homogenous
What is the I2 ?
p=0.99 for heterogeneity
I2=0%

50. Heterogeneous What is the I2 ? I2=89%
p-value for heterogeneity < 0.001
I2=89%

51. Consistency of Results
If inconsistency, look for explanation
patients, intervention, outcome, methods
Judgment of consistency
variation in size of effect
overlap in confidence intervals
statistical significance of heterogeneity I2

52. Directness of Evidence
Differences in patients (age, sex, ethnicity, condition)
Differences in interventions (dose, class)
Differences in outcomes (health-related quality of life, functional capacity, radiological)

53. Imprecision Small sample size Wide confidence intervals
small number of events
Wide confidence intervals
uncertainty about magnitude of effect

54. Imprecision
Necessary to use absolute risks as opposed to relative risks
Relative risk of 1.5 (CI 1.2 to 1.8) – Drug A vs Placebo
Baseline risk for Placebo = 5% - becomes (6% to 9%)
Baseline risk for Placebo = 30% - becomes (36% to 54%)

55. Imprecision
10% 0%

56. Publication Bias High likelihood could lower quality
Reporting of outcomes
Reporting of studies
publication bias
number of small studies
industry sponsored

59. Beta blockers in non-cardiac surgery
Quality Assessment
Summary of Findings
Quality
Relative Effect
(95% CI)
Absolute risk difference
Outcome
Number of participants
(studies)
Risk of Bias
Consistency
Directness
Precision
Publication Bias
Myocardial infarction
10,125
(9)
No serious limitations
No serious imitations
Not detected
High
0.71
(0.57 to 0.86)
1.5% fewer
(0.7% fewer to 2.1% fewer)
Mortality
10,205
(7)
Possiblly inconsistent
Imprecise
Moderate or low
1.23
(0.98 – 1.55)
0.5% more
(0.1% fewer
to 1.3% more)
Stroke
10,889
(5)
No serious limitaions
Possible imprecision
Moderate or High
2.21
(1.37 – 3.55)
(0.2% more to
1.3% more0

60. Questions/Discussion
Thank you!
Special thanks to Dr. Gordon Guyatt