Systematic Reviews and Meta-Analysis -Part 2- SPICE+B 2017 Systematic Reviews and Meta-Analysis -Part 2- Farid Foroutan, PhD (c) Health Research Methods, Evidence, and Impact McMaster University farid.foroutan@uhn.ca July 15, 2019 Ani Orchanian-Cheff, Information Specialist ani.orchanian-cheff@uhn.ca

Plan Importance of systematic reviews Process of systematic reviews Assessment of confidence in evidence (GRADE)

Systematic Reviews & Meta-Analysis Identify ALL studies EVER conducted Appraise the quality of identified studies (risk of bias) Meta-Analysis Combine results of similar studies quantitatively Produces summary statistics that represents different studies Summary statistics more precise – combine sample size

Thrombolytic Therapy Textbook/Review Recommendations Cumulative 0.5 1.0 2.0 Year RCTs Pts 1 23 1960 Experimental Not Mentioned Rare/Never Routine Specific 2 65 1965 3 149 21 4 316 5 1970 7 1793 1 10 10 2544 1 2 11 2651 P<.01 15 3311 2 8 17 3929 22 5452 7 23 5767 1980 8 1 12 27 6125 P<.001 30 6346 M 1 8 4 1985 33 6571 M 1 7 3 43 21 059 M 54 22 051 P<.00001 5 2 2 M 1 65 47 185 67 47 531 15 8 M 1 1990 70 48 154 M 6 1 Odds Ratio (Log Scale) Favours Treatment Favours Control

Formulate question question eligibility criteria / methodologic criteria a priori hypotheses to explain heterogeneity conduct search review titles and abstracts review full text of possibly eligible articles assess risk of bias, abstract data generate pooled estimates and confidence intervals look for explanations of heterogeneity GRADE evidence profile

Formulate question question eligibility criteria / methodologic criteria a priori hypotheses to explain heterogeneity conduct search review titles and abstracts review full text of possibly eligible articles assess risk of bias, abstract data generate pooled estimates and confidence intervals look for explanations of heterogeneity GRADE evidence profile

Formulation of Research Question Population Intervention Comparison Outcome Impact of acupuncture in patients with tension-type headache when compared to usual care, placebo or other interventions? P patients with tension-type headache I acupuncture C usual care, placebo, other intervention O number of headaches, duration functional limitation, quality of life adverse effects

Eligibility Criteria Study designs to include Unbiased studies RCTs RCTs with blinding RCTs with blinding > 99% fu Cohort studies > 80% fu Very high likelihood of bias Unbiased studies

Eligibility Criteria Methods: allocation explicitly randomized patients followed > 8 weeks after randomization Language: all languages Time Frame: no restriction Publication type: published and unpublished

Unpublished studies? To avoid/minimize Publication Bias: the selective publication of manuscripts based on the magnitude, direction, or statistical significance of the study results Negative studies are less likely to be published than studies than positive studies

Eligibility Criteria

Formulate question question eligibility criteria / methodologic criteria a priori hypotheses to explain heterogeneity conduct search review titles and abstracts review full text of possibly eligible articles assess risk of bias, abstract data generate pooled estimates and confidence intervals look for explanations of heterogeneity GRADE evidence profile

Vitamin D & Non-vertebral Fractures

Possible reasons for inter-study variability Technical term for inter-study variability = heterogeneity Possible reasons? Population characteristics… Intervention (dose, mode of administration, etc…) Comparator (placebo, usual practice, etc…) Outcome (definition, severity level, etc…) Risk of Bias (high/medium vs. low)

Formulate question question eligibility criteria / methodologic criteria a priori hypotheses to explain heterogeneity conduct search review titles and abstracts review full text of possibly eligible articles assess risk of bias, abstract data generate pooled estimates and confidence intervals look for explanations of heterogeneity GRADE evidence profile

Conducting search strategy Hope you were awake for the first half

Screening likely be many citations identified through the searches 2 independent reviewers Need to achieve consensus when the 2 reviewers disagree discussion or third reviewer

Formulate question question eligibility criteria / methodologic criteria a priori hypotheses to explain heterogeneity conduct search review titles and abstracts review full text of possibly eligible articles assess risk of bias, abstract data generate pooled estimates and confidence intervals look for explanations of heterogeneity GRADE evidence profile

Full Text Review Detailed review to establish relevance definitively Apply same criteria (PICO and design) 2 independent reviewers Need to achieve consensus when the 2 reviewers disagree discussion or third reviewer

Formulate question question eligibility criteria / methodologic criteria a priori hypotheses to explain heterogeneity conduct search review titles and abstracts review full text of possibly eligible articles assess risk of bias, abstract data generate pooled estimates and confidence intervals look for explanations of heterogeneity GRADE evidence profile

Data Extraction Data extraction by two independent reviewers ONLY done for eligible included Demographic characteristics of populations Population, intervention, comparison Risk of bias Outcomes

Risk of Bias (RCTs) 1.) Was allocation sequence adequately generated? 2.) Was allocation adequately concealed? 3.a) Were patients blinded? 3.b) Were healthcare providers blinded? 3.c) Were data collectors blinded? 3.d) Were outcome assessors blinded? 3.e) Were data analysts blinded? 4.) Was loss to follow-up (missing outcome data) infrequent? 5.) Are reports of the study free of suggestion of selective outcome reporting?

Risk of Bias (Observational Studies) cohorts drawn from same population measurement of exposure measurement prognostic factors adjustment for prognostic factors assessment of outcome completeness of follow-up

Formulate question question eligibility criteria / methodologic criteria a priori hypotheses to explain heterogeneity conduct search review titles and abstracts review full text of possibly eligible articles assess risk of bias, abstract data generate pooled estimates and confidence intervals look for explanations of heterogeneity GRADE evidence profile

Meta-analysis Combine results of similar studies quantitatively Produces summary statistics that represents different studies Summary statistics more precise – combine sample size

Meta-analysis When not to meta-analyze High clinical heterogeneity Severe vs. mild disease Different lengths of follow-up Varied interventions Methodological heterogeneity is too high High vs. low risk of bias studies

Meta-analysis Things to consider before starting: A priori hypotheses (sub-grouping of studies) Types of comparisons Treatment A vs placebo Treatment B vs Treatment A Treatment C vs Treatment A Types of outcomes Dichotomous Continuous

Two models for pooling Random effects vs. Fixed effect

Two models for pooling Train 50 teachers Questions one could ask 10 classes each over next year randomize 5 to old and new curricula Questions one could ask among these 50 teachers, what is the relative impact of the two curricula? among a wider group of teachers of whom these 50 are a random sample, what is the relative impact of the two curricula? Assumptions one could make relative impact same in all teachers relative impact differs across teachers

Two models for pooling differences question: these teachers versus all teachers assumption: effect same or different substitute studies for teachers and you have the question and assumption for fixed and random effects

Two models for pooling Fixed Random Conceptual Estimates effect in this sample of studies Effects same in all studies Estimates effect in population of which studies are random sample Effects differ– what is the average effect Statistical Error variance only within study Error variance within and between Practical Narrow CI weight large vs small ↑↑↑ Wider CI weight large vs small ↑

Fixed: CI same in the face of small or large variability

Random: CI wider in large variability than in small variability

Lots of variability: Random CI wider than fixed CI Fixed Random

Fixed Random Lots of variability and small studies have different estimates than large studies: Random CI wider than fixed CI and point estimate of random close to small studies Fixed Random

What to choose? Random uncertainty does increase with variability CIs should capture that argues for random effect we always use, unless…. dominating trial(s) with results discrepant with small trials

Quality (Strength) of Evidence The GRADE method Grading of Recommendations Assessment Development and Evaluation Classifies evidence as (very low, low, moderate, and high) Based on classification on 5 required Quality of studies Consistency of studies Directness of evidence to clinical question Precision of results Publication bias And four non-required domains Dose-response association Plausible confounding Strength of association

What are we grading? Two components quality of body of evidence confidence in estimate of effect high, moderate, low, very low RCTs start high observational studies start low strength of recommendation strong and weak

Determinants of Quality What can lower quality? Risk of Bias Inconsistency Indirectness Imprecision Publication bias

Risk of bias

Risk of bias

Risk of bias

Risk of Bias Statistically significant difference between pooled effect estimate of high and low risk of bias studies: Consider only reporting on low risk of bias studies Not significantly different, JUST a RISK of bias. If all studies high risk of bias, rate down!!

Consistency of Results

Consistency of Results

What Criteria Were You Using? Similarity of point estimates less similar, more inclined to say “heterogeneity” Overlap of confidence intervals less overlap, more inclined to say “heterogeneity”

Homogenous test for heterogeneity what is the p-value? what is the null hypothesis for the test for heterogeneity? Ho: RR1 = RR2 = RR3 = RR4 p=0.99 for heterogeneity

Heterogeneous test for heterogeneity what is the p-value? p-value for heterogeneity < 0.001

Homogenous What is the I2 ? p=0.99 for heterogeneity I2=0%

Heterogeneous What is the I2 ? I2=89% p-value for heterogeneity < 0.001 I2=89%

Consistency of Results If inconsistency, look for explanation patients, intervention, outcome, methods Judgment of consistency variation in size of effect overlap in confidence intervals statistical significance of heterogeneity I2

Directness of Evidence Differences in patients (age, sex, ethnicity, condition) Differences in interventions (dose, class) Differences in outcomes (health-related quality of life, functional capacity, radiological)

Imprecision Small sample size Wide confidence intervals small number of events Wide confidence intervals uncertainty about magnitude of effect

Imprecision Necessary to use absolute risks as opposed to relative risks Relative risk of 1.5 (CI 1.2 to 1.8) – Drug A vs Placebo Baseline risk for Placebo = 5% - becomes (6% to 9%) Baseline risk for Placebo = 30% - becomes (36% to 54%)

Imprecision 10% 0%

Publication Bias High likelihood could lower quality Reporting of outcomes Reporting of studies publication bias number of small studies industry sponsored

Beta blockers in non-cardiac surgery Quality Assessment Summary of Findings Quality Relative Effect (95% CI) Absolute risk difference Outcome Number of participants (studies) Risk of Bias Consistency Directness Precision Publication Bias Myocardial infarction 10,125 (9) No serious limitations No serious imitations Not detected High 0.71 (0.57 to 0.86) 1.5% fewer (0.7% fewer to 2.1% fewer) Mortality 10,205 (7) Possiblly inconsistent Imprecise Moderate or low 1.23 (0.98 – 1.55) 0.5% more (0.1% fewer to 1.3% more) Stroke 10,889 (5) No serious limitaions Possible imprecision Moderate or High 2.21 (1.37 – 3.55) (0.2% more to 1.3% more0

Questions/Discussion Thank you! Special thanks to Dr. Gordon Guyatt