1. Systematic Review: Meta-analysis II The nuts and bolts of the statistics Alka M. Kanaya, M.D. Assistant Professor of Medicine, Epi/Biostats April 19, 2007

2. Goals 1.Understand statistical issues for MA  summary estimate and variance models methods  heterogeneity  publication bias 2.Carry on an intelligent conversation with your statistician 3. Know if published MA used appropriate methods

3. Meta-analysis: the Steps 1.Formulate a question, eligibility criteria 2.Perform a systematic literature search 3.Abstract the data 4.Perform a statistical analysis 5.Calculate the summary effect size 6.Calculate the summary effect size for subgroups 7.Check for heterogeneity 8.Check for publication bias

4. Clinical Case  5 y.o. girl c/o ear pain and is found to have an acute otitis media.  Should she get antibiotics? Research Questions: 1.Are antibiotics effective for pain relief in children with acute OM? 2.Do antibiotics reduce complications of OM (mastoiditis, hearing problems)?

5. Systematic Review part  Inclusion criteria: –RCT of antibiotic vs. placebo –Children –Without tympanostomy tubes –With OM (regardless of setting of recruitment) –Patient-relevant outcomes  8 trials with total of 2,287 kids Glasziou, Cochrane Review, 2005

6. Goal #1: “Best” estimate  Combine findings from several studies to get the "best" estimate  Calculate weighted mean effect estimate, or a summary effect estimate 700Total 1.013003 0.982002 1.411001 RRNStudy Do antibiotics reduce pain? 3 RCTs: summary effect estimate= Σ (N i x effect estimate i ) = 640 =0.914 Σ(N i ) 700

7. Goal #1: Calculate weighted mean effect estimate Summary = Σ (weight i x effect estimate i ) = 30.3 = 0.99 effect estimate Σ(weight i ) 30.5 StudyNRRVar RRWeight 11001.413.00.33 22000.980.110 33001.010.0520 Total700

8. Goal #2: Determine if the summary effect is significant  Calculate variance of summary effect estimate, or the 95% CI around the summary estimate Variance of summary estimate = 1 Σ(weights i ) Variance of summary estimate = _1_ =.03 30.5 95% CI = + 1.96 √0.03 = + 0.34 Summary OR and 95% CI = 0.99 (0.65 - 1.33)

9. Type of Model? Variance RR s = 1/  w i Weight i = 1 + D variance RR i Weight i = 1 variance RR i Variance of individual studies + variance of differences between studies Weights: variance of individual studies Existing studies are a random sample Existing studies are the entire population Goal: estimate the “true” effect Goal: weighted average of risk from existing studies Random EffectsFixed Effects

10. Model: Random Effects Model: Summary RR b Summary RR a b a

11. Random VS. Fixed Effects Models Practical Implications of the Choice  Summary estimates: usually similar  Variance: RE model produces large variance of the summary estimate  Confidence intervals: RE model produces wider confidence intervals  Statistical significance: less likely with RE model BOTTOM LINE:  If the individual study findings are similar, the model makes little difference in estimate or statistical significance.  If the individual study findings are heterogeneous, the model can affect the statistical significance.

12. Which method? ModelMethodEffect estimate Data Fixed Effects: Mantel- Haenszel Ratio (OR, RR)Crude (2x2) General Variance-based Ratio (OR, RR) Difference (risk, rate) Adjusted ratio & CI Crude (2x2) Random Effects: DerSimonian & Laird Ratio, difference Crude (2x2) General Variance-based Ratio (OR, RR) Difference (risk, rate) Adjusted ratio & CI Crude (2x2)

13. Which method to use? Types of Studies in Meta-analysis Method to Use Randomized trials: Any method (usually Mantel-Haenszel or DerSimonian & Laird Observational studies: General Variance Based

14. Mantel-Haenszel Method (Fixed Effects Model) DiseasedNot diseased Treated (exposed) a i c i Not treated (unexposed) b i d i OR i = a i / c i = a i x d i lnOR mh = Σ (w i x lnOR i ) b i / d i b i x c i Σw i variance lnOR i = 1 + 1 + 1+ 1 variance OR mh = 1 a i b i c i d i Σ w i weight i = (w i ) = 1 variance lnOR i 95% CI = e lnORmh  (1.96 x √variance lnOR mh )

15. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 1. Calculate OR i for each study: OR 1 =1 x 116 = 0.34lnOR 1 = -1.08 3 x 114 OR 2 = _______ = 0.58lnOR 2 = -0.54

16. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 2. Calculate variance lnOR i for each study: Var ln OR 1 = 1 + 1 + 1 + 1 = 1.35 1 3 114 116 Var ln OR 2 = ______________ = 0.26 3. Calculate w i for each study: w 1 = 1 = 0.74 1.35 w 2 = ________ = 3.85

17. Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 4.Calculate the w i x ln OR i for each study: w 1 x lnOR 1 = 0.74 x -1.08 = -0.80 w 2 x lnOR 2 = 3.85 x -0.54 = -2.08 Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

18. 5. Calculate the sum of the w i w 1 + w 2 = 0.74 + 3.85 = 4.59 6.Calculate lnOR mh = Σ (w i x lnOR i ) = -0.80 + -2.08 = -0.63 Σ w i 4.59 = OR mh = 0.53 7.Calculate variance OR mh = 1 = 1 = 0.22 Σ w i 4.59 8.Calculate 95% CI = e lnORmh + (1.96 x √ variance lnORmh) = e -.63 + (1.96 x √ 0.22) = 0.21 - 1.34 Summary OR = 0.53 (95% CI 0.21 – 1.34) Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

19. Dersimonian and Laird Method (Random Effects Model) Similar formula to Mantel-Haenszel: ln OR dl = Σ (w i x ln OR i ) w i = 1 Σw i variance i + D  Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

20. General Variance-Based Method (Fixed or Random Effects) Confidence Intervals: ln OR s = Σ (w i x ln OR i )w i = 1 ______ Σw i variance lnOR i (+D) Variance lnOR i = ln OR i / OR l 2 or ln OR u / OR i 2 1.96 1.96 Where OR i = OR on i th study OR l = lower bound of 95 % CI for i th study OR u = upper bound of 95 % CI for i th study  Should always be used for MA of Observational studies  Uses adjusted effect estimates  Preserves adjustment for confounding

21. Choice of Model and Method in Meta-Analysis What type of studies are you summarizing? Randomized TrialsObservational Studies  Either Model  Any Method  Either Model  Confidence Interval Method

22. Heterogeneity  Are you comparing apples and oranges?  Clinical heterogeneity: are studies asking same question?  Statistical heterogeneity: is the variation likely to have occurred by chance?  Measures how different each individual OR/RR is from the summary OR/RR.  Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

23. Problem of Heterogeneity  Study findings are different and should not be combined StudyOR 10.01 21.0 310.0 StudyOR 10.35 20.56 30.97 41.15 51.75 61.95

24. Statistical tests of Homogeneity  Is the variation in the individual study findings likely due to chance? H o : Effect estimate in each study is the same (or homogeneous) H a : Effect estimate in each study is not the same (or heterogeneous) Q = Σ(w i x (ln OR mh – ln OR i ) 2 ) df = (N studies -1) p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous  8 trials of antibiotics vs. P o for OM, pain outcome:  Q for homogeneity: p=0.91

25. Subgroup & Sensitivity Analysis  Subgroup Analysis – MA of a subgroup of eligible studies  age  ethnicity  risk factors  treatment  Sensitivity Analysis – add or delete questionable studies  eligibility  treatment

26. Subgroup Analysis OR95% CIN Ever user Of estrogen: All eligible studies Cohort studies Case-Control studies 2.3* 1.7* 2.4* 2.1 - 2.5 1.3 - 2.1 2.2 - 2.6 29 4 25 Dose of estrogen: 0.3 mg 0.625 mg  1.25 mg 3.9 3.4 5.8 1.6 - 9.5 2.0 - 5.6 4.5 - 7.5 349349 Duration of use: < 1 year 1-5 years 5-10 years  10 years 1.4 2.8 5.9 9.5* 1.0 - 1.8 2.3 - 3.5 4.7 - 7.5 7.4 - 12.3 9 12 10 Regimen:Cyclic Daily 3.0* 2.9* 2.4 - 3.8 2.2 - 3.8 8888 * p for homogeneity < 0.05

27. Subgroup Analysis Antibiotics vs. Placebo for acute OM Outcome: abnormal tympanometry Study0 – 1 mo f/u RR (95% CI) 2 – 3 mo f/u RR (95% CI) Appelman, 19910.57 (0.25 – 1.26)NA Burke, 19911.07 (0.62 – 1.84)0.58 (0.31 – 1.08) Mygind, 19810.98 (0.49 – 1.94)1.09 (0.52 – 2.31) Summary estimate : 0.91 (0.62 – 1.32)0.75 (0.47 – 1.21) p-for-heterogeneity: 0.420.20

28. Sensitivity Analyses Performed to test the robustness of the findings To fairly assess and acknowledge the limitations Address publication bias (funnel plots, number needed to change result, etc..)

29. Sensitivity Analysis Aspirin + Heparin vs. Aspirin alone for Unstable Angina MI or Death Study (reference)AspirinAspirin + heparinRR (95% CI) Theroux et al, 1988 10 4/121 (3%)2/122 (2%)0.50 (0.18-2.66) RISC Group, 1990 7 7/189 (4%)3/210 (1%)0.39 (0.18-1.47) Cohen et al, 1990 11 1/32 (3%)0/37 (0%)0.29 (0.06-6.87) Cohen et al, 1994 12 9/109 (8%)4/105 (4%)0.46 (0.24-1.45) Holdright et al, 1994 13 40/131 (31%)42/154 (27%)0.89 (0.66-1.29) Gurfinkel et al, 1995 14 7/73 (10%)4/70 (6%)0.60 (0.29-1.95) Total/Summary:68/655 (10%)55/698 (8%)0.67 (0.44-1/02)  Remove Holdright: RR s = 0.45 (95% CI 0.23 -0.89) ; p-for-hetero=0.71  Add data from two additional trials of LMWH: RR s = 0.56 (95% CI 0.40-0.80); p for heterogeneity: 0.52  Fixed effects model, Mantel-Haenszel method = same findings

30. Publication Bias  Published studies may not be representative of all studies ever conducted.  Selective publication of studies based on strength & direction of results & language.  AKA “positive outcome bias”

31. Minimizing Publication Bias Search bibliographies of published papers Consult with experts Search for unpublished data Clinical Trial Registries (NIH, VA) Institutional Review Boards Pharmaceutical companies Hand searches Consider studies not published in English Stern, BMJ, 2001

32. Statistical Approaches to Publication Bias  Correlation between study sample size (or weight or variance) and effect estimate  Funnel plot  Other fancy statistical methods: estimate number of unpublished studies that must exist to invalidate the results of the meta- analysis. “File drawer” “Fail-safe N” eliminate the studies that may have been published due to bias

33. Association of Estrogen use and Endometrial Cancer Correlation of sample size and RR: rho = 0.68; p = 0.08 FUNNEL GRAPH Relative Risk of Endometrial Cancer

34. RCTs of Heparin plus ASA vs. ASA Correlation of sample size and RR: rho = 0.25; p= 0.64 FUNNEL GRAPH Relative Risk for MI or Death Favors Heparin+ASA Favors ASA Sample Size

35. Presentation of the Results Tables:  Study Characteristics  population  sample size  definition of intervention  definition of the outcome  important design features (validity of the data) - randomization - blinding - follow-up - compliance  Study Findings  main and secondary outcomes  outcomes by subgroup  sensitivity analysis findings

36. Table 1. Characteristics of 6 randomized trials of aspirin + heparin vs. aspirin alone to prevent MI and death in patients admitted with unstable angina Study (ref.)BlindingAspirin DoseGoal PTTDuration of Heparin Theroux, 1988 10 Participants & Investigators 325 mg twice per day 1.5-2 x normal6 days RISC Group, 1990 7 None75 mg dailyNot stated5 days Cohen, 1990 11 None 80/325 mg daily* 2 x normal3-4 days Cohen, 1994 12 Participants162.5 mg daily2 x normal3-4 days Holdright, 1994 13 Participants150 mg daily1.5-2 x normal2 days Gurfinkel, 1995 14 Participants & Investigators 200 mg daily2 x normal5-7 days

37. Bent, AMJ, 1999 No Caption Found Antibiotics vs. Placebo in Acute Bronchitis

38. Take Home Messages  You can do a meta-analysis  Good start on becoming an expert in your field  Your work should be reproducible  Your conclusions should be obvious  Include a statistician on you team