Presentation is loading. Please wait.

Presentation is loading. Please wait.

Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF.

Similar presentations


Presentation on theme: "Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF."— Presentation transcript:

1 Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF

2 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

3 Juni et al, Hazards of scoring the quality of clinical trials. JAMA. 1999;282:

4 Why conduct a systematic review?  The best way to summarize evidence on a scientific topic  Concisely communicates findings to others in the field  Identifies author(s) as experts  Identifies areas for future study  Perfect for background of grants  Don’t need to do primary data collection (so can be done while waiting for data from other projects)  You have to do the work anyway, so might as well get a publication!  You can effect change in clinical management

5 Cumulative Meta-analysis Antman EM et al: JAMA. 1992;268:

6 Systematic Review: Clinical Implications (Antiarrhythmic Drugs for Acute MI) Teo KK et al. JAMA. 1993;270:

7 Sample Systematic Reviews  Kangelaris KN, Bent S, Nussbaum RL, Garcia DA, Tice JA. Genetic testing before anticoagulation? A systematic review of the safety and efficacy of pharmacogenetic dosing of warfarin. Journal of General Internal Medicine (in press).  Nguyen SP, Bent S, Chen Y, Terdiman JP. Gender as a Risk Factor for Advanced Neoplasia and Colorectal Cancer: A Systematic Review and Meta-analysis. Clinical Gastroenterology. 2009;7:  Simon J, Chen Y, Bent S. The relation of alpha-linoleic acid to the risk of prostate cancer: a systematic review. Am J Clin Nutr. 2009;89:1-7S.  Li J, Winston LG, Moore DH, Bent S. Efficacy of short-course antibiotic regimens for community-acquired pneumonia: a meta- analysis. American Journal of Medicine. 2007;120(9):  Margaretten M, Kohlwes J, Moore D, Bent S. The rational clinical examination: does this patient have septic arthritis. JAMA. 2007;297:

8 Sample Systematic Reviews Hsu J, Kohlwes J, Bent S. Efficacy of antifungal therapy in chronic rhinosinusitis: A systematic review. J Allergy Clin Immunol :2 Guarnieri M, Bent S. Death from coronary artery disease in patients with systemic lupus erythematosus: a systematic review and meta-analysis of mortality cohort studies. (submitted to Arthritis Care and Research 1/2012). Lee JK, Liles EG, Bent S, Levin TR, Corley DA. Diagnostic Accuracy of Fecal Immunochemical Tests for Colorectal Cancer: Systematic Review and Meta-analysis (submitted to JAMA 4/2013).

9 8 Steps of Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

10 Create a Summary Measure  Before we get to the mechanics of a summary measure….  Be sure to provide your audience with a concise summary table  A “visual meta-analysis”  Readers should be able to examine Table 1 and reach their own conclusions about the data

11 Example  Antibiotics for acute bronchitis.  After search and application of inclusion/exclusion criteria, 8 studies were included.

12 RCTs in Acute Bronchitis Study, yrNAbxOutcome Result * Stott, 76207DoxyDays of Yellow Spit0.6 (-0.2 to 1.4) Franks, 8454TMP/SCough Amount Score0.2 (-0.2 to 0.6) Williamson, 8469DoxyDays of Purulent Sputum-0.2 (-1.2 to 0.8) Dunlay, 8745ErythroSputum production score0.5 (0.1 to 0.9) Scherl, 8731DoxyDays of sputum1.9 (-0.2 to 4.0) Verheij, 94140DoxyDays of productive cough0.5 (-0.4 to 1.4) Hueston, 9423ErythroDays of productive cough-0.4 (-2.4 to 1.6) King, 9691ErythroDays of sputum production0.7 (-1.3 to 2.7) * Positive numbers indicate antibiotics are superior to placebo

13

14 How do you create a summary measure?  Clinical example: 5 year old girl presents with ear pain and is found to have an acute otitis media.  Should she get antibiotics? Research Questions: 1.In children with OM, are antibiotics effective for pain relief? 2.In children with OM, do antibiotics reduce the rate of complications (mastoiditis, hearing problems)?

15 3 studies are identified (examining effect of Abx on Pain)  Study 1: N = 100RR=1.41  Study 2: N=200RR=0.98  Study 3: N=300RR=1.01  You could take the average effect: ( ) / 3 = 1.13  Is this a good summary measure?

16 Summary measure weighted by sample size  Provide “weight” for studies based on their sample size 600Total RRNStudy summary effect estimate= Σ (N i x effect estimate i ) = 640 =1.07 Σ(N i ) 600

17 More refined: Provide “weight” by using inverse of variance Summary = Σ (weight i x effect estimate i ) = 30.5 = 1.00 effect estimate Σ(weight i ) 30.3 StudyNRRVar RRWeight Total700

18 Does the largest study always have the lowest variance and therefore the greatest “weight”?  Dichotomous outcomes  Continuous outcomes

19 Confidence Intervals Around Summary Effect  Calculate variance of summary effect estimate, or the 95% CI around the summary estimate Variance of summary estimate = 1 Σ(weights i ) Variance of summary estimate = _1_ = % CI = √0.03 = Summary OR and 95% CI = 1.00 ( )

20 Type of Model? Variance RR s = 1/  w i Weight i = 1 variance RR i + D Weight i = 1 variance RR i Variance of individual studies + variance of differences between studies Weights: variance of individual studies Existing studies are a random sample Existing studies are the entire population Goal: estimate the “true” effect Goal: weighted average of risk from existing studies Random EffectsFixed Effects

21 Formulas for D

22 Fixed Effects Model: Random Effects Model: Summary RR b Summary RR a b a

23 Random VS. Fixed Effects Model Practical Implications of the Choice  Confidence intervals: RE model produces wider confidence intervals  Statistical significance: less likely with RE model BOTTOM LINE:  If the individual study findings are similar, the model makes little difference in estimate or statistical significance.  If the individual study findings are heterogeneous, the model can affect the statistical significance.

24 Mantel-Haenszel Method (Fixed Effects Model) DiseasedNot diseased Treated (exposed) a i c i Not treated (unexposed) b i d i OR i = a i / c i = a i x d i lnOR mh = Σ (w i x lnOR i ) b i / d i b i x c i Σw i variance lnOR i = variance OR mh = 1 a i b i c i d i Σ w i weight i = (w i ) = 1 variance lnOR i 95% CI = e lnORmh  (1.96 x √variance lnOR mh )

25 Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo Calculate OR and lnOR for each study: OR 1 =1 x 116 = 0.34lnOR 1 = x 114 OR 2 = 7 x 65 = 0.58lnOR 2 = x 65

26 Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 2. Calculate variance lnOR i for each study: Var ln OR 1 = = Var ln OR 2 = = Calculate w i for each study: w 1 = 1 = w 2 = 1 =

27 Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo Calculate the w i x ln OR i for each study: w 1 x lnOR 1 = 0.74 x = w 2 x lnOR 2 = 3.85 x = Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

28 5. Calculate the sum of the w i w 1 + w 2 = = Summary lnOR mh = Σ (w i x lnOR i ) = = Σ w i 4.59 = OR mh = Calculate variance OR mh = 1 = 1 = 0.22 Σ w i Calculate 95% CI = e lnORmh + (1.96 x √ variance lnORmh) = e (1.96 x √ 0.22) = Summary OR = 0.53 (95% CI 0.21 – 1.34) Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

29 Dersimonian and Laird Method (Random Effects Model) Similar formula to Mantel-Haenszel: ln OR dl = Σ (w i x ln OR i ) w i = 1 Σw i variance i + D  Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

30 But…All you need to know is:  When combined, individual study effect estimates are weighted by their inverse variance  Variance is related to sample size AND # of events (dichotomous) and precision (continuous)  Fixed effects just combines all weighted estimates, while random effects “penalizes” estimates for variation between studies

31 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

32 Heterogeneity  Are you comparing apples and oranges?  Clinical heterogeneity: are studies asking same question?  Statistical heterogeneity: is the variation likely to have occurred by chance?  Measures how far each individual OR/RR is from the summary OR/RR.  Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

33 Heterogeneity  Refers to the degree that the study results differ  Visual Approach  Statistical Approach Q =sum [weight i x (ES s – ES i )] p < 0.05 indicates heterogeneity

34 Summary RR = 0.93 ( )

35 Problem of Heterogeneity  Study findings are different – should they be combined? StudyOR StudyOR

36 Statistical tests of Heterogeneity  Is the variation in the individual study findings likely due to chance? H o : Effect estimate in each study is the same (or homogeneous) H a : Effect estimate in each study is not the same (or heterogeneous) Q = Σ(w i x (ln OR mh – ln OR i ) 2 ) df = (N studies -1) p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous

37 Heterogeneity – Interpret Findings (Example: RR of Colon CA, Men vs. Women)

38 8 Steps to Sytematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

39 Assessing for Publication Bias  Publication Bias – the publication or “non-publication” of research findings, depending on the nature and direction of the results.  Rosenthal, 1979 – published an article describing the “file-drawer problem” that journals publish only 5% of all negative studies, while the file drawers in the back of the lab contain the other 95%.

40 Methods for Assessing Publication Bias  Funnel plots – simple scatter plots of treatment effects (horizontal axis) vs. some measure of study size (vertical axis).  Choice of axes –Log scale for treatment effects (to ensure that treatment effects in opposite directions are the same distance from 1.0 – e.g., 0.5 and 2.0) –Standard error for measure of sample size Power depends on both sample size and # events Standard error is consistent with the statistical tests

41

42 Funnel Plot of Log Relative Risk vs Standard Error Log Relative Risk Standard error

43 Example: ALA and Prostate Cancer Risk RR=1.2 (1.01 to 1.43), Test for heterogeneity, p=0.00

44 ALA – Funnel Plot

45 Funnel Plot with Imputed Values for Publication Bias RR=0.94, 95% CI:

46 Publication bias caveat  Funnel plot asymmetry does not always indicate bias –It is possible that smaller studies enrolled higher risk patients, for example, and therefore found a greater effect. –Small studies are often conducted before larger studies. In the intervening years, other interventions may have improved, thus reducing the relative efficacy of the treatment.

47 Statistical methods to assess publication bias  Examine associations between study size and treatment effect. –Sensitivity is poor when < 20 studies  Begg’s test: an adjusted rank correlation  Egger’s test: a weighted regression of effect size vs. standard error. –Basically asks if the regression line has a non-zero slope –More sensitive than Begg’s test, but more false positives, especially when 1) large treatment effects, 2) few events per trial, 3) all trials of similar size. (In these cases, one may decide a priori to use Begg’s test).

48

49 Begg's Test adj. Kendall's Score (P-Q) = -30 Std. Dev. of Score = Number of Studies = 12 z = Pr > |z| = z = 1.99 (continuity corrected) Pr > |z| = (continuity corrected) Egger's test Std_Eff | Coef. Std. Err. t P>|t| [95% Conf. Interval] slope | bias |

50 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

51 Subgroup & Sensitivity Analysis  Subgroup Analysis – MA of a subgroup of eligible studies  age  ethnicity  risk factors  treatment  Sensitivity Analysis – add or delete questionable studies  eligibility  treatment

52 Subgroup Analysis OR95% CIN Ever user Of estrogen: All eligible studies Cohort studies Case-Control studies 2.3* 1.7* 2.4* Dose of estrogen: 0.3 mg mg  1.25 mg Duration of use: < 1 year 1-5 years 5-10 years  10 years * Regimen:Cyclic Daily 3.0* 2.9* * p for heterogeneity < 0.05

53 Analytical Methods: Summary Points  Always start the meta-analysis with a “visual meta-analysis” (i.e., a great table 1). –A clinician should be able to interpret the results  Step 1: Calculate a summary measure = “weighted mean effect estimate” –You can combine anything, but use judgment  Step 2: Assess for heterogeneity –Heterogeneity is not always a problem  Step 3: Assess for publication bias –Both visual and statistical methods  Step 4: Perform subgroup/sensitivity analyses –Ideally specify these a priori

54 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

55 Can you conduct a systematic review when there are only a few studies?

56 Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials  Objectives To determine whether parachutes are effective in preventing major trauma related to gravitational challenge.  Design Systematic review of randomised controlled trials.  Data sources: Medline,Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists.  Study selection: Studies showing the effects of using a parachute during free fall.  Main outcome measure Death or major trauma.  Results We were unable to identify any randomised controlled trials of parachute intervention.  Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute. Smith GCS and Pill JP. BMJ 2003;327:1459–61

57 Advanced Topics  Individual participant data  Missing data  Different types of data  Observational studies  Generalized synthesis of evidence  Meta-regression  Critique of a systematic review

58 Different types of data  Different scales (example)  Ordinal data  Binary data  Continuous outcomes  Diagnostic tests (example)

59 RCTs in Acute Bronchitis: Different Scales Study, yr NAbxOutcomeResult Stott, Doxy Days of Yellow Spit 0.6 (-0.2 to 1.4) Franks, 84 54TMP/S Cough Amount Score 0.2 (-0.2 to 0.6) Williamson, 84 69Doxy Days of Purulent Sputum -0.2 (-1.2 to 0.8) Dunlay, 87 45Erythro Sputum production score 0.5 (0.1 to 0.9) Scherl, 87 31Doxy Days of sputum 1.9 (-0.2 to 4.0) Verheij, Doxy Days of productive cough 0.5 (-0.4 to 1.4) Hueston, 94 23Erythro Days of productive cough -0.4 (-2.4 to 1.6) King, 96 91Erythro Days of sputum production 0.7 (-1.3 to 2.7)

60 Problem  How do you combine studies with slightly different outcomes?  Option 1: - don’t do it  Option 2: Transform all outcomes to an effect size

61 What is an Effect Size?  Effect size – a way of expressing results in a common metric  Units – standard deviation

62 Effect Size  ES = X 1 – X 2 SD pooled 1.ES increases as difference between means increases 2.ES increases as SD decreases 3.ES is expressed in units of SD 4.Summary ES combines the weighted ES from each study.

63 Effect Size

64  Rough Estimates –SMALL0.2 –MEDIUM0.5 –LARGE>0.7  Context –Mean Duration of Cough = 8 days –Standard Deviation = 3 days

65

66 Main Result  Summary ES = 0.21 (95% CI 0.05 to 0.36)

67 Summary Mean Differences Outcome Measure Summary Mean Difference (95% CI) Days of Productive Cough (6 studies) 0.4 days (-0.1 to 0.8) Days of cough (4 studies) 0.5 days (-0.1 to 1.1) Time off work (6 studies) 0.3 days (-0.6 to 1.1)

68 Different Types of Data: Diagnostic Tests

69 Sensitivity and Specificity Sensitivity TP/(TP + FN) Positive in Disease Specificity TN/(TN + FP) Negative in Health TNFNTest - FPTPTest + Disease - Disease +

70 (+) Likelihood Ratio = Sensitivity 1-Specificity (-) Likelihood Ratio = 1-Sensitivity Specificity

71 Does this patient have a specific disease?  What we thought before (pre-test probability)  + Clinical information (diagnostic test, LR)  = What we think after (post-test probability)

72 Diagnostic OR = +LR/-LR = TP x TN / FP x FN

73 Example: US and CT for Appendicitis  Goal: to determine whether US or CT is a “better” test for the evaluation of suspected appendicitis.  Diagnostic tests are complicated because there are 5 potential outcomes to summarize –LR+, LR- –Sensitivity, Specificity –Diagnostic OR –Assess heterogeneity, publication bias for EACH outcome

74

75

76

77

78 Advanced Topics  Individual participant data  Missing data  Different types of data  Observational studies  Generalized synthesis of evidence  Meta-regression  Critique of a systematic review

79 Meta-regression  Examines whether the study effects (outcomes) are related to one or more of the study characteristics.  Can be used to understand/explain heterogeneity.  Can be thought of as an epidemiological study of the trials or studies.

80 Clinical Questions: Meta-Regression Are there certain situations in which BCG may be more effective for preventing TB?

81 Meta-regression: example StudyOR 95% CI , , , , , , , , , , , , ,  BCG vaccine: used to prevent tuberculosis  Odds ratio estimates from 13 trials (right)  Scientists have suggested that effects may be related to geographic latitude

82 Funnel Plot

83 Funnel Plot – Organized by Latitude

84 Meta-regression: example, continued Log odds ratio versus absolute latitude :

85 Meta-regression: example, cont Same plot, showing precision:

86 Meta-regression: example, cont Same plot, with fitted (meta-)regression line:

87 Meta-regression: example, cont  Is the slope of the line significantly different from 0?  If yes, we conclude that the study effects are in fact related to latitude

88 Meta-regression: details  In a regression model for the data: each study represents one observation  Weights equal to the study precision  Random effects model (recommended)  Built-in function in Stata: ‘metareg’

89 Critique of a Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

90 Reviewing Journal Articles  Very little formal teaching  “Because reviews are often highly negative, the new researcher implicitly learns from the negative reviews received on his or her own submitted papers that reviews are supposed to be negative. It is as if the implicit message is: A reviewer’s job is to criticize the manuscript.”

91 12 Tips on Reviewing Articles  1. Know your mission  2. Be speedy  3. Read carefully  4. Say positive things in your review  5. Don’t exhibit hostility  6. Keep it brief  7. Don’t nitpick  8. Develop your own style  9. Be careful in recommending further experiments  10. Watch for egocentrism  11. Make a recommendation  12. Sign your review

92 Conclusions  You can combine almost anything  Use clinical judgment to guide you in deciding how and whether to combine studies.  Remember the main mission of a systematic review: to summarize a body of literature in a concise and clear way.  Get statistical input as needed.


Download ppt "Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF."

Similar presentations


Ads by Google