Systematic Review Lecture 3: Advanced Topics Stephen Bent, MD Assistant Professor of Medicine, UCSF Course Director, Systematic Reviews (EPI 214)

8 Steps to Systematic Review 1. Research Question 1. Research Question 2. Protocol 2. Protocol 3. Search 3. Search 4. Study selection (inclusion/exclusion) 4. Study selection (inclusion/exclusion) 5. Quality assessment 5. Quality assessment 6. Data abstraction 6. Data abstraction 7. Analysis 7. Analysis A) Create summary measure A) Create summary measure B) Assess for heterogeneity B) Assess for heterogeneity C) Assess for publication bias C) Assess for publication bias D) Conduct sensitivity/subgroup analyses D) Conduct sensitivity/subgroup analyses E) Advanced issues/techniques E) Advanced issues/techniques 8. Interpretation 8. Interpretation

Bottom Line You can combine just about anything. You can combine just about anything. Use judgment to decide whether it is appropriate and clinically meaningful to combine data on a particular topic. Use judgment to decide whether it is appropriate and clinically meaningful to combine data on a particular topic.

Can you conduct a systematic review when there are only a few studies?

Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials Objectives To determine whether parachutes are effective in preventing major trauma related to gravitational challenge. Design Systematic review of randomised controlled trials. Data sources: Medline,Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists. Study selection: Studies showing the effects of using a parachute during free fall. Main outcome measure Death or major trauma. Results We were unable to identify any randomised controlled trials of parachute intervention. Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute. Smith GCS and Pill JP. BMJ 2003;327:1459–61

Overview Individual participant data Individual participant data Missing data Missing data Different types of data Different types of data Observational studies Observational studies Generalized synthesis of evidence Generalized synthesis of evidence Meta-regression Meta-regression Critique of a systematic review Critique of a systematic review

Individual Participant Data Identify all relevant studies and obtain full datasets Identify all relevant studies and obtain full datasets Potential advantages: Potential advantages: Detailed data checking Detailed data checking Ensure appropriateness of analyses Ensure appropriateness of analyses Subgroup analyses Subgroup analyses Survival and other time-to-event analyses Survival and other time-to-event analyses Standardize inclusion/exclusion criteria Standardize inclusion/exclusion criteria

Individual Participant Data Potential disadvantages Potential disadvantages Time Time Cost Cost No studies have yet documented superiority No studies have yet documented superiority “standard meta-analysis estimates of treatment contrasts are identical to the analysis of individual participant data when homogeneity is assumed.” “standard meta-analysis estimates of treatment contrasts are identical to the analysis of individual participant data when homogeneity is assumed.” Conclusion: Conclusion: more evidence needed to justify IPD more evidence needed to justify IPD Start with standard meta-analysis Start with standard meta-analysis

Overview Individual participant data Individual participant data Missing data Missing data Different types of data Different types of data Observational studies Observational studies Generalized synthesis of evidence Generalized synthesis of evidence Meta-regression Meta-regression Survival data Survival data

Missing Data 3 Types to consider 3 Types to consider Whole studies (publication bias) Whole studies (publication bias) Data missing at the study level Data missing at the study level Missing study effect sizes (the actual magnitude of effect may not be shown, especially when negative). Missing study effect sizes (the actual magnitude of effect may not be shown, especially when negative). Missing measures of precision (no SD, for example) Missing measures of precision (no SD, for example) Missing information on covariates Missing information on covariates Data missing at the individual patient level (when doing meta-analysis of individual patient data) Data missing at the individual patient level (when doing meta-analysis of individual patient data)

Missing Data General approaches General approaches Analyze only the complete cases Analyze only the complete cases Single-value imputation (missing values replaced with “reasonable” estimates) Single-value imputation (missing values replaced with “reasonable” estimates) Regression imputation (missing variables assumed to be linearly related, and regression used) Regression imputation (missing variables assumed to be linearly related, and regression used) Multiple imputation – multiple values imputed for each missing observation. Multiple imputation – multiple values imputed for each missing observation.

Example: Missing SD’s Meta-analysis of SSRI’s compared to TCA’s Meta-analysis of SSRI’s compared to TCA’s 53 RCT’s identified, all used Hamilton Scale 53 RCT’s identified, all used Hamilton Scale Only 20 provided SD’s Only 20 provided SD’s This would require exclusion of 33 studies! This would require exclusion of 33 studies! The authors pooled the SD’s of the scale for the studies reporting this outcome, and used this as an estimate in studies where it was not reported. The authors pooled the SD’s of the scale for the studies reporting this outcome, and used this as an estimate in studies where it was not reported. Song F et al. Selective serotonin reuptake inhibitors: a meta-analysis of efficacy and Acceptability. BMJ. 1993;306:683-7.

SSRI’s vs. TCA’s Results: Results: Only studies reporting SD’s Only studies reporting SD’s Mean difference = 0.13 (95% CI, to 1.28) Mean difference = 0.13 (95% CI, to 1.28) All studies using pooled SD’s All studies using pooled SD’s Mean difference = 0.15 (95% CI, to 0.89) Mean difference = 0.15 (95% CI, to 0.89)

Another common variation of missing information on SD’s Studies may report mean responses and SD’s at baseline and follow-up Studies may report mean responses and SD’s at baseline and follow-up Often do NOT report the SD of the change (which is required to account for correlation between individuals). Often do NOT report the SD of the change (which is required to account for correlation between individuals). A sensitivity analysis can be conducted, imputing a range of values for this missing correlation. A sensitivity analysis can be conducted, imputing a range of values for this missing correlation.

Weight loss study – no SD of change Treatment Group* Baseline weight in kg (SD) Week 6 weight in kg (SD) Percent Change p-value† Citrus aurantium product 90.9 (17.5)89.5 (16)-1.5%0.05 Placebo 83.6 (17.5)82.7 (18)-1.1%0.10 No-placebo control 78.1 (11.5)77.7 (10.5)-0.05%‡Not reported Colker et al Current Therapeutic Research

Overview Individual participant data Individual participant data Missing data Missing data Different types of data Different types of data Observational studies Observational studies Generalized synthesis of evidence Generalized synthesis of evidence Meta-regression Meta-regression Survival data Survival data

Different types of data Different scales (example) Different scales (example) Ordinal data Ordinal data Binary data Binary data Continuous outcomes Continuous outcomes Diagnostic tests (example) Diagnostic tests (example)

RCTs in Acute Bronchitis: Different Scales Study, yr NAbxOutcomeResult Stott, Doxy Days of Yellow Spit 0.6 (-0.2 to 1.4) Franks, 84 54TMP/S Cough Amount Score 0.2 (-0.2 to 0.6) Williamson, 84 69Doxy Days of Purulent Sputum -0.2 (-1.2 to 0.8) Dunlay, 87 45Erythro Sputum production score 0.5 (0.1 to 0.9) Scherl, 87 31Doxy Days of sputum 1.9 (-0.2 to 4.0) Verheij, Doxy Days of productive cough 0.5 (-0.4 to 1.4) Hueston, 94 23Erythro Days of productive cough -0.4 (-2.4 to 1.6) King, 96 91Erythro Days of sputum production 0.7 (-1.3 to 2.7)

Problem How do you combine studies with slightly different outcomes? How do you combine studies with slightly different outcomes? Option 1: - don’t do it Option 1: - don’t do it Option 2: Transform all outcomes to an effect size Option 2: Transform all outcomes to an effect size

What is an Effect Size? Effect size – a way of expressing results in a common metric Effect size – a way of expressing results in a common metric Units – standard deviation Units – standard deviation

Effect Size ES = X 1 – X 2 ES = X 1 – X 2 SD pooled SD pooled 1. ES increases as difference between means increases 2. ES increases as SD decreases 3. ES is expressed in units of SD 4. Summary ES combines the weighted ES from each study.

Effect Size

Rough Estimates Rough Estimates SMALL0.2 SMALL0.2 MEDIUM0.5 MEDIUM0.5 LARGE>0.7 LARGE>0.7 Context Context Mean Duration of Cough = 8 days Mean Duration of Cough = 8 days Standard Deviation = 3 days Standard Deviation = 3 days

Main Result Summary ES = 0.21 (95% CI 0.05 to 0.36) Summary ES = 0.21 (95% CI 0.05 to 0.36)

Summary Mean Differences Outcome Measure Summary Mean Difference (95% CI) Days of Productive Cough (6 studies) 0.4 days (-0.1 to 0.8) Days of cough (4 studies) 0.5 days (-0.1 to 1.1) Time off work (6 studies) 0.3 days (-0.6 to 1.1)

Different Types of Data: Diagnostic Tests

Sensitivity and Specificity Sensitivity TP/(TP + FN) Positive in Disease Specificity TN/(TN + FP) Negative in Health TNFNTest - FPTPTest + Disease - Disease +

(+) Likelihood Ratio =Sensitivity1-Specificity (-) Likelihood Ratio = 1-Sensitivity Specificity Specificity

Likelihood ratios LR+ = likelihood of + test in patient WITH dz. LR+ = likelihood of + test in patient WITH dz. likelihood of a + test in patient WITHOUT dz. likelihood of a + test in patient WITHOUT dz. LR- = likelihood of a negative test in patient WITH dz. likelihood of a negative test in a patient WITHOUT dz. LR- = likelihood of a negative test in patient WITH dz. likelihood of a negative test in a patient WITHOUT dz.

Does this patient have a specific disease? What we thought before What we thought before (pre-test probability) + Clinical information + Clinical information (diagnostic test, LR) = What we think after = What we think after (post-test probability)

Diagnostic OR = +LR/-LR = TP x TN / FP x FN

Summary Sensitivity and Specificity TP(i) FN(i) FP(i) TN(i) n1n1 n2n2 Summary sensitivity = ∑TP(i) [positives] ∑TP(i)+FN(i) [with disease] ∑TN(i) [negatives] ∑TN(i)+FP(i) [without disease] Summary specificity =

Quality of Studies of Diagnostic Tests A recently evaluated tool assesses the quality of studies of diagnostic tests ( A recently evaluated tool assesses the quality of studies of diagnostic tests (QUADAS - Quality Assessment of Diagnostic Accuracy Studies) In general, you need to assess all 3 parts of the research question: In general, you need to assess all 3 parts of the research question: E.g., in patients with symptom/presentation X, does the use of Y test predict disease Z? E.g., in patients with symptom/presentation X, does the use of Y test predict disease Z? So, are patients well characterized and typical? So, are patients well characterized and typical? Is the test well described? Is the test well described? Is there an adequate gold standard? Is there an adequate gold standard? Do all patients get the gold standard? If not – potential for “verification bias.” Do all patients get the gold standard? If not – potential for “verification bias.”

Example: US and CT for Appendicitis Goal: to determine whether US or CT is a “better” test for the evaluation of suspected appendicitis. Goal: to determine whether US or CT is a “better” test for the evaluation of suspected appendicitis. Diagnostic tests are complicated because there are 5 potential outcomes to summarize Diagnostic tests are complicated because there are 5 potential outcomes to summarize LR+, LR- LR+, LR- Sensitivity, Specificity Sensitivity, Specificity Diagnostic OR Diagnostic OR Assess heterogeneity, publication bias for EACH outcome Assess heterogeneity, publication bias for EACH outcome

Other Systematic Reviews of Diagnostic Tests: The JAMA Rational Clinical Examination Series Very interesting type of review for those interested in the value of the history and physical examination. Very interesting type of review for those interested in the value of the history and physical examination. Both history and physical are diagnostic tests Both history and physical are diagnostic tests Helps define the interview questions and exam maneuvers Helps define the interview questions and exam maneuvers Examples: Examples: Does this woman have a UTI? Does this woman have a UTI? Does this patient with a swollen joint have septic arthritis? Does this patient with a swollen joint have septic arthritis?

Case 1 A 32 y.o. woman leaves a phone message at your office saying she has dysuria and wants to know what to do. A 32 y.o. woman leaves a phone message at your office saying she has dysuria and wants to know what to do.

1 st step: Formulate a differential diagnosis and assign pre-test probabilities. DiagnosisProbability UTI50% STD20% Vaginal Yeast Infxn 20% Bacterial Vaginosis 10% Other < 1%

General rules about LR’s LR+ LR+ +2 – increases post-test prob 15% +5 – increases post-test prob 30% +10 – increases post-test prob 45% LR- LR- 0.5 – decreases post-test prob 15% 0.2 – decreases post-test prob 30% 0.1 – decreases post-test prob 45%

Likelihood ratios for suspected UTI TestLR+LR- Dysuria Frequency1.8(NS) Hematuria2.0(NS) Vag D/C Dipstick Self Diagnosis 4.0 Bent et al. JAMA. 2002;287:

Copyright restrictions may apply. Bent, S. et al. JAMA 2002;287: Proposed Algorithm for Evaluating a Women With Symptoms of Acute Urinary Tract Infection (UTI)

Overview Individual participant data Individual participant data Missing data Missing data Different types of data Different types of data Observational studies Observational studies Generalized synthesis of evidence Generalized synthesis of evidence Meta-regression Meta-regression Survival data Survival data

Meta-analysis of observational studies Traditionally, most meta-analyses were done with RCT’s Traditionally, most meta-analyses were done with RCT’s What are the unique concerns with doing meta- analyses of observational studies? What are the unique concerns with doing meta- analyses of observational studies?

Concerns with Meta-analyses of Observational Studies Potential for greater publication bias Potential for greater publication bias In studies that are published, authors may only report outcomes with the greatest statistical significance. In studies that are published, authors may only report outcomes with the greatest statistical significance. Some studies may present RR’s (cohort), and others OR’s (case-control). Some studies may present RR’s (cohort), and others OR’s (case-control). *Different studies adjust for different potential confounders (age, SES, DM, HTN, Medication, etc). *Different studies adjust for different potential confounders (age, SES, DM, HTN, Medication, etc).

Dealing with variability in observational studies Many different methods are available Many different methods are available For example, if one study adjusts for a potential confounder – estimates of the effect from this confounder can be applied to the data in the unadjusted study (if raw data on the prevalence of the confounder is present). For example, if one study adjusts for a potential confounder – estimates of the effect from this confounder can be applied to the data in the unadjusted study (if raw data on the prevalence of the confounder is present). If you are combining data from markedly different observational studies, consider using these adjustment techniques as sensitivity analyses. If you are combining data from markedly different observational studies, consider using these adjustment techniques as sensitivity analyses.

AuthorStudy DesignSample SizeSource of ALARR95% CI Giovannucci, et al., 1993† United States Prospective cohort 47,855Diet –1.92 Gann, et al., 1994 United States Nested case/control 120 cases/120 controlsPlasma –5.29 Godley, et al., 1996 United States Case/control89 cases/38 controlsErythrocytes Adipose tissue – –10.61 Andersson, et al., 1996 Sweden Case/control526 cases/536 controlsDiet Harvei, et al., 1997 Norway Nested case/control 141 cases/282 controlsSerum Meyer, et al., 1997 Canada Case/control215 cases/593 controlsDiet ALA and Prostate Cancer

Example: ALA and Prostate Cancer Risk RR=1.2 (1.01 to 1.43), Test for heterogeneity, p=0.00

ALA – Funnel Plot

Funnel Plot with Imputed Values for Publication Bias RR=0.94, 95% CI:

Overview Individual participant data Individual participant data Missing data Missing data Different types of data Different types of data Observational studies Observational studies Generalized synthesis of evidence Generalized synthesis of evidence Meta-regression Meta-regression Survival data Survival data

Generalized synthesis of evidence Methods have been developed to combine RCT’s, observational studies, and even single- arm (no control) studies. Methods have been developed to combine RCT’s, observational studies, and even single- arm (no control) studies. In simple terms, a treatment effect is calculated for each arm of each study – treatment arms are then combined and these summary estimates are compared. In simple terms, a treatment effect is calculated for each arm of each study – treatment arms are then combined and these summary estimates are compared. “the authors stress that caution should be used at the interpretation stage when using this method” “the authors stress that caution should be used at the interpretation stage when using this method”

Overview Individual participant data Individual participant data Missing data Missing data Different types of data Different types of data Observational studies Observational studies Generalized synthesis of evidence Generalized synthesis of evidence Meta-regression Meta-regression Critique of a systematic review Critique of a systematic review

Meta-regression Examines whether the study effects (outcomes) are related to one or more of the study characteristics. Examines whether the study effects (outcomes) are related to one or more of the study characteristics. Can be used to understand/explain heterogeneity. Can be used to understand/explain heterogeneity. Can be thought of as an epidemiological study of the trials or studies. Can be thought of as an epidemiological study of the trials or studies.

Clinical Questions: Meta-Regression 1. Are there certain situations in which BCG may be more effective for preventing TB? 2. What characteristics (clinical and study design) might explain heterogeneity of the sensitivity and specificity of echo? 3. If fish oil lowers heart rate, what population and preparation characteristics may relate to its greatest effect?

Meta-regression: example StudyOR 95% CI , , , , , , , , , , , , , BCG vaccine: used to prevent tuberculosis BCG vaccine: used to prevent tuberculosis Odds ratio estimates from 13 trials (right) Odds ratio estimates from 13 trials (right) Scientists have suggested that effects may be related to geographic latitude Scientists have suggested that effects may be related to geographic latitude

Funnel Plot

Funnel Plot – Organized by Latitude

Meta-regression: example, cont Log odds ratio versus absolute latitude:

Meta-regression: example, cont Same plot, showing precision:

Meta-regression: example, cont Same plot, with fitted (meta-)regression line:

Meta-regression: example, cont Is the slope of the line significantly different from 0? Is the slope of the line significantly different from 0? If yes, we conclude that the study effects are in fact related to to latitude If yes, we conclude that the study effects are in fact related to to latitude

Meta-regression: details In a regression model for the data: each study represents one observation In a regression model for the data: each study represents one observation Weights equal to the study precision Weights equal to the study precision Random effects model (recommended) Random effects model (recommended) Built-in function in Stata: ‘metareg’ Built-in function in Stata: ‘metareg’

Meta-regression: warnings Relationships found via meta-regression do not necessarily suggest causality. Note that relationships may be confounded by other factors (so latitude may be confounded by other health factors). Relationships found via meta-regression do not necessarily suggest causality. Note that relationships may be confounded by other factors (so latitude may be confounded by other health factors). Avoid using summaries of patient characteristics as the covariates of interest. E.g., note that a relationship between study effects and average age does not necessarily suggest that there is a relationship between age and study effect within each study. Avoid using summaries of patient characteristics as the covariates of interest. E.g., note that a relationship between study effects and average age does not necessarily suggest that there is a relationship between age and study effect within each study.

Meta-regression Example 2: Echo, Sensitivity & Specificity

Echo ROC Curve

Heterogeneity Tests, Echo Sensitivity: p =.43 Sensitivity: p =.43 Specificity: p =.059 Specificity: p = Likelihood Ratio: p = Likelihood Ratio: p = Likelihood Ratio: p = Likelihood Ratio: p =.008 ROC curve: p <.0001 ROC curve: p <.0001 DOR: p < DOR: p <

. metareg lnor pmi, wsse(selnor) eform Meta-regression Number of studies = lnor | exp(b) Std. Err. t P>|t| [95% Conf. Interval] pmi | Prior MI & Diagnostic OR Univariate Analysis

. metareg lnor men, wsse(selnor) eform Meta-regression Number of studies = lnor | exp(b) Std. Err. t P>|t| [95% Conf. Interval] men | Gender and Diagnostic OR Univariate Analysis

. metareg lnor age, wsse(selnor) eform Meta-regression Number of studies = lnor | exp(b) Std. Err. t P>|t| [95% Conf. Interval] age | Age and Diagnostic OR Univariate Analysis Mean age = 59 For a year over the mean, the DOR increases by e.8277, (approximately 2-fold increase for 1 year)

Critique of a Systematic Review 1. Research Question 1. Research Question 2. Protocol 2. Protocol 3. Search 3. Search 4. Study selection (inclusion/exclusion) 4. Study selection (inclusion/exclusion) 5. Quality assessment 5. Quality assessment 6. Data abstraction 6. Data abstraction 7. Analysis 7. Analysis A) Create summary measure A) Create summary measure B) Assess for heterogeneity B) Assess for heterogeneity C) Assess for publication bias C) Assess for publication bias D) Conduct sensitivity/subgroup analyses D) Conduct sensitivity/subgroup analyses E) Advanced issues/techniques E) Advanced issues/techniques 8. Interpretation 8. Interpretation

Reviewing Journal Articles Very little formal teaching Very little formal teaching “Because reviews are often highly negative, the new researcher implicitly learns from the negative reviews received on his or her own submitted papers that reviews are supposed to be negative. It is as if the implicit message is: A reviewer’s job is to criticize the manuscript.” “Because reviews are often highly negative, the new researcher implicitly learns from the negative reviews received on his or her own submitted papers that reviews are supposed to be negative. It is as if the implicit message is: A reviewer’s job is to criticize the manuscript.”

12 Tips on Reviewing Articles 1. Know your mission 1. Know your mission 2. Be speedy 2. Be speedy 3. Read carefully 3. Read carefully 4. Say positive things in your review 4. Say positive things in your review 5. Don’t exhibit hostility 5. Don’t exhibit hostility 6. Keep it brief 6. Keep it brief 7. Don’t nitpick 7. Don’t nitpick 8. Develop your own style 8. Develop your own style 9. Be careful in recommending further experiments 9. Be careful in recommending further experiments 10. Watch for egocentrism 10. Watch for egocentrism 11. Make a recommendation 11. Make a recommendation 12. Sign your review 12. Sign your review

Conclusions You can combine almost anything You can combine almost anything Use clinical judgment to guide you in deciding how and whether to combine studies. Use clinical judgment to guide you in deciding how and whether to combine studies. Remember the main mission of a systematic review: to summarize a body of literature in a concise and clear way. Remember the main mission of a systematic review: to summarize a body of literature in a concise and clear way. Get statistical input as needed. Get statistical input as needed.