1. Meta Analyses and Systematic Reviews HINF 371 - Medical Methodologies Session 12

2. Objective Understand what is meant by “evidence-based medicine” Understand what is meant by “evidence-based medicine” Understand the types of research informing evidence Understand the types of research informing evidence Understand meta-analyses and compare two meta analyses examples Understand meta-analyses and compare two meta analyses examples

3. Reading Ioannidis JPA and Lau J (2000) Chapter 4: Evidence Based Medicine: A Quantitative Approach to Decision Making, in Decision Making In Health Care: Theory, Psychology and Applications, Cambridge University Press, USA Ioannidis JPA and Lau J (2000) Chapter 4: Evidence Based Medicine: A Quantitative Approach to Decision Making, in Decision Making In Health Care: Theory, Psychology and Applications, Cambridge University Press, USA Campbell CL, Smyth S, Montalescot G, Steinhubl SR. (2007) Aspirin dose for the prevention of cardiovascular disease: a systematic review, JAMA, 2007 May 9;297(18):2018-24. Campbell CL, Smyth S, Montalescot G, Steinhubl SR. (2007) Aspirin dose for the prevention of cardiovascular disease: a systematic review, JAMA, 2007 May 9;297(18):2018-24. Clark RA, Inglis SC, McAlister FA, Cleland JG, Stewart S. (2007) Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis, BMJ. 2007 May 5;334(7600):942. Clark RA, Inglis SC, McAlister FA, Cleland JG, Stewart S. (2007) Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis, BMJ. 2007 May 5;334(7600):942. Schechtman, E (2002) Odds Ratio, Relative Risk, Absolute Risk Reduction, and the Number Needed to Treat – Which of These Should We Use? Value In Health, Vol:5, No:5, pp.431 - 36 Schechtman, E (2002) Odds Ratio, Relative Risk, Absolute Risk Reduction, and the Number Needed to Treat – Which of These Should We Use? Value In Health, Vol:5, No:5, pp.431 - 36

4. Evidence-based Medicine To introduce more objective, quantifiable estimates of clinical variables to the practice of medicine To introduce more objective, quantifiable estimates of clinical variables to the practice of medicine Real data is better than speculation or opinion – application of formal synthesis of evidence complements traditional “experience based” medicine Real data is better than speculation or opinion – application of formal synthesis of evidence complements traditional “experience based” medicine Evidence-based Medicine = literature based medicine Evidence-based Medicine = literature based medicine High quality, controlled experimental studies have long been preferable to poor quality, uncontrolled studies High quality, controlled experimental studies have long been preferable to poor quality, uncontrolled studies

5. Clinical Studies Double blind (masked) both subjects and evaluators, randomized, placebo controlled studies Double blind (masked) both subjects and evaluators, randomized, placebo controlled studies Single blind (masked) Subjects, randomized, placebo controlled studies Single blind (masked) Subjects, randomized, placebo controlled studies Open-label (unmasked) randomized, placebo controlled studies Open-label (unmasked) randomized, placebo controlled studies Non-randomized, controlled, observational studies with concurrent groups Non-randomized, controlled, observational studies with concurrent groups Observational studies with historical control groups – Framingham study Observational studies with historical control groups – Framingham study Uncontrolled observational studies, especially individual case reports, case series, and descriptive reports based on large databases and clinical registries. Uncontrolled observational studies, especially individual case reports, case series, and descriptive reports based on large databases and clinical registries.

6. Framingham Study The original study cohort consisted of respondents of a random sample of 2/3 of adults, 30 to 62 years of age, residing in Framingham, Massachusetts in 1948. Of the original 5209, there are approximately 1095 known alive as of February 1998. The original study cohort consisted of respondents of a random sample of 2/3 of adults, 30 to 62 years of age, residing in Framingham, Massachusetts in 1948. Of the original 5209, there are approximately 1095 known alive as of February 1998. The Offspring Study was initiated in 1971 when the need for establishing a prospective epidemiologic study of young adults was recognized. A sample of 5135 men and women, consisting of the offspring of the original cohort and their spouses was established. As of February 24, 1998, there were approximately 4524 offspring surviving with only 20 lost to follow-up and 4 in whom survival status was unknown. The Offspring Study was initiated in 1971 when the need for establishing a prospective epidemiologic study of young adults was recognized. A sample of 5135 men and women, consisting of the offspring of the original cohort and their spouses was established. As of February 24, 1998, there were approximately 4524 offspring surviving with only 20 lost to follow-up and 4 in whom survival status was unknown.

7. Problems with Studies Lack of concealment (open label, historical controls) – amplification of effects Lack of concealment (open label, historical controls) – amplification of effects No controlled environment in observational studies – hard to link cause and effect No controlled environment in observational studies – hard to link cause and effect Studies without control groups – bias, spurious claims of large treatment effects Studies without control groups – bias, spurious claims of large treatment effects Length of time – Average randomized trial takes 5 to 7 years from conception to publishing Length of time – Average randomized trial takes 5 to 7 years from conception to publishing

8. Characteristics of Studies Internal Validity Internal Validity Robust comparison throughout the study Robust comparison throughout the study Lack of adherence Lack of adherence Cross over to the other arm Cross over to the other arm Reduced number of follow-up patients Reduced number of follow-up patients Missing measurements Missing measurements External Validity External Validity Study results must be generalizable to the population Study results must be generalizable to the population More controlled the study is harder to generalize More controlled the study is harder to generalize Baseline risk of the population must be considered Baseline risk of the population must be considered Information about patient opt out of treatment is necessary Information about patient opt out of treatment is necessary

9. Measurement of End Points Example: Example: Dyskinesia after Ropinirole (ROP) or Levodopa (LD) for early Parkinson’s disease Dyskinesia after Ropinirole (ROP) or Levodopa (LD) for early Parkinson’s disease 17 out of 179 patients developed dyskinesia in the ROP arm 17 out of 179 patients developed dyskinesia in the ROP arm 23 out of 89 patients developed dyskinesia in the LD arm 23 out of 89 patients developed dyskinesia in the LD arm

10. Measurement of End Points Absolute Risk Reduction Absolute risk: The observed or calculated probability of an event in the population under study Absolute risk: The observed or calculated probability of an event in the population under study Absolute Risk Reduction (risk difference): Is the risk of an event reduced by a clinically meaningful amount Absolute Risk Reduction (risk difference): Is the risk of an event reduced by a clinically meaningful amount The difference between the risk of an event in the control group and the risk of an event in the treatment group The difference between the risk of an event in the control group and the risk of an event in the treatment group Absolute risk: ROP arm 17/179=0.095 Absolute risk: ROP arm 17/179=0.095 Absolute risk: LD arm 23/89=0.258 Absolute risk: LD arm 23/89=0.258 Absolute risk reduction: 0.258-0.095=0.163 Absolute risk reduction: 0.258-0.095=0.163

11. Measurement of End Points Number needed to treat (NNT) Based on Absolute risk reduction. Based on Absolute risk reduction. The number of patients that need to be treated, to get the desired outcome of in one patient who would not have benefited otherwise The number of patients that need to be treated, to get the desired outcome of in one patient who would not have benefited otherwise NNT: 1/0.163=6.13 NNT: 1/0.163=6.13 When negative than Number needed to Harm (NNH) When negative than Number needed to Harm (NNH)

12. Measurement of End Points Relative Risk and Relative Risk Reduction Relative risk: the ratio of risks of the treated group and the control group, also called risk ratio Relative risk: the ratio of risks of the treated group and the control group, also called risk ratio Relative Risk: 0.095/0.258=0.368 Relative Risk: 0.095/0.258=0.368 Relative Risk Reduction: relative risk subtracted from 1. Relative Risk Reduction: relative risk subtracted from 1. Relative Risk Reduction: 1-0.368=0.632 Relative Risk Reduction: 1-0.368=0.632 Relative Risk Reduction=ARR/Control Group risk = 0.163/0.258 = 0.632 Relative Risk Reduction=ARR/Control Group risk = 0.163/0.258 = 0.632

13. Measurement of End Points Odds and Odds Ratio Odds: a proportion in which the numerator contains the number of times an event occurs and the denominator includes the number of times the event does not occur Odds: a proportion in which the numerator contains the number of times an event occurs and the denominator includes the number of times the event does not occur Odds of ROP arm:17/(179-17)=17/162=0.105 Odds of ROP arm:17/(179-17)=17/162=0.105 Odds of LD arm: 23/(89-23)=23/89=0.348 Odds of LD arm: 23/(89-23)=23/89=0.348 Odds ratio: is a common measure of the size of and effect and the goal is to look at associations rather than differences. Odds ratio: is a common measure of the size of and effect and the goal is to look at associations rather than differences. The ratio between the odds of the treated group and the odds of the control group The ratio between the odds of the treated group and the odds of the control group The Odds ratio (OR) less than 1 means that the odds have decreased, and similarly OR greater than 1 means that the odds have increased. The Odds ratio (OR) less than 1 means that the odds have decreased, and similarly OR greater than 1 means that the odds have increased. Odds ratio: 0.105/0.348=0.302 – the odds of dyskinesia is reduced for the ROP arm. Odds ratio: 0.105/0.348=0.302 – the odds of dyskinesia is reduced for the ROP arm.

14. Measurement of End Points Intention to Treat - Cautions Intention to treat: "Intention to treat" is a strategy for the analysis of randomised controlled trials that compares patients in the groups to which they were originally randomly assigned. This is generally interpreted as including all patients, regardless of whether they actually satisfied the entry criteria, the treatment actually received, and subsequent withdrawal or deviation from the protocol. Intention to treat: "Intention to treat" is a strategy for the analysis of randomised controlled trials that compares patients in the groups to which they were originally randomly assigned. This is generally interpreted as including all patients, regardless of whether they actually satisfied the entry criteria, the treatment actually received, and subsequent withdrawal or deviation from the protocol. Better measure of Effectiveness Better measure of Effectiveness Intention to treat gives a pragmatic estimate of the benefit of a change in treatment policy rather than of potential benefit in patients who receive treatment exactly as planned Full application of intention to treat is possible only when complete outcome data are available for all randomised subjects About half of all published reports of randomised controlled trials stated that intention to treat was used, but handling of deviations from randomised allocation varied widely Many trials had some missing data on the primary outcome variable, and methods used to deal with this were generally inadequate, potentially leading to bias Intention to treat analyses are often inadequately described and inadequately applied

15. Other Characteristics Adjustment – may be required in non randomized studies Adjustment – may be required in non randomized studies Sub-group analyses – may distort the results, especially not intended in the original study Sub-group analyses – may distort the results, especially not intended in the original study Stratification – must happen before the randomization Stratification – must happen before the randomization Multiple and Secondary end points – must be defined before the study Multiple and Secondary end points – must be defined before the study Significance – not a good indicator – confidence intervals are better because they allow for sensitivity analyses Significance – not a good indicator – confidence intervals are better because they allow for sensitivity analyses

16. Systematic Review of Evidence Meta-Analyses Meta analysis: a set of qualitative methods for statistically combining the results of different studies on the same topic to explore the degree of and reasons for heterogeneity and bias in the combined results and to provide a quantitative synthesis of these results Meta analysis: a set of qualitative methods for statistically combining the results of different studies on the same topic to explore the degree of and reasons for heterogeneity and bias in the combined results and to provide a quantitative synthesis of these results Example: Cochrane collaboration Example: Cochrane collaboration Meta analyses are Meta analyses are Important for clinical decision making Important for clinical decision making Conducted if there is better than average quality research Conducted if there is better than average quality research Conducted if adequate information available Conducted if adequate information available

17. Meta Analysis Methodology Develop a protocol Develop a protocol Search literature Search literature Extract data (from literature and may be directly from authors of studies Extract data (from literature and may be directly from authors of studies Assess the evidence qualitatively – may be for assigning weights Assess the evidence qualitatively – may be for assigning weights Assess the evidence quantitatively – Assess the evidence quantitatively – Correlation co-efficients, mean difference in the event rate, standardized mean difference (mean difference/control group SD) Correlation co-efficients, mean difference in the event rate, standardized mean difference (mean difference/control group SD) Combine treatment effects Combine treatment effects Truth is fixed – fixed effects model – variation within studies (all studies approximated the same truth, no variation between them, so variation is zero) Truth is fixed – fixed effects model – variation within studies (all studies approximated the same truth, no variation between them, so variation is zero) Truth is not fixed – random effects model – variation within and across studies Truth is not fixed – random effects model – variation within and across studies Presenting results – confidence intervals and overall results Presenting results – confidence intervals and overall results

18. Presenting Results

19. Methods for Conducting Analyses Assess heterogeneity Assess heterogeneity Meta regression analyses – relationship between the magnitude of the treatment effects and different predictors Meta regression analyses – relationship between the magnitude of the treatment effects and different predictors Differences in sample size and variance – publication bias – funnel plot Differences in sample size and variance – publication bias – funnel plot Differences in the duration of follow-up Differences in the duration of follow-up Effect of event rate in the control group Effect of event rate in the control group

20. Measurement of Heterogeneity Funnel Plot Precision of the estimate of treatment effect Treatment effect Log Relative Risk or Sample size

21. Meta Analysis Specific types Meta-analyses of diagnostic tests – receiver operating characteristics = trade off between sensitivity and specificity Meta-analyses of diagnostic tests – receiver operating characteristics = trade off between sensitivity and specificity Meta-analyses of individual patients Meta-analyses of individual patients More detailed time to event analyses More detailed time to event analyses Ability to generate individual based predictive models Ability to generate individual based predictive models But retrieval bias and cumbersome But retrieval bias and cumbersome

22. Other concerns Non-statistically significant results - our concern is magnitude of treatment effects or the strength of the association Non-statistically significant results - our concern is magnitude of treatment effects or the strength of the association Use in public policy – averages might be good but identifying specific populations are also very useful Use in public policy – averages might be good but identifying specific populations are also very useful Sensitivity analysis - might be necessary to identify the most beneficial patient group Sensitivity analysis - might be necessary to identify the most beneficial patient group