An Overview of Meta-analysis in Drug Safety Assessments Jesse A. Berlin, ScD Johnson & Johnson Pharmaceutical Research and Development DIA – FDA – PhRMA.

Slides:



Advertisements
Similar presentations
Agency for Healthcare Research and Quality (AHRQ)
Advertisements

Data Monitoring Models and Adaptive Designs: Some Regulatory Experiences Sue-Jane Wang, Ph.D. Associate Director for Adaptive Design and Pharmacogenomics,
ADVANCED STATISTICS FOR MEDICAL STUDIES Mwarumba Mwavita, Ph.D. School of Educational Studies Research Evaluation Measurement and Statistics (REMS) Oklahoma.
Meta-analysis: summarising data for two arm trials and other simple outcome studies Steff Lewis statistician.
天 津 医 科 大 学天 津 医 科 大 学 Clinical trail. 天 津 医 科 大 学天 津 医 科 大 学 1.Historical Background 1537: Treatment of battle wounds: 1741: Treatment of Scurvy 1948:
ODAC May 3, Subgroup Analyses in Clinical Trials Stephen L George, PhD Department of Biostatistics and Bioinformatics Duke University Medical Center.
Estimation and Reporting of Heterogeneity of Treatment Effects in Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare.
Elements of a clinical trial research protocol
Critical Appraisal for MRCGP Jim McMorran Coventry GP GP trainer Editor GPnotebook (
Common Problems in Writing Statistical Plan of Clinical Trial Protocol Liying XU CCTER CUHK.
Journal Club Alcohol, Other Drugs, and Health: Current Evidence January–February 2011.
Meta-analysis & psychotherapy outcome research
Clinical Trials Hanyan Yang
By Dr. Ahmed Mostafa Assist. Prof. of anesthesia & I.C.U. Evidence-based medicine.
Sample Size Determination
EVIDENCE BASED MEDICINE
Sample Size Determination Ziad Taib March 7, 2014.
Studying treatment of suicidal ideation & attempts: Designs, Statistical Analysis, and Methodological Considerations Jill M. Harkavy-Friedman, Ph.D.
Making all research results publically available: the cry of systematic reviewers.
Are the results valid? Was the validity of the included studies appraised?
Multiple Choice Questions for discussion
Clinical Trials. What is a clinical trial? Clinical trials are research studies involving people Used to find better ways to prevent, detect, and treat.
Intervention Studies Principles of Epidemiology Lecture 10 Dona Schneider, PhD, MPH, FACE.
RESEARCH A systematic quest for undiscovered truth A way of thinking
Funded through the ESRC’s Researcher Development Initiative
Published in Circulation 2005 Percutaneous Coronary Intervention Versus Conservative Therapy in Nonacute Coronary Artery Disease: A Meta-Analysis Demosthenes.
Lecture 16 (Oct 28, 2004)1 Lecture 16: Introduction to the randomized trial Introduction to intervention studies The research question: Efficacy vs effectiveness.
What is a Clinical Trial (alpha version) John M. Harris Jr., MD President Medical Directions, Inc.
Journal Club Hallie Lee PharmD Candidate 2013 Mercer University COPHS PHA 618 Geriatrics-Continuous Care Multivitamins in the Prevention of Cardiovascular.
Epidemiology The Basics Only… Adapted with permission from a class presentation developed by Dr. Charles Lynch – University of Iowa, Iowa City.
Evidence-Based Medicine 3 More Knowledge and Skills for Critical Reading Karen E. Schetzina, MD, MPH.
Program Evaluation. Program evaluation Methodological techniques of the social sciences social policy public welfare administration.
Lecture 17 (Oct 28,2004)1 Lecture 17: Prevention of bias in RCTs Statistical/analytic issues in RCTs –Measures of effect –Precision/hypothesis testing.
Study design P.Olliaro Nov04. Study designs: observational vs. experimental studies What happened?  Case-control study What’s happening?  Cross-sectional.
 Is there a comparison? ◦ Are the groups really comparable?  Are the differences being reported real? ◦ Are they worth reporting? ◦ How much confidence.
Antidepressants and Suicidality in Adults: Statistical Evaluation Mark Levenson, Ph.D.* and Chris Holland, M.S. Statistical Safety Reviewers Quantitative.
Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.
Consumer behavior studies1 CONSUMER BEHAVIOR STUDIES STATISTICAL ISSUES Ralph B. D’Agostino, Sr. Boston University Harvard Clinical Research Institute.
Effect Size Estimation in Fixed Factors Between- Groups Anova.
Meta-analysis and “statistical aggregation” Dave Thompson Dept. of Biostatistics and Epidemiology College of Public Health, OUHSC Learning to Practice.
Successful Concepts Study Rationale Literature Review Study Design Rationale for Intervention Eligibility Criteria Endpoint Measurement Tools.
Lessons Learned From Recent Safety Meta-Analyses Mark Levenson, Ph.D. Quantitative Safety and Pharmacoepidemiology Group Office of Biostatistics Center.
Landmark Trials: Recommendations for Interpretation and Presentation Julianna Burzynski, PharmD, BCOP, BCPS Heme/Onc Clinical Pharmacy Specialist 11/29/07.
Lecture 5 Objective 14. Describe the elements of design of experimental studies: clinical trials and community intervention trials. Discuss the advantages.
RevMan for Registrars Paul Glue, Psychological Medicine What is EBM? What is EBM? Different approaches/tools Different approaches/tools Systematic reviews.
What is a non-inferiority trial, and what particular challenges do such trials present? Andrew Nunn MRC Clinical Trials Unit 20th February 2012.
PH 401: Meta-analysis Eunice Pyon, PharmD (718) , HS 506.
Lecture 9: Analysis of intervention studies Randomized trial - categorical outcome Measures of risk: –incidence rate of an adverse event (death, etc) It.
Federal Institute for Drugs and Medical Devices The BfArM is a Federal Institute within the portfolio of the Federal Ministry of Health (BMG) The use of.
Issues concerning the interpretation of statistical significance tests.
1 Study Design Issues and Considerations in HUS Trials Yan Wang, Ph.D. Statistical Reviewer Division of Biometrics IV OB/OTS/CDER/FDA April 12, 2007.
EBM --- Journal Reading Presenter :呂宥達 Date : 2005/10/27.
Systematic Synthesis of the Literature: Introduction to Meta-analysis Linda N. Meurer, MD, MPH Department of Family and Community Medicine.
Compliance Original Study Design Randomised Surgical care Medical care.
EVALUATING u After retrieving the literature, you have to evaluate or critically appraise the evidence for its validity and applicability to your patient.
Journal Club Alcohol, Other Drugs, and Health: Current Evidence November-December 2012.
Effect of Rosiglitazone on the Risk of Myocardial Infarction And Death from Cardiovascular Causes Alternative Interpretations of the Evidence George A.
Systematic Reviews and Meta-analyses. Introduction A systematic review (also called an overview) attempts to summarize the scientific evidence related.
Course: Research in Biomedicine and Health III Seminar 5: Critical assessment of evidence.
CONSORT 2010 Balakrishnan S, Pondicherry Institute of Medical Sciences.
Is a meta-analysis right for me? Jaime Peters June 2014.
Meta-analysis of observational studies Nicole Vogelzangs Department of Psychiatry & EMGO + institute.
Supplementary Table 1. PRISMA checklist
Randomized Trials: A Brief Overview
Heterogeneity and sources of bias
Lecture 4: Meta-analysis
Critical Reading of Clinical Study Results
Common Problems in Writing Statistical Plan of Clinical Trial Protocol
Gerald Dyer, Jr., MPH October 20, 2016
What is a review? An article which looks at a question or subject and seeks to summarise and bring together evidence on a health topic. Ask What is a review?
Presentation transcript:

An Overview of Meta-analysis in Drug Safety Assessments Jesse A. Berlin, ScD Johnson & Johnson Pharmaceutical Research and Development DIA – FDA – PhRMA Drug Safety Conference October 2008 Arlington, VA / Oct 15, 2008 Jesse A. Berlin, ScD Johnson & Johnson Pharmaceutical Research and Development DIA – FDA – PhRMA Drug Safety Conference October 2008 Arlington, VA / Oct 15, 2008

2 2 The Obligatory Disclaimer The views expressed herein represent those of the presenter and do not necessarily represent the views or practices of the presenter’s employer or any other party.

3 3 Outline Recommendations for the use of meta-analysis for safety assessment during product development: methodologic questions Case studies of the use of historical randomized trial data to address potential safety concerns (including observational studies) Emphasis on exploration of patient-level characteristics as potential effect modifiers –Some more methodologic “heads up” Recommendations for the use of meta-analysis for safety assessment during product development: methodologic questions Case studies of the use of historical randomized trial data to address potential safety concerns (including observational studies) Emphasis on exploration of patient-level characteristics as potential effect modifiers –Some more methodologic “heads up”

4 4 What is Meta-analysis? An optional component of a systematic review Definition: ‘the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” Glass (1976)  : ‘after’, ‘above’, ‘transcending’ An optional component of a systematic review Definition: ‘the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” Glass (1976)  : ‘after’, ‘above’, ‘transcending’

5 5 Is it sampling variability? Problem: How do we distinguish sampling variability from “real” variability (possibly) associated with different effects of treatment in different subgroups of patients (or with different dosing algorithms or other specific aspects of treatment)?

6 6 Why do a meta-analysis? To increase power and precision –detect effect as statistically significant; narrower Cis To quantify effect sizes and their uncertainty –reduce problems of interpretation due to sampling variation To systematically assess the overall findings from a body of literature –Reduce the tendency to focus only on results that support prior beliefs To answer questions not posed by the individual studies –Study-level factors (e.g., double-blind vs. open-label) –Patient-level factors To increase power and precision –detect effect as statistically significant; narrower Cis To quantify effect sizes and their uncertainty –reduce problems of interpretation due to sampling variation To systematically assess the overall findings from a body of literature –Reduce the tendency to focus only on results that support prior beliefs To answer questions not posed by the individual studies –Study-level factors (e.g., double-blind vs. open-label) –Patient-level factors

7 7 Combined: RR = 0.79 (95% CI 0.72,0.87) Risk ratio Estimates with 95% confidence intervals IV streptokinase for acute MI (3 month mortality)

8 8 Estimated OR for IHD events by extent of serum cholesterol reduction (from Thompson, SMMR 1993; 2: )

9 9 Guidelines from the Safety Planning, Evaluation, and Reporting Team (SPERT) white paper Multi-company PhRMA committee with goal of recommending an industry-wide standard for safety planning, data collection, evaluation, and reporting A few selected items for your consideration for what to do during development Multi-company PhRMA committee with goal of recommending an industry-wide standard for safety planning, data collection, evaluation, and reporting A few selected items for your consideration for what to do during development

10 SPERT Recommendations (1) PRINCIPLES: Safety questions can be investigated by aggregating the cumulative safety data on an ongoing basis to obtain a single estimate of treatment effect for individual safety parameters We recommend that sponsors develop a Program Safety Analysis Plan (PSAP) as a tool to proactively plan for meta-analysis of the program safety data. PRINCIPLES: Safety questions can be investigated by aggregating the cumulative safety data on an ongoing basis to obtain a single estimate of treatment effect for individual safety parameters We recommend that sponsors develop a Program Safety Analysis Plan (PSAP) as a tool to proactively plan for meta-analysis of the program safety data.

11 SPERT Recommendations (2) Specify important adverse events prior to commencing pivotal clinical trials This facilitates subsequent integration and interpretation of data by collecting important data in a standard fashion in all relevant studies. Specify important adverse events prior to commencing pivotal clinical trials This facilitates subsequent integration and interpretation of data by collecting important data in a standard fashion in all relevant studies.

12 Program Safety Analysis Plan (PSAP) Second section focuses on analyses –Those to be analyzed using formal inferential statistics (Tier 1 events: specified a priori) –Statistical and graphical methodologies –Should address missing values, multiplicity, analysis population, etc., much like a single- trial statistical analysis plan does. POINT: Make safety analysis plans look more like efficacy analysis plans than they have in the past Second section focuses on analyses –Those to be analyzed using formal inferential statistics (Tier 1 events: specified a priori) –Statistical and graphical methodologies –Should address missing values, multiplicity, analysis population, etc., much like a single- trial statistical analysis plan does. POINT: Make safety analysis plans look more like efficacy analysis plans than they have in the past

13 Analytical considerations Power and precision considerations for the contemplated pooled/meta-analyses (including subgroup analyses) –Traditional hypothesis testing versus “ruling out” an increase in risk of a certain size (like “non-inferiority”) Are dedicated clinical safety studies needed to address specific safety endpoints? The PSAP should be discussed with the regulatory authorities at an agreed-upon milestone (e.g., end-of-Phase II meeting) –Therefore the first version of the analysis plan should be completed prior to this meeting! Power and precision considerations for the contemplated pooled/meta-analyses (including subgroup analyses) –Traditional hypothesis testing versus “ruling out” an increase in risk of a certain size (like “non-inferiority”) Are dedicated clinical safety studies needed to address specific safety endpoints? The PSAP should be discussed with the regulatory authorities at an agreed-upon milestone (e.g., end-of-Phase II meeting) –Therefore the first version of the analysis plan should be completed prior to this meeting!

14 Planning meta-analyses (you will hear this again) ICH E9 guideline states that meta-analyses should be prospectively planned with the clinical trials program in the development of a new treatment Not just planning the logistics, but planning the scientific questions to be addressed (Berlin and Colditz, JAMA 1999; 281: ) –Standardization of definitions of endpoints –Standardization of data collection to allow combination of results across all studies in the development program. –“Meta-design” considerations ICH E9 guideline states that meta-analyses should be prospectively planned with the clinical trials program in the development of a new treatment Not just planning the logistics, but planning the scientific questions to be addressed (Berlin and Colditz, JAMA 1999; 281: ) –Standardization of definitions of endpoints –Standardization of data collection to allow combination of results across all studies in the development program. –“Meta-design” considerations

15 Meta-experimental design Plan and control variation in the different factors in a systematic manner. –Like a factorial experiment or a single randomized trial with stratified randomization Better to conduct 2 studies, each including both men and women, and to stratify (either in the randomization or post hoc) by sex, rather than to do one study in men and a separate study in women. –Separate studies by sex confounds sex and “study” –Might be further confounded by different doses POINT: Think about what the “meta design space” will look like when you’re done Plan and control variation in the different factors in a systematic manner. –Like a factorial experiment or a single randomized trial with stratified randomization Better to conduct 2 studies, each including both men and women, and to stratify (either in the randomization or post hoc) by sex, rather than to do one study in men and a separate study in women. –Separate studies by sex confounds sex and “study” –Might be further confounded by different doses POINT: Think about what the “meta design space” will look like when you’re done

16 What Happens in Practice? Whether we’re doing the meta-analysis before or after approval, we need to think about how to address “heterogeneity” (a recurring theme for today) Whether we’re doing the meta-analysis before or after approval, we need to think about how to address “heterogeneity” (a recurring theme for today)

17 What is heterogeneity? Clinical heterogeneity Participants –e.g., conditions under investigation, eligibility criteria for trials, geographical variation Interventions –e.g., intensity / dose / duration, sub-type of drug, mode of administration, experience of practitioners, nature of the control (placebo/none/standard care) Outcomes –e.g., follow-up duration, ways of measuring, cut-off points on scales Clinical heterogeneity Participants –e.g., conditions under investigation, eligibility criteria for trials, geographical variation Interventions –e.g., intensity / dose / duration, sub-type of drug, mode of administration, experience of practitioners, nature of the control (placebo/none/standard care) Outcomes –e.g., follow-up duration, ways of measuring, cut-off points on scales

18 What is heterogeneity? Methodologic heterogeneity Design –e.g., randomized vs. non-randomized, crossover vs. parallel group vs. cluster randomized, length Conduct –e.g., allocation concealment, blinding (masking) of subjects, treating physicians, outcome evaluation, etc., approach to analysis (intent-to-treat vs. “completers”) Methodologic heterogeneity Design –e.g., randomized vs. non-randomized, crossover vs. parallel group vs. cluster randomized, length Conduct –e.g., allocation concealment, blinding (masking) of subjects, treating physicians, outcome evaluation, etc., approach to analysis (intent-to-treat vs. “completers”)

19 What is heterogeneity? Statistical heterogeneity Common views –Variation in the results of studies –More variation than would be expected by chance In truth: –Variation in the true effects underlying the studies –that may manifest itself in more observed variation than expected by chance –may be due to different treatment effects or different biases Is statistical heterogeneity inevitable? Statistical heterogeneity Common views –Variation in the results of studies –More variation than would be expected by chance In truth: –Variation in the true effects underlying the studies –that may manifest itself in more observed variation than expected by chance –may be due to different treatment effects or different biases Is statistical heterogeneity inevitable?

20 Identifying heterogeneity How do we tell whether statistical variation among (between) results is due to chance or real differences? Eyeballing –a graphical inspection of the results is usually the first step –a lack of overlap in confidence intervals indicates heterogeneity (but overlap does not imply absence of heterogeneity) How do we tell whether statistical variation among (between) results is due to chance or real differences? Eyeballing –a graphical inspection of the results is usually the first step –a lack of overlap in confidence intervals indicates heterogeneity (but overlap does not imply absence of heterogeneity)

Favors opioidFavors placebo Estimates with 95% confidence intervals Standardized mean difference Opioids for breathlessness Estimates with 95% confidence intervals Favors LR Favors control Risk ratio Early light reduction for ROP

22 Identifying heterogeneity Statistical test –A chi-squared (  2 ) test (Cochran’s Q) –Has low power because there are usually very few studies:  i.e., test is not very good at detecting heterogeneity when it exists –But, has excessive power to detect clinically unimportant heterogeneity when there are many studies Statistical test –A chi-squared (  2 ) test (Cochran’s Q) –Has low power because there are usually very few studies:  i.e., test is not very good at detecting heterogeneity when it exists –But, has excessive power to detect clinically unimportant heterogeneity when there are many studies

23 Identifying heterogeneity Test is not asking a useful question if heterogeneity is inevitable Quantify inconsistency –based on  2 statistic, Q, and its degrees of freedom. describes the proportion of variability that is due to heterogeneity as opposed to sampling error (d.f. = degrees of freedom = the number of studies minus 1) Test is not asking a useful question if heterogeneity is inevitable Quantify inconsistency –based on  2 statistic, Q, and its degrees of freedom. describes the proportion of variability that is due to heterogeneity as opposed to sampling error (d.f. = degrees of freedom = the number of studies minus 1)

24 What can we do with heterogeneity? Ignore it Check the data Encompass it Explore it Ignore it Check the data Encompass it Explore it Don’t do that! (worse yet – some people throw out the outliers) Incorrect data extraction; unit of analysis errors (e.g., with crossover trials, cluster randomized trials, counts) Random effects meta-analysis Subgroup analysis Meta-regression Funnel plot

25 True effect Random error Result Fixed effect meta-analysis model (statistical homogeneity)

26 Random effects meta-analysis model Random error Trial specific effect True mean effect The width of the curve reflects the amount of heterogeneity

27 Random effects meta-analysis The ‘amount’ of heterogeneity can be estimated Weights are adjusted to account for both within-study and among-study variability Random effects analyses give –similar results when there is no heterogeneity –similar pooled effect, wider confidence interval when there is ‘symmetric’ heterogeneity –different results when there is funnel plot asymmetry – they give more weight to the potentially biased sample of small studies The ‘amount’ of heterogeneity can be estimated Weights are adjusted to account for both within-study and among-study variability Random effects analyses give –similar results when there is no heterogeneity –similar pooled effect, wider confidence interval when there is ‘symmetric’ heterogeneity –different results when there is funnel plot asymmetry – they give more weight to the potentially biased sample of small studies

28 Identical results Early light reduction for ROP Estimates with 95% confidence intervals Favours LR Favours control Risk ratio Kennedy 1997 Locke 1952A Lopes 1997 Reynolds 1998 Seiberth 1994 Random effects Fixed effect

29 Slightly different results Opioids for breathlessness –0.31 ( –0.50, –0.13 ) Trial Woodcock 1981 Woodcock 1982 Johnson Eiser (A) Eiser (B) Bruera Light Chua Poole Davis Leung Noseda Random effects Opioid betterPlacebo better Estimates with 95% confidence intervals Standardised mean difference Fixed effect–0.32 ( –0.43, –0.20 )

30 Very different results Risk ratio Study Morton Rasmussen Smith Abraham Feldstedt Shechter 1990 Ceremuzynski LIMIT-2 Bertschat Singh Pereira Golf Thogersen Shechter 1995 Estimates with 95% confidence intervals ISIS-4 MAGIC 1.01 (0.97,1.07) Fixed effect 0.76 (0.62,0.92) Random effects IV magnesium for acute MI (mortality)

31 RE models can be counter-intuitive Study 1DeadAliveTotal Treatment30 (60%)2050 Control10 (20%)4050

32 Counter-intuitive RE (2) Study 2DeadAliveTotal Treatment100 (1%)9,90010,000 Control200 (2%)9,80010,000

33 Counter-intuitive RE (3) Study 1: RR = 60% / 20% = 3N = 100 Study 2: RR = 1% / 2% = 0.50N = 20,000 Heterogeneity test p-value < Fixed effect summary OR = 0.60 (0.48, 0.76) Random effects summary OR = 1.66 (0.14, 19) Study 1: RR = 60% / 20% = 3N = 100 Study 2: RR = 1% / 2% = 0.50N = 20,000 Heterogeneity test p-value < Fixed effect summary OR = 0.60 (0.48, 0.76) Random effects summary OR = 1.66 (0.14, 19)

34 Examples and challenges

35 SSRIs and Suicidal Behaviors

36 Did we already know this? “With beginning convalescence (following initiation of treatment with tricyclic antidepressants), the risk of suicide once more becomes serious as retardation fades.” – [Clinical Psychiatry, by Mayer-Gross, Slater, and Roth, 1960, p. 231] “While this and other mechanisms all have some plausibility as explanations for the clinical observation of worsening depression or suicidality in depressed patients being treated with antidepressants, proposing a mechanism is quite a different matter from demonstrating empirically that there is a causal association between antidepressant use and induction of suicidality.” –FDA Briefing Book for PDAC, 2006 “With beginning convalescence (following initiation of treatment with tricyclic antidepressants), the risk of suicide once more becomes serious as retardation fades.” – [Clinical Psychiatry, by Mayer-Gross, Slater, and Roth, 1960, p. 231] “While this and other mechanisms all have some plausibility as explanations for the clinical observation of worsening depression or suicidality in depressed patients being treated with antidepressants, proposing a mechanism is quite a different matter from demonstrating empirically that there is a causal association between antidepressant use and induction of suicidality.” –FDA Briefing Book for PDAC, 2006

37 SSRI Methods: adjudication? Possibly suicide-related adverse events (PSRAEs) were adjudicated by the sponsors using the algorithm developed by the group at Columbia U. (K. Posner) Reason: “…large number of subjects (approximately 100,000) in the adult suicidality analysis, which made impracticable more detailed adjudication of all potentially suicidal behaviors by the FDA.” So – what’s the standard? –Independent third party? –What would be the anticipated direction of any bias related to lack of adjudication? Possibly suicide-related adverse events (PSRAEs) were adjudicated by the sponsors using the algorithm developed by the group at Columbia U. (K. Posner) Reason: “…large number of subjects (approximately 100,000) in the adult suicidality analysis, which made impracticable more detailed adjudication of all potentially suicidal behaviors by the FDA.” So – what’s the standard? –Independent third party? –What would be the anticipated direction of any bias related to lack of adjudication?

38 Conclusions about Adjudication “A wide variety of approaches can help assure that outcome assessment in large simple trials is clinically relevant, accurate, and without differential misclassification” (JB added emphasis) Adjudication increases cost and complexity “Based on the available data from cardiovascular trials, adjudication has not been shown to improve the ability to determine treatment effects.” –Granger CB, Vogel V, Cummings SR, et al. Do we need to adjudicate major clinical events? Clinical Trials 2008;5: “A wide variety of approaches can help assure that outcome assessment in large simple trials is clinically relevant, accurate, and without differential misclassification” (JB added emphasis) Adjudication increases cost and complexity “Based on the available data from cardiovascular trials, adjudication has not been shown to improve the ability to determine treatment effects.” –Granger CB, Vogel V, Cummings SR, et al. Do we need to adjudicate major clinical events? Clinical Trials 2008;5:56-60.

39 Broad versus narrow definitions Common view is that more sensitive definitions –Are more “conservative” by being inclusive –Increase power by generating more events Overly broad inclusion of events can lead to an underestimation of the true relative risk –might include events less likely to be related to the true (but possibly unknown) mechanism of action or –by their nature, are simply more likely to be misclassified in clinical trials Implications of “non-differential” misclassification in efficacy versus safety settings? (MORE LATER) Common view is that more sensitive definitions –Are more “conservative” by being inclusive –Increase power by generating more events Overly broad inclusion of events can lead to an underestimation of the true relative risk –might include events less likely to be related to the true (but possibly unknown) mechanism of action or –by their nature, are simply more likely to be misclassified in clinical trials Implications of “non-differential” misclassification in efficacy versus safety settings? (MORE LATER)

40 What endpoints (AEs) were included? Primary outcome: suicidal ideation or worse (outcomes 1, 2, 3 or 4 below), also called suicidality or suicidal behavior and ideation. –1. Completed suicide –2. Suicide attempt –3. Preparatory acts toward imminent suicidal behavior –4. Suicidal ideation –5. Self-injurious behavior, intent unknown –6. Not enough information (Fatal) –7. Not enough information (Non-Fatal) Primary outcome: suicidal ideation or worse (outcomes 1, 2, 3 or 4 below), also called suicidality or suicidal behavior and ideation. –1. Completed suicide –2. Suicide attempt –3. Preparatory acts toward imminent suicidal behavior –4. Suicidal ideation –5. Self-injurious behavior, intent unknown –6. Not enough information (Fatal) –7. Not enough information (Non-Fatal)

41 Statistical methods Aggregate-level analyses: –Mantel-Haenszel (fixed-effect primary) –DerSimonian-Laird –“Double zero” studies excluded –Single zero – continuity correction Aggregate-level analyses: –Mantel-Haenszel (fixed-effect primary) –DerSimonian-Laird –“Double zero” studies excluded –Single zero – continuity correction

42 What about those “no event” studies? The exclusion of trials with no events in either placebo or primary active drug arms is problematic. The absence of events provides some information because of the background rate of events independent of drug effect. Studies with no events are dropped from the likelihood for usual ratio estimates (OR, RR) Risk differences are perhaps more promising, although also have their own problems (e.g., variance estimation) DO SENSITIVITY ANALYSES (and attend the talk later) The exclusion of trials with no events in either placebo or primary active drug arms is problematic. The absence of events provides some information because of the background rate of events independent of drug effect. Studies with no events are dropped from the likelihood for usual ratio estimates (OR, RR) Risk differences are perhaps more promising, although also have their own problems (e.g., variance estimation) DO SENSITIVITY ANALYSES (and attend the talk later)

43 Patient-level analyses Allow exploration of subgroups defined by patient-level characteristics Ecological bias can be a problem when regressing study result (e.g., log OR) against aggregate-level patient characteristics (e.g., mean age, percent male, etc.) –Statistics in Medicine, 2002; 21: FDA used conditional logistic regression NOTE: also allows proper time-to-event analyses when appropriate Allow exploration of subgroups defined by patient-level characteristics Ecological bias can be a problem when regressing study result (e.g., log OR) against aggregate-level patient characteristics (e.g., mean age, percent male, etc.) –Statistics in Medicine, 2002; 21: FDA used conditional logistic regression NOTE: also allows proper time-to-event analyses when appropriate

44 Analyses by age Young vs. Older Adults <25, 25+ Young, Middle-aged and Elderly <25, 25-64, 65+ Age by Decade <25, 25-34, 35-44, 45-54, 55-64, 65-74, 75+ Age by Double Decade <25, 25-44, 45-64, 65+ (Assessing sensitivity of results to choice of definition of age categories) Could use non-linear fitting algorithms, like multivariate restricted splines –(e.g. Royston P, Sauerbrei W. Multivariable modeling with cubic regression splines: A principled approach. The Stata Journal 2007;7(1):45-70) Young vs. Older Adults <25, 25+ Young, Middle-aged and Elderly <25, 25-64, 65+ Age by Decade <25, 25-34, 35-44, 45-54, 55-64, 65-74, 75+ Age by Double Decade <25, 25-44, 45-64, 65+ (Assessing sensitivity of results to choice of definition of age categories) Could use non-linear fitting algorithms, like multivariate restricted splines –(e.g. Royston P, Sauerbrei W. Multivariable modeling with cubic regression splines: A principled approach. The Stata Journal 2007;7(1):45-70)

45 Results overall Suicidality Risk for Active Drug relative to Placebo– Ideation or Worse – All Adults – All Diagnoses –0.85 (0.71 – 1.02), p = 0.08 by conditional LR –0.86 (0.71 – 1.04), p = 0.12 Exact Method Suicide-related behavior (preparatory acts, attempts and completed suicide) –OR = 1.12 (95% CI, 0.79 – 1.58), by conditional logistic regression) (LOOKS DIFFERENT?) Suicidality Risk for Active Drug relative to Placebo– Ideation or Worse – All Adults – All Diagnoses –0.85 (0.71 – 1.02), p = 0.08 by conditional LR –0.86 (0.71 – 1.04), p = 0.12 Exact Method Suicide-related behavior (preparatory acts, attempts and completed suicide) –OR = 1.12 (95% CI, 0.79 – 1.58), by conditional logistic regression) (LOOKS DIFFERENT?)

46 Results by indication

47 Results by age

48 MA of observational studies (briefly) Meta-analysis of observational studies remains controversial –How many epidemiologists does it take to change a light bulb? The point will often NOT be to produce a single summary estimate, but to explore (presumed) sources of heterogeneity of findings Meta-analysis of observational studies remains controversial –How many epidemiologists does it take to change a light bulb? The point will often NOT be to produce a single summary estimate, but to explore (presumed) sources of heterogeneity of findings

49

50 Conclusions (1) Meta-analysis has valuable applications in pharmacoepidemiology –Evaluation of safety using existing randomized trials –Evaluation of safety using non-experimental studies (need more time to show) Meta-analysis has valuable applications in pharmacoepidemiology –Evaluation of safety using existing randomized trials –Evaluation of safety using non-experimental studies (need more time to show)

51 Conclusions (2) There are challenging methodologic issues in the meta-analysis of safety data –Rare events, multiplicity, adjudication, … Sensitivity analyses should always be performed –Then more sensitivity analyses should always be performed Use patient-level data when possible There are challenging methodologic issues in the meta-analysis of safety data –Rare events, multiplicity, adjudication, … Sensitivity analyses should always be performed –Then more sensitivity analyses should always be performed Use patient-level data when possible

Another example (if time permits)

53 Example: Galantamine Acetylcholinesterase inhibitors (AchEIs) are used as a standard treatment for Alzheimer’s Disease (AD) Galantamine, an AChEI, has been extensively studied in patients with mild to moderate AD Galantamine has also been studied in patients with AD with concomitant cerebrovascular disease (CVD) and in patients with VaD (16). The benefit is to slow the progress of cognitive decline (relative to placebo) Acetylcholinesterase inhibitors (AchEIs) are used as a standard treatment for Alzheimer’s Disease (AD) Galantamine, an AChEI, has been extensively studied in patients with mild to moderate AD Galantamine has also been studied in patients with AD with concomitant cerebrovascular disease (CVD) and in patients with VaD (16). The benefit is to slow the progress of cognitive decline (relative to placebo)

54 Safety “signal” for Galantamine in Mild Cognitive Impairment Two 2-year randomized controlled trials –Individuals with mild cognitive impairment –Findings replicated in both studies –13 deaths versus 1 death Higher mortality observed in galantamine-treated patients, compared with placebo –Overall mortality rates were low in both groups The findings prompted a reevaluation in patients with dementia Two 2-year randomized controlled trials –Individuals with mild cognitive impairment –Findings replicated in both studies –13 deaths versus 1 death Higher mortality observed in galantamine-treated patients, compared with placebo –Overall mortality rates were low in both groups The findings prompted a reevaluation in patients with dementia

55 Galantamine Methods All galantamine trials (J&J or Shire-sponsored) for which J&J could access data Also searched MEDLINE and the Cochrane Controlled Trials Register (2005) Issue 4 Trials included were independently reviewed, verified by two readers, and met the following criteria: –a) randomized –b) placebo-controlled –c) parallel group –d) blinded –e) at least one treatment arm with galantamine All galantamine trials (J&J or Shire-sponsored) for which J&J could access data Also searched MEDLINE and the Cochrane Controlled Trials Register (2005) Issue 4 Trials included were independently reviewed, verified by two readers, and met the following criteria: –a) randomized –b) placebo-controlled –c) parallel group –d) blinded –e) at least one treatment arm with galantamine

56 Meta-analysis of survival in galantamine randomized trials (6 months duration)

57 Other Galantamine Analyses Nested case-control study of deaths was used to investigate potential mechanism for the mortality increase –Baseline ECG findings –Comorbidities –Concomitant medications Findings were inconclusive due to small sample size Mortality analyses in press (Feldman et al.; Acta Neurologica Scandinavica) We are doing a large, placebo-controlled study with mortality as the primary endpoint Nested case-control study of deaths was used to investigate potential mechanism for the mortality increase –Baseline ECG findings –Comorbidities –Concomitant medications Findings were inconclusive due to small sample size Mortality analyses in press (Feldman et al.; Acta Neurologica Scandinavica) We are doing a large, placebo-controlled study with mortality as the primary endpoint