Analyzing Intervention Studies Shaun M. Eack, Ph.D. University of Pittsburgh July 14, 2011

Overview Analytic Principles Analytic Goals and Methods Fixed-effects models Mixed-effects models Reporting (CONSORT)

Analytic Principles of Intervention Studies Trial Registration (www.clinicaltrials.gov) – all clinical trials funded by the NIH and other federal agencies must be registered Required for publication in high-impact journals (Archives, NEJM, JAMA) Investigators are held to the details outlined in the registration, particularly for analysis and outcomes

Analytic Principles of Intervention Studies Primary outcomes 1 or 2 key outcome measures Simple and easy to understand Secondary outcomes Additional exploratory measures Not necessarily powered to detect in your study

Analytic Principles of Intervention Studies Completer Analysis – Examining only those who complete the study Partial Completer Analysis – Examining those who receive a specified minimum amount of treatment exposure Intent-to-treat analysis – Examining all individuals who you intended to treat

Methods of Handling Attrition Ignore it (completer analysis) Impute Mean Regression Multiply Impute LOCF Estimate Expectation-Maximization/Maximum Likelihood

Goals of Analysis Purposes Examples Examine change in averages over time Examine differences in averages between groups Examine differences in changes in averages over time between groups Examples Pretest-Posttest, therapy and depression Posttest Only, aspirin and heart attack RCT, disulfiram and alcohol use

Pretest-Posttest, Therapy and Depression

Posttest Only, Aspirin and Heart Attack

RCT, Disulfiram and Alcohol Abuse (no placebo)

RCT, Disulfiram and Alcohol Abuse (no placebo)

Fixed-Effects Models Independent T-Test - compare averages of 2 groups at 1 time point Analysis of Variance (ANOVA) - compare averages between 2 or more groups at 1 time point Dependent T-Test - compare averages of 1 group at 2 time points Repeated Measures ANOVA - compare averages between 2 or more groups at 2 or more time points

Fixed-Effects Models and Designs Independent T-Test Posttest Only, Static-Group Comparison Dependent T-Test Time Series, Repeated Treatment, One-Group Pretest-Posttest ANOVA Posttest Only and Static-Group Comparison with more than 2 groups Repeated Measures ANOVA Classic Experiment, Active Control Group, Placebo Control Group, Nonequivalent Comparison Groups, Multiple Time Series

Mixed-Effects Models Some effects are fixed Average intercept for the sample Average slope for the sample Some effects are random and allowed to vary across individuals Random (individual) deviations from the mean intercept Random (individual) deviations from the mean slope

Mixed-Effects Models Mixed-effects models of longitudinal data are a special type of hierarchical linear modeling – Growth Curve Modeling Considered HLM because of the multilevel nature of data examined Level 1 – Individual measurements over time Level 2 – Constant individual characteristics Level 1 is nested within Level 2 (measurement occasions nested within individuals)

Unconditional Growth Model Level 1: Yti = β0i + β1i(Time)t + rti (1) Level 2: β0i = γ00 + μ0i (2) β1i = γ10 + μ1i (3) β0i - Intercept for person i β1i - Slope for person i γ00 - Average intercept γ10 - Average slope

Conditional Growth Model Level 1: Yti = β0i + β1i(Time)t + rti (4) Level 2: β0i = γ00 + γ01(Treatment)i + μ0i (5) β1i = γ10 + γ11(Treatment)i + μ1i (6) Treatment: 0 = Control; 1 = Therapy γ01 - Effect of treatment on intercept γ11 - Effect of treatment on slope

Conditional Growth Model Level 1: Yti = β0i + β1i(Time)t + rti (7) Level 2: β0i = γ00 + γ01(Treatment)i + μ0i (8) β1i = γ10 + γ11(Treatment)i + μ1i (9) Mixed Model: Yti = γ00 + γ01(Treatment)i + γ10(Time)t + γ11(Treatment)i(Time)t + μ0i + μ1i(Time)t + rti (10)

Coding Time The coding of the time variable is very important for interpretation Baseline = 0, Month 9 = 1, Month 18 = 2 Intercept = average score on outcome Y at baseline Baseline = 2, Month 9 = 1, Month 18 = 0 Intercept = average score on outcome Y at 18 months Slope = change is linear Baseline = 0, Month 9 = 1, Month 18 = 4 Slope = change is quadratic

Advantages of Growth Modeling Automatically estimates missing data Includes individuals who are missing time points Does not require equal timepoints for all participants Allows correlated lagged residuals Capable of modeling heterscedasticity

Disadvantages of Growth Modeling Requires larger number of observations Obs > 50 Obs = N * T Feasible with large N or T Estimation is more difficult Easy to ask questions you do not intend Generally more conceptually complex

Multiple Inference Testing Probability of 1/10 tests reaching < .05 is 50%. Odds increase as you conduct more tests. Adjustments have to be made: Test only a small number of primary outcomes first Apply a correction for inflated Type I error

Corrections for Multiple Inference Testing Bonferroni – p.crit = p.obs / k tests Low power Sacrifices Type II error 20 tests, p.crit = .05/20 = .0025 Ignores correlations among tests Ignores decreased probability of Type I error with multiple lower p-values

Corrections for Multiple Inference Testing Hochberg (1988) Benjamini and Hochberg (1995) False Discovery Rate Nothing trumps a good hypothesis and plan

Reporting CONSORT – Consolidated Standards of Reporting Trials Meta-analysts were having trouble summarizing the literature Investigators were leaving important information out of their treatment reports

CONSORT Requirements Title: Randomized trial must be stated if the study is an RCT Methods: Consort diagram of participant flow Randomization method Randomization implementer Blinding Defined primary and secondary outcomes Results: Baseline demographic data Number of participants included in analyses Presentation of effect sizes