1. Multivariate Meta-analysis: Notes on Correlations Robert Platt Department of Epidemiology & Biostatistics McGill University Jack Ishak United BioSource

2. Outline Background – Multivariate MAs Motivation – correlations must matter Correlations in multivariate meta-analysis Impact of errors in approximating within- study covariances

3. Background Clinical and epidemiological studies typically examine the effect of a treatment or exposure on a number of outcomes E.g., CVD: effect of a new anti-hypertensive agent on MI, stroke, mortality, etc. Meta-analyses of these studies will also aim to summarize the evidence about all outcomes Standard approach: separate meta-analysis for each outcome Plausible that outcomes are correlated: Common biological pathways, common risk factors, etc. Alternative: joint meta-analysis of outcomes.

4. Models for Joint Meta-Analyses For k outcomes: Covariance (correlation) Between Effects Across Studies Estimated from the data Covariance between Effects in each study Assumed known, but covariance rarely reported k = 2

5. Motivation Correlations must matter Other evidence – eg. MA of longitudinal studies – suggests that it does But is this a special case? Consider the general case of two correlated outcomes

6. Objectives Correlations between treatment effects Do they reflect dependence in treatment response, or something else? Approximating within-study covariances Given that we dont usually observe estimates of within- study covariances, what is the impact of errors in approximations of within-study covariances on estimates of model parameters?

7. Correlations in MMA Rationale: Nam et al.: Effect of environmental tobacco smoke (ETS) on incidence of asthma and lower respiratory disease (LRD) OR Asthma : 1.20 (1.07-1.35), OR LRD : 1.27 (1.10-1.47) Correlation (between effect sizes): 0.55 (-0.8, 0.99) How can we interpret this correlation? Studies that found strong association between ETS and asthma also found strong association with LRD? Asthma and LRD tend to occur together (regardless of underlying cause)? People who develop asthma because of ETS are likely to also develop LRD? Other explanations? Not hard to think of more… Correlations in Multivariate Meta-analysis: Do Correlated Effect Sizes Imply Correlated Effects?

8. Simulation Study Effect of treatment on two outcomes (A and B) Generate studies from a process that allows three levels of dependence/correlation: Events Treatment response Studies Simulate true values - not estimates - of correlation Since estimates depend on number of studies, sample sizes, etc. If correlations were estimated perfectly, would they reflect the true level of dependence in treatment response?

9. Parameterization of Dependence Events: P = (P[A], P[B|A], P[B|not A]) Frequency: Moderate or Common Dependence: Independent: P[B|A] = P[B|not A] Dependent: P[B|A] P[B|not A] Strength of dependence: Moderate or High Treatment Effects: OR = (OR[A], OR[B|A], OR[B|not A]) Strength: Moderate or Strong Effect Dependence in Response: Independent Effects: OR[B|A] = OR[B|not A] Dependent Effects: OR[B|A] OR[B|not A] Strength of dependence: Moderate or High

10. Study-Level Dependence Factors that cause variability between studies will cause correlation within studies Aspects of design Study population Correlated errors Variability in P Correlation between components of P Variability in OR Correlation between components of OR

11. Simulations Define a scenario Fix values for P, OR and D P and D OR Generate large number of studies (N=100K) Represents pool of studies in which meta-analyses would be carried out Derive marginal ORs for each study Calculate correlation between (log) ORs for two events across all studies Represents true value of correlations estimated in meta-analyses Does this correlation reflect the strength of the dependence between treatment effects?

12. Observed Correlations 0 0.25 0.5 0.75 1 Mod. - Mod. Dep. Mod. - Highly Dep. Common, Mod. Dep. Common, Highly Dep. Mod. - Mod. Dep. Mod. - Highly Dep. Common, Mod. Dep. Common, Highly Dep. Mod. - Mod. Dep. Mod. - Highly Dep. Common, Mod. Dep. Common, Highly Dep. Mod. - Mod. Dep. Mod. - Highly Dep. Common, Mod. Dep. Common, Highly Dep. Moderately Strong, Moderately Dependent Moderately Strong, Highly Dependent Strong, Moderately DependentStrong, Highly Dependent Low Correlations btw REs Moderate Correlations btw REs High Correlations btw REs Corr (LOR[A], LOR[B])

13. Independent Treatment Effects 0 0.25 0.5 0.75 1 Mod. - Mod. Dep. Mod. - Highly Dep. Common, Mod. Dep. Common, Highly Dep. Mod. - Mod. Dep. Mod. - Highly Dep. Common, Mod. Dep. Common, Highly Dep. Mod. - Mod. Dep. Mod. - Highly Dep. Common, Mod. Dep. Common, Highly Dep. No Effect on A or BModerate Effect on A, No Effect on BStrong Effect on A, No Effect on B Low Correlations btw REs Moderate Correlations btw REs High Correlations btw REs

14. Summary of Findings Correlations reflect the general ecological association between outcomes Within-study correlation was the main driver of the observed correlations But, even when we assumed independence within studies, correlations under or over-estimated based on commonness and dependence in events Dependence in treatment response was one of the least influential parameters Multivariate Normal model for marginal log-ORs not ideal to isolate dependence in treatment response.

15. Approximating Covariances Rationale: Within-study covariance matrices fixed to observed values to weight observations Covariance components rarely reported and are approximated or ignored S ABi ρ * × S Ai × S Bi What if we get ρ * wrong? Impact of Approximating or Ignoring Within-Study Covariances in Multivariate Meta-analyses

16. Simulations Simulate joint meta-analyses of two outcomes Generate patient level data so observed covariances (S ABi ) are known Compare accuracy and precision of results from meta-analyses with known covariances to those using approximations Overestimate by 50%: ρ * = 1.5 ρ Underestimate by 50%: ρ * = 0.5 ρ Ignore correlations: ρ * = 0* ρ Wrong direction of association: ρ * = -0.5 ρ Key Simulation Parameter: Relative size of between-patient and between-study variances and covariances.

17. Results: S i >> D -200 -150 -100 -50 0 50 100 150 200 Known Over-Est Under-Est Independent Negative Known Over-Est Under-Est Independent Negative Known Over-Est Under-Est Independent Negative Effect EstimatesBtw-Study VarianceCorrelation Proportional Bias (%)

18. Summary Model is generally robust to poor within- study approximations Particularly for effect estimates If only treatment effects are of interest, can do fairly well assuming independence within studies Estimates of correlations prone to large errors even when covariances known Bias larger when studies in the meta-analysis were small, D < S i, or D < S Implications: Can we trust the observed estimates of the correlations?

19. Future Research Joint analysis of related endpoints in a single study Same problem with correlations measured with patient-level data? New application: meta-analysis of composite endpoints Headache relief within 4 hours without adverse events with migraine therapy Proposed method: model joint (multinomial) distribution of components Joint analysis/inference for marginal and composite endpoints

20. Acknowledgements McGill Lawrence Joseph James Hanley United BioSource Jaime Caro Funding: NSERC, CIHR, FRSQ