1. Heterogeneity is not always noise Frank Davidoff 29 March 2012

2. Heterogeneity Heterogeneity is not always noise 2 Composition from diverse elements or parts; multifarious composition Oxford English Dictionary

3. The Heterogeneity Problem Heterogeneity is not always noise 3 Heterogeneity: You can’t live with it, and you can’t live without it

4. Today’s territory Heterogeneity is not always noise 4 How heterogeneity interferes with causal inference in clinical science How heterogeneity also deepens our knowledge How the effects of heterogeneity play out differently in improvement science How we can begin to manage the effects of heterogeneity

5. Heterogeneity is not always noise 5 Benefit from Drug X: treated population Results from a standard clinical trial in “ICA” patients RCT Rx benefit: ARR 2 percentage points (pp)

6. Heterogeneity of treatment effect: main sources Heterogeneity is not always noise 6 Variation in outcome risk when the primary disease is untreated (mainly biological and behavioral variation) Treatment-related harm Competing risk Direct treatment-effect modification

7. How summary results of trials can be misleading Hypothetical example Heterogeneity is not always noise 7 Control - event rate/100 Rx - event rate/100 RRRARR – PP (Prob) NNT Overall result860.252 (0.02)50 Average risk subjects430.251 (0.01100 High risk subjects80600.2520 (0.20)5 Modified from Kent et al, Trials 2010;11:85

8. Major differences in therapeutic benefit Low- vs. high-risk subgroups in risk stratified analysis Heterogeneity is not always noise 8 Surgery for carotid stenosis Anticoagulation in non-valvular atrial fibrillation CABG for coronary artery disease Statin therapy as primary prevention in coronary disease Invasive and non-invasive therapies for acute coronary syndromes tPA and PCI in ST-elevation myocardial infarction Drotrecogin in severe sepsis Kent et al, Trials, 2010:11:85

9. Heterogeneity is not always noise 9 Benefit from Drug X: high-risk patient subgroup Risk-stratification of results from clinical trial in “ICA” patients RCT ARR 2 pp Risk stratification Rx benefit: ARR 20 pp

10. Heterogeneity is not always noise 10 Benefit from Drug X: individual high-risk patient Real world results in a “usual” local care system RCT ARR 2 pp Risk stratification ARR 20 pp General Hospital - Admin rate 40% Rx benefit: ARR 8 pp

11. Heterogeneity is not always noise 11 Benefit from Drug X: individual high-risk patient Real world results in a local care system that successfully supports changes RCT ARR 2 pp Risk stratification ARR 20 pp Community Hospital – Admin rate 95% Rx benefit: 19 pp QI Program ???

12. Heterogeneity is not always noise 12 Benefit from Drug X: individual high-risk patient Real world results in a local care system that has trouble supporting changes RCT ARR 2 pp Risk stratification ARR 20 pp Proprietary Hospital – Admin rate 60% Net benefit: 12 pp QI program ???

13. Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 13 Improvement interventions: Consist of multiple components: hard to standardize; easily mixed and matched

14. A multi-component improvement intervention: The Michigan ICU central line infection control study Heterogeneity is not always noise 14 In addition to introducing checklists, prep carts, new skin antiseptic, organizers and leaders: Recruited advocates within the organization Kept the team focused on goals Created alliances with central administration to secure resources Shifted power relations (particularly with nurses) Developed social and reputational incentives for cooperating Opened channels of communication with units that face the same challenges Used audit and feedback

15. Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 15 Improvement interventions: Consist of multiple components: hard to standardize; easily mixed and matched Must first be absorbed and adapted: they change in the process (also easily shared, spread)

16. Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 16 Improvement interventions: Consist of multiple components: hard to standardize; easily mixed and matched Must first be absorbed and adapted: change in the process (also easily shared, spread) Are context-dependent: context can’t be “controlled out”

17. Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 17 Improvement interventions: Consist of multiple components: hard to standardize; easily mixed and matched Must first be absorbed and adapted: they change in the process (also easily shared, spread) Are context-dependent: context can’t be “controlled out” Are unstable by design: refined over time in response to feedback (“reflexiveness”)

18. Change factor analysis: detail-level An “ex post” theory of a quality improvement program: Michigan study Heterogeneity is not always noise 18 Isomorphic (peer) pressure applied to join the project Networked community formed with strong horizontal links Bloodstream infections reframed as a social problem Interventions used to shape a “culture of commitment” Data harnessed as a disciplinary force “Hard edges” used Dixon-Woods et al, Milbank Quarterly, 2011;89:167-205

19. Change factor analysis: mid-level Improving survival after acute myocardial infarction: (AMI) Heterogeneity is not always noise 19 Organizational values and goals Senior management involvement Broad staff presence and expertise in AMI care Communication and coordination among staff groups Support for staff problem solving and learning Curry LA, et al. Ann Intern Med 2011;154:384-90

20. Change factor analysis: high-level In-depth field studies in 9 US/UK hospitals Heterogeneity is not always noise 20 Six universal challenges: structural, cultural, political, educational, emotional, physical & technological Single factor (even “dominant” set) rarely explains “heterogeneity of improvement effect” Answers lie in interactions among a multiplicity of factors Quality a multi-level phenomenon “Universal but variable” thesis: six challenges same everywhere, but specifics vary within them – the “cityscape phenomenon” Bate P, et al. Organizing for Quality, 2008

21. SUMMARY Heterogeneity is not always noise 21 Heterogeneity is everywhere in medicine Interferes with detection of causal relationships  noise BUT Also key source of information regarding individual risk and outcome  signal

22. CONCLUSIONS Heterogeneity is not always noise 22 In order to use heterogeneity as a source of knowledge In clinical science Need better techniques for understanding effects of biological and behavioral variation on clinical outcomes In improvement science Need better techniques for understanding effects of social factor variation on performance change outcomes Everyday challenge for everyone Observe, record, reflect, model, share: you might just come up with the techniques we need

23. REFERENCES Heterogeneity is not always noise 23 Davidoff F. Heterogeneity is not always noise. JAMA 2009;302:2580-6. Kent DM, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010;11:85 Provost L. Analytical studies: a framework for quality improvement design and analysis. BMJ Qual Saf 2011;20 [Suppl 1]:i-92-i96. Dixon-Woods M, et al. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011;89:167-205. Kaplan HC et al. The Model for Understanding Success In Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf 2012;21:13-20. Curry LA., et al. What distinguishes top-performing hospitals in acute myocardial infarction mortality rates. Ann Intern Med 2011;154:384-90. Bate P, et al. Organizing for Quality. 2008; New York: Radcliffe Publishing

24. ACKNOWLEDGMENTS For helpful comments on this presentation Heterogeneity is not always noise 24 Yale-New Haven Hospital medical directors leadership council SQUIRE development group: David Stevens, Paul Batalden, Greg Ogrinc Mary Dixon-Woods Jane Roessner Jules Hirsch