Heterogeneity is not always noise Frank Davidoff 29 March 2012

Heterogeneity Heterogeneity is not always noise 2 Difference or diversity in kind from other things Oxford English Dictionary

The Heterogeneity Problem Heterogeneity is not always noise 3 Heterogeneity: You cant live with it, and you cant live without it

Heterogeneity is not always noise 4 Benefit from Drug X: entire treated population Results from a standard clinical trial RCT Rx benefit: 20%

Heterogeneity of treatment effect: main sources Heterogeneity is not always noise 5 Variation in outcome risk when the primary disease is untreated (mainly biological and behavioral variation) Treatment-related harm Competing risk Direct treatment-effect modification

How summary results of trials can be misleading Heterogeneity is not always noise 6 Control - event rate Rx - event rate RRRARRNNT Overall result Average risk subjects High risk subjects Kent et al, Trials 2010;11:85

Heterogeneity is not always noise 7 Benefit from Drug X: high-risk patient subgroup Results from a risk-stratified clinical trial RCT Risk stratification Rx benefit: 50%

Heterogeneity is not always noise 8 Benefit from Drug X: specific high-risk patient Real world results in a usual local care system RCT Risk stratification Hospital P Rx benefit: 15%

Heterogeneity is not always noise 9 Benefit from Drug X: specific high-risk patient Real world results in a local care system that successfully supports changes RCT Risk stratification Hospital Q Rx benefit: 47.5% QI Program Change factor stratification

Heterogeneity is not always noise 10 Benefit from Drug X: specific high-risk patient Real world results in a local care system that has trouble supporting changes RCT Risk stratification Hospital R Net benefit: 25% QI program Change factor stratification

Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 11 Improvement interventions: Consist of multiple components: hard to standardize; get mixed and matched

A multi-component improvement intervention: the Michigan central line infection study Heterogeneity is not always noise 12 Introduce checklists, prep carts, new skin antiseptic Recruit advocates within the organization Keep the team focused on goals Create alliances with central administration to secure resources Shift power relations (particularly with nurses) Create social and reputational incentives for cooperating Open channels of communication with units that face the same challenges Use audit and feedback

Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 13 Improvement interventions: Consist of multiple components: hard to standardize; get mixed and matched Must first be absorbed and adapted: change in the process (easily shared, spread)

Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 14 Improvement interventions: Consist of multiple components: hard to standardize; get mixed and matched Must first be absorbed and adapted: change in the process (easily shared, spread) Are context-dependent: context cant be controlled out

Heterogeneity of improvement effect: main sources Heterogeneity is not always noise 15 Improvement interventions: Consist of multiple components: hard to standardize; get mixed and matched Must first be absorbed and adapted: change in the process (easily shared, spread) Are context-dependent: context cant be controlled out Are unstable, by design: evolve in response to feedback (reflexiveness)

Change factor stratification: facilitators An ex post theory of a quality improvement program Heterogeneity is not always noise 16 Isomorphic pressures to join the project Networked community with strong horizontal links Reframing bloodstream infections as a social problem Using interventions to shape a culture of commitment Harnessing data as a disciplinary force Using hard edges Dixon-Woods et al, Milbank Quarterly, 2011;89:

Change factor stratification: facilitators The Model for Understanding Success in Quality (MUSIQ) Heterogeneity is not always noise 17 Twenty-five contextual factors including: -External environment -Organization -QI support and capacity -Microsystem -QI team -Miscellaneous Kaplan et al., BMJ Quality and Safety, 2012; 21:13-20

Change factor stratification: obstructers Resisters and constipators affect infection prevention efforts Heterogeneity is not always noise 18 Active resisters: - Hospital personnel who actively and openly oppose various changes in practice Organizational constipators: - Mid- to high-level executives who prevent or delay change actions through passive resistance or control Saint, et al, Jt Comm J Quality and Safety, 2009; 35:239-46

Change factor stratification: obstructers Understanding rapid response systems Heterogeneity is not always noise 19 Boundaries between senior and junior staff: - tend to reinforce existing power relations Standardization of practice (including use of Early Warning Systems): - tend to attenuate staff authority Downstream response behavior, including handovers and referral across medical teams: - are more problematical than early steps in response Mackintosh et al, BMJ Quality and Safety, 2012:21:135-44

SUMMARY Heterogeneity is not always noise 20 Heterogeneity is everywhere in medicine It interferes with detection of causal relationships (noise) AND Is a vital source of information on individual risk (signal) We need to use heterogeneity as a source of knowledge More techniques like risk stratification, that can expose the effects of biological and behavioral variation on clinical outcomes Better methods for exploring the effects of variation in social change factors on the outcomes of improvement interventions