1. Working Group: Can Six Blind Men Find Apples & Oranges? Measuring Variable Implementation of QI Interventions Using Multiple Data Sources

2. Presenters Alexander S. Young, MD, MSHS Elizabeth (Becky) Yano, PhD, MSPH Lisa V. Rubenstein, MD, MSPH Alison Hamilton, PhD

3. Overview 90 minutes: presentations 60 minutes: group discussion and breakout groups 30 minutes: group consensus on priorities, suggested next steps, directions Working group moves to plenary – 5 minute summary presented

4. Overview of Working Group Presentations – Introduction and overview (Alex) – QUITS smoking cessation trial (Becky) – TIDES depression collaborative care (Lisa) – EQUIP evidence-based practice in schizophrenia (Alison)

5. Goal of Presentations Brief orientation to example QI intervention How context matters  sets us up for variable QI intervention deployment Process for intentional adaptation of evidence into context of local practice Types of data sources brought to bear on measuring implementation – including development of a fidelity score Triangulation of data sources to tell story

6. QI Intervention (QII) Examples QUITS (Quality Improvement Trial for Smoking cessation) – evidence-based quality improvement to implement smoking cessation guidelines – Scott Sherman MD & Becky Yano PhD (co-PIs) TIDES (Translating Interventions for Depression into Evidence-based Solutions) – depression collaborative care model – Lisa Rubenstein MD & Ed Chaney PhD (co-PIs)

7. QI Intervention Examples (cont’d) EQUIP (Enhancing QUality of care In Psychosis) – evidence-based quality improvement to implement effective treatments in schizophrenia – Alex Young MD & Amy Cohen PhD (co-PIs)

8. QUITS Smoking Cessation Trial EBQI Education “toolkit” Local QI plan development Expert review/feedback Performance feedback Leadership support “local buy-in” “priority-setting” Evidence base: SC clinic referrals Tobacco quitlines PC-based intn’s

9. TIDES Depression Collaborative Care EBQI Provider/patient education Depression care manager QI Informatics support Performance feedback Leadership support “adaptation” “priority-setting” Evidence base: >20 RCTs Depression toolkit

10. EQUIP Effective Schizophrenia Care EBQI Provider/patient education Quality manager QI Informatics support Performance feedback Leadership support “infrastructure” “priority-setting” Evidence base: TMAP EQUIP-1

11. Context Matters: Design for It TIDES – 2:1 intervention-to-control sites x 3 VISNs (6 intervention + 3 control sites total) – VISN leaders chose sites, we randomized within network (block on network characteristics) QUITS – regional concentration in southwest (3 VISNs) – matched on size/academic affiliation within VISN – we chose sites and randomized within network EQUIP – 1:1 intervention-to-control sites x 4 diverse VISNs – sites chosen based on leadership interest

12. Context Matters: Input from Sites Attitudes / beliefs / experiences – perceived need for the intervention – competing demands – staff openness to innovation Resources – perceived time to use program and participate in implementation – organizational structure, staffing, prior QI experience, informatics support Source: Kirchner JE, Parker LE, Yano EM, COVES evaluation (2007).

13. Multiple Data Sources: Measuring Implementation TIDESQUITSEQUIPExamples Semi-structured interviews: leaders, clinicians, mgrs -- participation, level of implementation Organizational site surveys: PC/MH leaders, SC mgrs clinic structure, processes, change Field journals -- group-level dynamics, implementation details Administrative data visits, Rxs, costs Patient surveys PHQ9, BASIS, quits Clinician surveys & activity logs knowledge, attitudes, behaviors Practice checklists (experts) -- -- QII components

14. Multiple Data Sources: Strengths and Challenges StrengthsChallenges Semi-structured interviews: leaders, clinicians, mgrs rich data, diverse perspectives expensive, time- consuming Organizational site surveys: PC/MH leaders, SC mgrs site profiles, fast, easier to analyze limited discovery, key informant view Field journalsdetailed contextual data variation between observers Administrative datareadily available, historical value not QII-specific, local coding differences Patient surveysexperience, exposure, outcomes expensive, sensitive to sample Clinician surveys & activity logsimplementers!difficult responders Practice checklists (experts)face validity, eyes on the ground variation between observers

15. Triangulation Critical to collect information about implementation from multiple sources – be prepared for disagreement – perspectives and opportunities for observation differ for managers, providers vs. patients Recognize differences between “exposed” sample and practice population – does the “enrolled” group represent the practice? – did the intervention penetrate among all providers?