1. Measuring Organizational Readiness for Quality Improvement Elizabeth Yano, PhD VA Greater Los Angeles HSR&D Center for the Study of Healthcare Provider Behavior UCLA School of Public Health

2. Perspectives Measuring organizational readiness for change –Traditional approaches (attitudes, beliefs, culture) –Expanding to clinical structure and care processes Examples of studies of organizational influences on quality –National VA reorganization (emphasizing PC) –VA quality improvement interventions (QUERI) Pre-implementation organizational assessment Post-implementation predictors of sustainability/spread Review key informant surveys as one method

3. Organizational Readiness Readiness represents a state of mind about the need for an innovation and the capacity to undertake change...Readiness represents a state of mind about the need for an innovation and the capacity to undertake change... Readiness consists of peoples beliefs, attitudes, and intentions about the desirability of changes, and perceptions about the ability of individuals and the organization to successfully make those changes.Readiness consists of peoples beliefs, attitudes, and intentions about the desirability of changes, and perceptions about the ability of individuals and the organization to successfully make those changes. Readiness represents the predisposition to unfreeze established patterns of behavior.Readiness represents the predisposition to unfreeze established patterns of behavior. Sources: Beer 1990, 2004; Beer & Eisenstat 1996, 2004; METRIC 2005

4. Organizational Readiness Commonly measured as: –Culture (innovativeness, flexibility, climate) –Definition of roles –Investment of resources (including competition) –Past contracts or agreements (can limit changes in behavior) –Threats to powercontrol over decisions, resources Sources: Beer 1990, 2004; Beer & Eisenstat 1996, 2004; METRIC 2005

5. Contribution of Organizational Readiness Measures to VA Quality Culture (staff surveys) –QI orientation associated with tobacco counseling rates Investments of resources/competition (PC leader survey) –Sufficiency of clinical support arrangements accounts for substantial variation in prevention performance Past contracts/agreements (PC leader survey) –Stringent PC patient assignment associated with lower breast & cervical cancer screening Control over decisions, resources (PC leader survey) –PC practice autonomy over internal operations associated with higher CRC screening rates and better diabetic control Sources: Yano et al., JGIM, 2002; Soban & Yano, JACM, 2005; Yano HCOC, 2005; Goldzweig et al, AJMC, 2004; Jackson, et al., AJMC, 2005.

6. Organizational Readiness: Expanding Beyond Attitudes, Beliefs, Culture Interventions associated with changes in quality –Organizational change (biggest effects) –Reminders –Audit and feedback/profiling –Incentives Organizational change interventions focus on structures and processes of care (org enablers) –Not attitudes, beliefs, culture – hard to change –Clinical processes of care, management strategies, clinic structure, provider training, decision support Source: Stone, et al., Ann Intern Med, 2002.

7. Major Reorganization of VA Health Care System (1996-present) Historically, VA was… –Individually-managed hospitals focused on specialty care –Funded through prior-year cost reimbursement –Extremely poor quality of care reputation Congressionally approved VA reorganization… –Veterans integrated service networks (VISNs) (n=22) –Network-level budget control and performance agreements –Incentivized audit-and-feedback on quality/satisfaction –Funded by population served (capitated) –Eligibility reform changed VA to health plan and payor –Computerized patient record system (CPRS) w/decision support –Primary care as platform for restructuring care delivery

8. VA Health Care System as the Organizational Context VA health care system largest in U.S. –163 hospitals, >750 freestanding community-based outpatient clinics, >130 nursing homes, >200 vet ctrs –Served over 5 million patients in past year Affiliated with over 1,200 academic institutions – –Including 107 medical schools, 55 dental schools – –110,000 students and trainees in >45 disciplines/year > ½ of all U.S. MDs received part of their training in VA VA market penetration growing rapidly –20% of those 65 VA health care budget $25.9 billion (2003)

9. VA Improves 12 of 13 Leading Quality Indicators (1995-2000) Jha NEJM 348:22 2003

10. VA Beats Medicare 12-1 in 2000 Jha NEJM 348:22 2003

11. Primary Care Organizational Changes PC Teams Pt assignment PC-based QI Separate PC budget Percent of VAs w/PC Program

12. Changes in PC Physician Volume/Mix Mean FTEEs

13. Staff Alignment to Primary Care Percent of VAs with PC Staff Reporting Only to Primary Care

14. Changes in PC Resource Sufficiency Percent of VAs Reporting Always/Mostly Sufficient * * * * *

15. Organizational Readiness: Implications for QI Research Basic Science Clinical Research Health Services Research Routine Care TRANSLATION new discoveries EFFICACY to EFFECTIVENESS new treatments new cures improved access better quality of care IMPLEMENTATION barriers TRIP over the proverbial brick wall VA intramural research program

16. Organizational Readiness: Implications for QI Research Health Services Research Routine Care Measure organizational readiness Use to select promising sites Locally tailor QI intervention(s) Fixed characteristics (eg, urban/rural) Mutable characteristics (change/adapt)

17. Organizational Readiness for QI: Preparing to Implement Depression Collaborative Care Depression collaborative care model in 7 VAs –Depression care manager –PC-MH collaboration –Informatics/registry (screening, reminders) –Leadership support Pre-intervention semi-structured telephone interviews of all PC and MH leaders –Assess current screening and referral processes –Assess local barriers (eg, turf, staffing gaps, history) –Fed back into planning calls, adapted protocols Sources: Parker LE, Yano EM, Rubenstein LV, 2003; Ficket et al, in prep.

18. Organizational Readiness for QI: Preparing to Implement HIV QI Intervention HIV quality improvement intervention trial (16 VAs) (Asch et al) –Group-based QI, audit-and-feedback, reminders Used national HIV provider survey (n=118 VAs) to examine how organizational factors affected adoption of HIV QI activities before starting trial –Assessed local QI activities, HIV guideline use –Measured attitudes toward proposed QI modalities –Evaluated regional, facility and practice variations –Fed back to team (site selection, adapted protocols) Sources: Anaya, Am J Med Qual 2004; Korthuis et al, JAIDS, 2003; Yano, et al., Mil Med 2005.

19. Organizational Readiness for QI: Implementing/Sustaining Depression Collaborative Care Onsite in-person stakeholder interviews –Network, medical center, clinic site –PC and MH leaders, PC and MH providers, nurses, care managers, patients, consumer reps (n=106) Semi-structured interviews exploring implementation of each care model component –Leadership support/opinion leaders –Depression care manager interaction/contacts –Provider interactions and ongoing education needs Fed back to implementation/spread teams and developing diffusion tools

20. Different Measurement Approaches Knowledge/ Evidence Base Qualitative Interviews Key Informant Surveys Provider Surveys HIGH LOW Informs survey design know domains/items? who has knowledge? can you get to them? will they cooperate? if variation important if QI intervention requires their change AND all of above telephone or in-person different levels of interview structure different levels of stakeholders/informants

21. Measuring Organizational Characteristics Using Key Informant Surveys Step #1: Translate ideas into survey domains Example: Translate HIV QI strategic plan into domains: –HIV screening policies and protocols –Practice arrangements for management of HIV disease –Provider ratings of effectiveness of diff QI interventions –Potential barriers to adoption of HIV guidelines Example: Evaluate PC organizational predictors of quality –Institute of Medicine primary care domains (access, continuity…) –Primary care strategic plans –PC practice managers (observation and interview) Example: Depression collaborative care implementation –Disaggregate care model componentsexplicitly open black box

22. Measuring Organizational Characteristics Using Key Informant Surveys Step #2: Select measures allowing benchmarking against other health care organizations Example: VA QUERI HIV & HCSUS Example: VA, NCQA PSAS, & Kaiser Example: VA & DHHS Office of Womens Health COEs Step #3: Develop new measures that match structure-process or -outcome model or QI goals Literature review, expert panel methodsbuild on evidence Talk to real people who live in world you are studying Begin with qualitative interviews or focus groups

23. Measuring Organizational Characteristics Using Key Informant Surveys Step #4: Test and adapt survey to target health care settings Cognitive interviews with sample respondents Vary types of organizations included (big/small) Develop multiple modules if needed –By setting (hospitals, freestanding outpatient clinics) –By respondent type (hosp director, PC chief, lab tech) Step #5: Identify key informants Desired knowledge base, incentives to participate Social desirability and need for validation, politics

24. Measuring Organizational Characteristics Using Key Informant Surveys Step #6: Sampling organizations Whats in it for the organization? Sampling to represent what? –Types of organizations, units w/in organizations, different size and complexity organizations, different size and complexity –Regions, urban/rural locations, other area characteristics Obtaining their cooperation… –Leadership support, uses of data, IRB, HIPAA –Funding to compensate for administrative time Where do you get this kind of information?

25. Measuring Organizational Characteristics Using Key Informant Surveys Step #7: Field preparation Identify and market to venues common to respondents Determine regular communication options –Management meetings, conference calls, broadcast fax, advance mass mailing Involve senior leaders/opinion leaders – spokespersons Market value of participation –Demonstrate previous uses of data (good works) –Offer incentive (eg, summary of survey results, publications order form, financial) Contact organizations and talk to support staff

26. Measuring Organizational Characteristics Using Key Informant Surveys Step #8: Administer surveys Hardcopy express-mail with prepaid returns Web-based with varying email introductions and reminders Quality review of survey content with active follow-up of missing data and non-respondents –2-week second wave mailouts –4-week telephone follow-up Continual data entry (if hardcopy), quality checking Follow-up postcards and repeat announcements in original venues

27. Example Key Informant Surveys Primary care practice organization (93, 96, 99) –Care arrangements, teams, staffing, authority, resources, QI, decision support, care coordination, profiling, incentives, management structure HIV practice structure/delivery models (00) –Screening, PC vs. specialty management, HIV clinics, staffing, provider experience, HIV case management, HIV guideline use, barriers, provider preferences Womens health care delivery organization (01) –Clinic structures, service availability, referral arrangements, decision support, QI, leadership, policies, staffing, authority, provider training 93%- 100% 83% 82%- 100%

28. Example Key Informant Surveys Clinical practice organizational survey (05) –3 modules: Network directors (n=21) (~$1 billion each) Chiefs of staff (aka medical directors) (n=160) Primary care clinic directors (n=259) –Mapped to prior VA surveys time trends –Mapped to NCQA PSAS © and Kaiser IT surveys benchmarking Womens primary care organizational survey (06) –Senior WH clinician or medical director –Classify every VA by organizational taxonomy –Evaluate quality of care within different types

29. Thank you Elizabeth.yano@med.va.gov (818) 895-9449