1. Using the Critical Incident Technique to Better Understand Patient Experiences of Ambulatory Care Presented by: Kristin L. Carman, Ph.D. American Institutes for Research Presented at: Academy for Health Services Research Annual Meeting, San Diego, CA June, 2004

2. Project Team Roger Levine, PhD Managing Research Scientist, AIR Karen K. Shore, PhD Senior Social Scientist Margarita Hurtado, PhD Principal Research Scientist Kristin L. Carman, PhD Principal Research Scientist Judy Mitchell, MS Senior Research Scientist Steven A. Garfinkel, PhD Managing Research Scientist San Keller, PhD Principal Research Scientist Funding Agency for Healthcare Quality and Research Part of the CAHPS II grant Partners HMSA, Hawaii Humana, Chicago

3. Purpose of our project To develop an A-CAHPS Survey To use Critical Incident data in novel ways to address issues related to: Instrumentation Quality improvement Reporting, and Cultural comparability

4. Critical Incident (CI) Technique Critical Incident Incident=an observable, specific behavior Critical=means incident was crucial to the outcome of interest Organized structure for data collection; focuses on observable behavior Used to collect and analyze reports of behaviors associated with specific outcomes Qualitative method; in-depth interviews

5. Methods and data 200 interviews; 40 providers and 260 patients Patient respondents divided equally among four different racial/ethnic groups Open-ended responses are transcribed Each interview usually generates 10 incidents; well have 2000+ incidents

6. CI data management and processing Gathering extensive data Using qualitative software to create and manage a very complex data base Developed a very careful data processing protocol Raw data is open-ended responses, transcribed Data is transformed into incident write ups These two types of data are the sources for analyses

7. Specific Goals

8. Instrumentation goals Develop a complete taxonomy of the components of quality ambulatory health care, based on both patient and clinician perspectives Confirm that the domains measured by the draft CAHPS® instrument are salient to patients and providers and can be assessed by patients Determine whether the domains are salient and measurable for both men and women, for individuals with different levels of education, and across a range of racial and ethnic groups Identify additional domains that should be measured

9. Instrumentation goals (contd) Identify CAHPS® item content that can result in a spread of scores at the high end of the score distribution to minimize ceiling effects Generate objective patient reports of health care experience Create items for Ambulatory CAHPS® for which a positive rating would be rare

10. Instrumentation analysis Developing the taxonomy Randomly select at least 200 incidents Two teams classify incidents into major categories, then subcategories Iteratively validate and refine the taxonomy with additional set of incidents

11. Instrument analysis (contd) Using incidents as the unit of analysis, then respondents, we investigate statistical associations between personal characteristics and the way people conceive of quality of care Tabulation of respondent characteristics by taxonomy categories which are quality of care themes (e.g., coordination of care) Regress taxonomy categories on respondent characteristics

12. Instrument analysis (contd) Logistic Regression Simple and multiple Dependent Variable=taxonomy category Independent Variables= Individual demographics Respondent: provider or patient

13. Quality improvement goals Identify physician behaviors associated with excellent and poor quality of care based on experiences of patients and clinicians Identify combinations or co-occurrences of behavior Identify key facilitators and barriers to quality of care, specifically related to CAHPS domains Create tools for QI interventions to improve CAHPS scores

14. Quality improvement analysis Analyze interview and CI files Conduct additional coding of data Focus on which behaviors or actions by clinicians co-occur to create positive (or negative) experiences for patients Compare findings by respondent characteristics

15. Cultural Comparability goals Identify variations in taxonomic structure for different racial/ethnic groups If differences exist, identify the implications for: Supplemental domains, concepts and items CAHPS domain labels and explanatory vignettes Culturally appropriate interventions to improve care

16. Reports goals and analysis Identify narratives (phraseology) that clearly and effectively explains CAHPS measures in reports

17. Next steps Complete interviews Complete analyses Disseminate findings

18. For more information, contact: Kristin L. Carman, PhD Principal Research Scientist American Institutes for Research 1000 Thomas Jefferson Washington, DC 20007 (202) 342-5090 Kcarman@air.org www.air.org Karen K. Shore Senior Research Scientist American Institutes for Research ADDRESS Palo Alto, CA (ZIP) PHONE NUMBER kshore@air.orgshore@air.org www.air.org

19. [CI 1-2] [CI 1-3] [CI 2-1] [CI 2-2] [CI 3-1] [CI 3-2] [CI 1-1] Episode 1 Episode 3 Episode 2 Critical Incident behaviors CI 1-1 Critical Incident Forms CI 1-2 CI 1-3 CI 2-1 Verbatim Transcript Marked by interviewer for Episode # and for CI behaviors (not numberedonly numbered here for illustration) in Atlas/ti CI Forms filled out by cut and paste from Transcript, numbered to reflect source episode. This takes place in Word CI 1-1 CI 1-2 CI 1-3 CI 2-1 CI forms cut into separate text files for sorting into Taxonomy Links are logical only: episode # in CI # allows analyst to trace back to transcript TranscriptCIs for Interview Separate CIs Database Creation

20. [CI 1-2] [CI 1-3] [CI 2-1] [CI 2-2] [CI 3-1] [CI 3-2] [CI 1-1] Episode 1 Episode 3 Episode 2 CI 1-1 CI 1-2 CI 1-3 CI 2-1 At this stage, the finalized concatenated CI file, with taxonomy codes embedded, is imported into Atlas/ti. Taxonomy codes are autocoded by searching for Taxonomy codewords. At this point, hypertext links can be created between CI forms and episodes in the transcript. If desired, Taxonomy codes may be applied manually to the transcript. P-01-M-JM-1-1-A TAX-Communicates P-01-M-JM-1-2-B TAX-Clarifies The ID code includes #s for episode and CI. Since ID #s include identifiers for referents, they are assigned to demographic code families for referents (See next page for detail) TranscriptConcatenated CIs Final Atlas/ti Database Both transcript and concatenated CI files are assigned to doc families for respondent demographics (See next page for detail) Taxonomy codes are assigned by autocoding

21. Respondent demographics are represented by placing primary documents in PD Families or sets: Example: Gender::Male = {PD1; PD4; PD5; PD8…} Gender::Female = {PD2; PD3; PD6; PD7…} Age::20s = {PD1; PD2…} Age::30s = {PD4; PD8…} Age::40s = {PD3…} Age::50s = {PD6…} Age::60s = {PD5; PD7…} Imported in a table: GenderAge PD1Male20s PD2Female20s PD3Female40s PD4Male30s PD5Male60s PD6Female50s PD7Female60s PD8Male30s … Dataset can be parsed according to Boolean combinations of set-memberships, for example, to restrict a query to documents that belong in both the Male and 20s sets. Referent demographics are represented by placing CI ID codes into Code Families or sets. The logic is the same as for PD families, but at present there is no table import feature and assignment is made with the code family manager tool in Atlas/ti. Again, set memberships can be used to focus queries on different classes of referents. RESPONDENT DEMOGRAPHICS REFERENT DEMOGRAPHICS Strategies for associating data at episode or interview levels Layered IDs: Since IDs are layered, a hierarchy of ID codes can be created in Atlas/ti. For example, a hierarchy could be structured as follows: Interview ID +--Episode ID +--CI ID +--Referent ID Code Families: A code family that included all the CI ID codes for a given episode would enable searching by episode in the concatenated CI files. Hypertext: Hypertext links can be created from the CI forms to the episodes in the transcript, or even to the specific descriptions of behaviors from which the CIs are derived.