1. ®® Developing the CAHPS for Cancer Care Survey Steven Garfinkel, American Institutes for Research Kathleen Yost, Mayo Clinic AHRQ Annual Conference Bethesda, Maryland September 10, 2012 Project Cosponsored by the National Cancer Institute and the Agency for Healthcare Research and Quality under AHRQ Contract# HHSA290200600019I, Task Order# HHSA29032003T TO#9 Steven Garfinkel, American Institutes for Research Kathleen Yost, Mayo Clinic AHRQ Annual Conference Bethesda, Maryland September 10, 2012 Project Cosponsored by the National Cancer Institute and the Agency for Healthcare Research and Quality under AHRQ Contract# HHSA290200600019I, Task Order# HHSA29032003T TO#9

2. ®® Why Develop a CAHPS for Cancer Care Survey? Uniformly measure cancer care by providers from a patient perspective Compare the care provided by different providers Provide information for quality improvement

3. ®® CAHPS Development Process Call for Measures Inventory items Focus groups Expert review Formative Research Draft domains Draft items Cognitive testing Expert review Revise survey Develop Draft Field test Analyze data Test report domains Expert review Finalize survey Disseminate Test and Finalize

4. ®® Formative Research

5. ®® Item Inventory 1781 items  CAHPS surveys  Cancer surveys All items coded into hypothesized domains  e.g., affective communication, shared decision- making Three independent reviewers of initial domains and items

6. ®® Focus Groups 16 focus groups to assess how consumers identify cancer providers and quality of care Stage 0/I (solid and common tumors) Stage II/III (solid and common tumors) Stage IV (any cancer) Rare cancers Hematologic cancers Spanish speaking Family and Caregiver

7. ®® Focus Group Topics Who are the key providers that patients focus on when they think about quality of care? What are key factors that patients use to determine quality of care? Are domains developed by the team relevant to patients? Which domains are most important?

8. ®® Focus Group Findings Cancer care providers  Oncologists and surgeons most often  Other clinical staff: nurses, nurse practitioners, physician assistants, primary care physicians, and technicians  Non-clinicians: support groups, family members, chaplains, and even a therapy dog Perceived care coming from a provider and a team or teams

9. ®® Focus Group Findings, continued Top two domains of quality:  How providers interact with patients “They care about you. There’s nobody else in the world but you.” “He’s always been straightforward. He didn’t sugarcoat.”  Technical communication skills “My doctor explained to me that chemo is very hard on the body. He didn’t want to jump in and do the chemo. When he found another lump in my neck then he said we have to treat it.”

10. ®® Focus Group Findings, continued Consumers also interested in  Care coordination  Access to information  Providers who are up-to-date “There is so much research and so many new drugs coming out that you have to be up on it and pay attention to it because the more targeted the chemo the better.”  Shared decision making

11. ®® Stakeholder Interviews Technical Expert Panel of clinicians and patients Interviews with representatives of 5 clinical professional organizations Interviews with representatives of 5 community cancer centers Discussions with Mayo Clinic Oncology staff

12. ®® Stakeholder Findings Most cancer care is delivered through cancer centers  Over 1,500 accredited by ACS Commission on Cancer Survey will be most useful if it provides scores by major treatment modalities:  Medical, radiation, and surgical oncology

13. ®® Cognitive Testing Detects unexpected problems Provides an underlying model of the questionnaire item response process

14. ®® Cognitive Testing Findings Need to define “team” Medical oncology = drug therapy Specify “cancer” surgery to distinguish focal surgery from surgery for other conditions received by cancer patients

15. ®® Major Design Decisions Use the cancer center as the unit to assess Focus on post-diagnosis and active treatment Include inpatient and ambulatory experiences Use 3-month reference period Sample equally from medical, radiation, and surgical oncology patients Parallel questionnaires for medical, radiation, and surgical oncology

16. ®® Example Item In the last 3 months, how often did your radiation therapy team listen carefully to you? In the last 3 months, how often did your cancer surgery team listen carefully to you? In the last 3 months, how often did your drug therapy team listen carefully to you?

17. ®® Field Test

18. ®® Field Test Activities 1. Site recruitment 2. Eligibility and Sampling 3. Response rate and response bias analysis 4. Evaluation of mode effect 5. Comorbidity Index Study 6. Item functioning 7. Analysis to determine structure of composites 8. Case-mix adjustment

19. ®® Field Test Design and Preliminary Results on Survey Operations

20. ®® Purpose of Field Test To develop an assessment of the quality of cancer care that would obtain the CAHPS trademark  Assess psychometric properties of survey Item non-response (including failed skip patterns) Mode effects (mail vs. phone) Composites Case-mix adjusters  Assess ability of survey scores to discriminate quality of cancer care across sites  Evaluate feasibility of administering the survey in different treatment settings Challenges of enumerating a sample Unit non-response

21. ®® Site Recruitment Initially intended to work with a clinical trials cooperative group American College of Surgeons Commission on Cancer (CoC) expressed interest in possibly sponsoring the survey  6 CoC member organizations were recruited  Large enough case volume to provide 750 eligible patients (250 per treatment modality) within data collection timeframe 300 completes per site (100 per treatment modality) Assumed 40% response rate  Geographic representation Sampling frames sent to Mayo Clinic Survey Research Center  Waiver of written HIPAA authorization at 5 sites  Exception of Site 3

22. ®® Site recruitment Site Type National Cancer Institute (NCI) designation Site 1Large, academicComprehensive Cancer Center Site 2Large, academicComprehensive Cancer Center Site 3Large, community-based NCI Community Cancer Centers Program (NCCCP) Site 4Medium, community-basedNCCCP Site 5Healthcare system comprising 5 hospitals Non-NCCCP Site 6Large, academicComprehensive Cancer Center

23. ®® Patient Eligibility Age 18 or older at the time of the recruitment material mailing Received treatment for cancer in the 10 weeks prior to being selected for the sampling frame.  Treatment setting can be inpatient or outpatient.  Treatment includes surgery, medical oncology and radiation therapy.  Medical oncology includes chemotherapy, hormonal therapy, immunotherapy, and any other treatment for cancer delivered as a drug.  Treatment intent can be curative or palliative  For radiation and medical oncology care, an additional criterion is that the patient had at least two visits to your organization for cancer treatment in the previous 10 weeks. In one of the following three phases of the cancer control continuum: receiving treatment for a new diagnosis, receiving treatment for a recurrence, or completed cancer treatment within the sampling window and is transitioning to survivorship Not receiving care for squamous cell or basal cell skin cancer.

24. ®® Data Collection Mayo Clinic Survey Research Center Mail arm  Goal – 1500 completes  Initial recruitment packet with letter and survey  Second mailing to non-respondents after 3 weeks  3 attempted reminder phone call to non-respondents after 3 weeks No survey administration by phone if contact was made Phone arm  Goal – 300 completes  Prenotification letter  Phone interviews begin 2 weeks after prenotification  6 attempts – 2 weekday day, 2 weekday night, 2 weekend across 2 weeks Embedded study of comorbidity questions Double data-entry

25. Data Collection Protocol for All Sites

26. ®® Misidentified Cases Site 2  2 cases in cancer surgery arm self-identified as ineligible – did not receive surgery for treatment of cancer  Unanticipated Problems Involving Risk to Subjects or Others (UPIRTSO) Unanticipated risk of emotional harm  Suspension of recruitment of cancer surgery patients at Sites 1 and 2 pending chart review  Chart review found 1 additional case  Follow-up protocol not completed at Site 2 due to IRB delay Sites 3 – 6 changed IRB protocols to acknowledge misidentification as an anticipated risk

27. ®® Eligibility Determination Among Respondents Q1 Response Time since service < 100 days Time since service > 100 days Missing, and entire survey left blank Refusal 0 Refusal 0 Missing, but quality of care questions answered Eligible 2 Ineligible 1 Yes Eligible 656 Eligible 120 No Eligible 12 Ineligible 30

28. ®® Response Rate Response Rate (%)p value Overall (N=1083, not counting deceased/ineligible) 64.1 By Mode 0.64 1 Mail (N=956)64.3 Phone (N=127)62.2 By Treatment 0.11 1 Drug (N=492)63.4 Radiation (N=353)68.0 Surgery (N=238)59.7 1 Chi-Square

29. ®® Distribution of Demographic and Cancer Type Among Responders and Nonresponders Response Bias Omitting observations from Site 2/Surgery due to break in protocol Nonresponders (N=389) Responders (N=694)p value Sex 0.90 1 F212 (54.5%)381 (54.9%) M177 (45.5%)313 (45.1%) Age 0.04 2 N389694 Mean (SD)61.4 (13.6)63.1 (12.5) Median62.263.9 Q1, Q353.3, 70.956.0, 72.0 Range(18.7-91.9)(18.4-91.0) Disease type 0.63 1 Malignant neoplasm of lip, oral cavity, and pharynx 12 (3.1%)22 (3.2%) Malignant neoplasm of digestive organs and peritoneum 70 (18.0%)112 (16.1%) Malignant neoplasm of respiratory and intrathoracic organs 35 (9.0%)59 (8.5%) Malignant neoplasm of bone, connective tissue, skin, and breast 113 (29.0%)240 (34.6%) Malignant neoplasm of genitourinary organs 61 (15.7%)113 (16.3%) Malignant neoplasm of other and unspecified sites 40 (10.3%)56 (8.1%) Malignant neoplasm of lymphatic and hematophoietic tissue56 (14.4%)87 (12.5%) Neuroendoctrine tumors2 (0.5%)5 (0.7%) 1 Chi-Square 2 Kruskal Wallis

30. Quality of Care Questions That Differ by Mode of Survey Administration Mail (N=697) Phone (N=79)p value How much did a Dr/HCP talk about other ways to treat your cancer 0.003 1 Missing91 A lot215 (31.3%)32 (41.0%) Some224 (32.6%)10 (12.8%) A little91 (13.2%)10 (12.8%) Not at all158 (23.0%)26 (33.3%) How much did a Dr/HCP talk about radiation/drug/surgery as an option 0.008 1 Missing16727 A lot214 (40.4%)31 (59.6%) Some131 (24.7%)5 (9.6%) A little64 (12.1%)2 (3.8%) Not at all121 (22.8%)14 (26.9%) Dr/HCP clearly explain disadvantages of each choice for cancer tx 0.004 1 Missing17127 Yes, definitely353 (67.1%)30 (57.7%) Yes, somewhat134 (25.5%)11 (21.2%) No39 (7.4%)11 (21.2%) Dr/HCP involve you in decisions about cancer tx as much as wanted 0.047 1 Missing171 Yes, definitely572 (84.1%)73 (93.6%) Yes, somewhat93 (13.7%)3 (3.8%) No15 (2.2%)2 (2.6%) Tx team encouraged you to contact them with questions between visits 0.036 1 Missing71 Yes, definitely585 (84.8%)63 (80.8%) Yes, somewhat78 (11.3%)7 (9.0%) No27 (3.9%)8 (10.3%)

31. Mode of survey administration (cont.) Mail (N=697) Phone (N=79)p value How many times visit the cancer center appt with tx team 0.014 1 Missing140 0 times53 (7.8%)3 (3.8%) 1 to 5 times292 (42.8%)26 (32.9%) 6 to 10 times138 (20.2%)13 (16.5%) 11 or more times200 (29.3%)37 (46.8%) How often cancer center visits scheduled at times convenient 0.048 1 Missing130 Never19 (2.8%)3 (3.8%) Sometimes42 (6.1%)4 (5.1%) Usually241 (35.2%)16 (20.3%) Always382 (55.8%)56 (70.9%) Did your tx team tell you what the next steps in your tx would be 0.032 1 Missing131 Yes, definitely509 (74.4%)62 (79.5%) Yes, somewhat131 (19.2%)7 (9.0%) No44 (6.4%)9 (11.5%) Did you and your tx team talk about pain related to your cancer or tx 0.014 1 Missing141 Yes578 (84.6%)74 (94.9%) No105 (15.4%)4 (5.1%) Were you bothered by pain from your cancer or tx 0.022 1 Missing40 Yes349 (50.4%)29 (36.7%) No344 (49.6%)50 (63.3%) Did you need help arranging for these additional services 0.033 1 Missing59462 Yes59 (57.3%)5 (29.4%) No44 (42.7%)12 (70.6%) 1 Chi-Square

32. ®® Embedded Study of Comorbidity Questions 2 versions of survey Version 1: “Are you currently being treated by a health professional for…” Diabetes/high blood sugar, high blood pressure/hypertension, heart condition, chronic lung disease/COPD/emphysema/chronic bronchitis, arthritis/joint problems, depression/anxiety Number of conditions reported –0 = 43.4% –1 = 30.2% –2 = 18.5% –3+ = 7.9%

33. ®® Embedded Study Comorbidity index 2 Duration of medicationComorbidity index 3 0 (no repeated care) 0 (no medication)0 (no care & no meds) 1 (acute medication)1 (acute care or acute meds) 2 (chronic medication)2 (chronic care or chronic meds) 1 (acute care) 0 (no medication)1 (acute care or acute meds) 1 (acute medication)1 (acute care or acute meds) 2 (chronic medication)2 (chronic care or chronic meds) 2 (chronic care) 0 (no medication)2 (chronic care or chronic meds) 1 (acute medication)2 (chronic care or chronic meds) 2 (chronic medication) 3 (chronic care and chronic meds) Version 2 Duration of care “In the last 12 months, did you get health care 3 or more times for a condition or problem? Is this a condition or problem that has lasted for at least 3 months?” Duration of prescription medication “Do you now need or take medicine prescribed by a doctor?” “Is this medicine to treat a condition that has lasted for at least 3 months?” Indices were defined as shown below:

34. ®® Embedded Study N (%) Index 2 – Duration of care 0 (no repeated care)238 (68.6) 1 (acute care)30 (8.6) 2 (chronic care)79 (22.8) Index 3 – Duration of care + meds 0 (no repeated care & no meds)94 (28.2) 1 (acute care or acute meds)14 (4.2) 2 (chronic care or chronic meds)160 (48.1) 3 (chronic care & chronic meds)65 (19.5) Version 2 indices results:

35. ®® Composite Measure Analysis 1. Create analytic files 2. Confirmatory Test of Original Composite Structure 3. Identification and Evaluation of Alternative Composite Structures 4. Evaluation of Validity of Best Alternative

36. ®® Create Analytic Files Recode to facilitate interpretation  For some questions higher numbered response indicated better care; for others, a higher numbered response indicated worse care  Reverse-scored questions where higher indicated worse care so that higher always indicates better care Impute missing values  Multivariate analysis requires complete data for all variables in the analysis  Used CAHPS-adopted method of multiple- imputation

37. ®® Original Composite Structure DOMAIN # Items Question Numbers 1 Affective Communication 5 23-27 2 Shared Decision Making 12 4-15 3 Exchanging Information 7 16, 29, 32, 33, 36-38 4 Access to Care & Info. 5 18, 22, 28, 34, 35 5 Coordination of Care 2 30, 31 6 Enable Patient Self-Mgt. 16 17, 19, 39-52 7 Patient safety & AEs* 12 16, 17, 19, 37, 38, 48-52, 55, 56 8 Caregivers & Family 4 53-56 *All questions in this domain are also listed under other domains.

38. ®® Revised Composite Structure DOMAIN # Item s Question Numbers 1 Affective Communication 6 23-28 2 Shared Decision Making 4 9-12 3 Communication about Therapy 8 16-19, 29, 32, 36, 38 4 Enabling Patient Self-Mgt. 5 44, 45, 47, 48, 52

39. ®® Preliminary Composite Results Original Domain# Items Resulting Domains 1Affective Communication5 / 6Included 2Shared Decision Making12 / 4Included 3Exchanging Information7 / 8Included, but re-named “Communication about Treatment” 4Access to Care & Info.5 / 0The question about “spending enough time” loaded on Affective Communication, the rest either did not load, had small loadings or loaded on more than one factor 5Coordination of Care2 / 0Too few questions to form a reliable composite 6Enable Patient Self-Mtg.16 / 5Included 7Patient Safety & AEs12 / 06 items loaded better on other domains. We retained 1 in “Enabling Self-Management” and 5 in “Communication about Treatment” composites 8Caregivers & Family4 / 0Too few questions to form a reliable composite

40. ®® Affective Communication In the last 3 months,  how often did your radiation therapy team treat you with courtesy and respect?  how often did your radiation therapy team show respect for what you had to say?  how often did you feel your radiation therapy team really cared about you as a person?  how often did your radiation therapy team listen carefully to you?  how often was your radiation therapy team direct and straightforward when talking with you about your cancer and radiation therapy?  how often did your radiation therapy team spend enough time with you?

41. ®® Shared Decision Making Since your cancer was diagnosed, did a doctor or other health care professional at [NAME OF CANCER CENTER]  clearly explain the advantages of each choice for cancer treatment, including the treatments you did not get?  clearly explain the disadvantages of each choice for cancer treatment, including the treatments you did not get?  ask for your opinion about each choice of cancer treatment, including the treatments you did not get?  involve you in decisions about your cancer treatment as much as you wanted?

42. ®® Communication About Therapy Since it was decided that you would have radiation therapy,  did your radiation therapy team encourage you to contact them with questions between visits?  tell you to call them immediately if you have certain symptoms or side effects?  give you clear instructions about how to contact them outside of regular office hours?  clearly explain how your cancer and radiation therapy could affect your normal daily activities? In the last 3 months, did your radiation therapy team  seem up-to-date about how to treat your type of cancer?  tell you what the next steps in your radiation therapy would be?  explain test results in a way that was easy to understand?  explain what that medicine was for in a way that was easy to understand?

43. ®® Enabling Patient Self-Management In the last 3 months, did your radiation therapy team  advise you about or help you deal with these changes in your energy levels?  talk about any emotional problems, such as anxiety or depression, related to your cancer or radiation therapy?  advise you about or help you deal with these emotional problems? In the last 3 months, did you and your radiation therapy team  talk about additional services to manage your cancer care at home, such as home health care, special medical equipment, or special supplies?  talk about things you can do to maintain your health during cancer treatment such as what to eat and what exercises to do?

44. ®® Evaluation of Composites Composite α % SS % floor % ceilin g Affective Communication0.87100067 Shared Decision Making0.74100020 Communication about Therapy 0.76100042 Enabling Patient Self-Mgt.0.8010013 RMSEACFINNFI Model Fit Indices 0.060.920.91

45. ®® Questions Significantly, Uniquely Related to Overall Care Net of 4 Composites Q16. …did your therapy team encourage you to contact them between visits? Q18 ….did your therapy team give you clear instruction about how to contact them outside of regular office hours? Q19. …did your radiation therapy team clearly explain how your cancer and therapy could affect your normal daily activities? Q34. How often were the blood tests,… scans, or other procedures scheduled to be done as soon as you thought you needed? Q36. …how often did your team explain test results in a way that was easy to understand?

46. ®® Next Steps Continue data collection at 4 sites and analysis with additional support from the California HealthCare Foundation Revise analysis Final recommended composite structure Final recommended questionnaire.

47. ®® Challenges Extremely challenging project  Focuses on a single clinical condition across multiple care sites Choosing unit to be assessed  Maximizing potential for adoption and sustainability  Defining the care team—all or modality-specific Sampling feasibility  Determining patient eligibility  Identifying sampling frames

48. ®® Acknowledgements AHRQ: Judy Sangl, Chris Crofton, Elma Chowdry NCI: Neeraj Arora, Steve Clauser CoC: Steve Edge, David Winchester, Andrew Stewart Donna O’Brien and Arnold Kaluzny

49. ®® Preliminary Case Mix Adjusters Age Education Self-rating of Mental Health Race: Other‡ Treatment/Services Received: Surgery (q58_3) Treatment/Services Received: none/ w&w (q58_6) Number Visits to Cancer Center for Appts (q21) How long have you been treated for cancer (q2)