1. Measuring Patients’ Experiences With Individual Physicians: Are We Ready for Primetime? Presented at: Academy Health Annual Research Meeting San Diego, CA 7 June 2004 Commonwealth Fund and Robert Wood Johnson Foundation Presented at: Academy Health Annual Research Meeting San Diego, CA 7 June 2004 Commonwealth Fund and Robert Wood Johnson Foundation Dana Gelb Safran, ScD The Health Institute Institute for Clinical Research and Health Policy Studies Tufts-New England Medical Center ___________________________________________________________________________

2. Focusing on Physicians u Survey-based measurement of patients’ experiences with individual physicians is not new. u What’s new: Efforts to standardize and potential for public reporting. u IOM report Crossing the Quality Chasm gave “patient-centered care” a front row seat. u Methods and metrics have been honed through 15 years of research and through several recent large-scale demonstration projects u But putting these measures to use raises many questions about feasibility and value. u Survey-based measurement of patients’ experiences with individual physicians is not new. u What’s new: Efforts to standardize and potential for public reporting. u IOM report Crossing the Quality Chasm gave “patient-centered care” a front row seat. u Methods and metrics have been honed through 15 years of research and through several recent large-scale demonstration projects u But putting these measures to use raises many questions about feasibility and value.

3. Ambulatory Care Experiences Survey Project u Statewide demonstration project in Massachusetts u Collaboration: v 6 Payers v 6 Physician Network Organizations v Massachusetts Medical Society v Massachusetts Health Quality Partners u Testing the feasibility and value of measuring patients’ experiences with individual primary care physicians and practices u Primary impetus: plans seeking to standardize surveys u IOM “Chasm” report further propelled the work u Statewide demonstration project in Massachusetts u Collaboration: v 6 Payers v 6 Physician Network Organizations v Massachusetts Medical Society v Massachusetts Health Quality Partners u Testing the feasibility and value of measuring patients’ experiences with individual primary care physicians and practices u Primary impetus: plans seeking to standardize surveys u IOM “Chasm” report further propelled the work

4. Principal Questions of the Statewide Pilot u What sample size is needed for highly reliable estimate of patients’ experiences with a physician? u What is the risk of misclassification under varying reporting frameworks? u Is there enough performance variability to justify measurement? u How much of the measurement variance is accounted for by physicians as opposed to other elements of the system (practice site, network organization, plan)? u What sample size is needed for highly reliable estimate of patients’ experiences with a physician? u What is the risk of misclassification under varying reporting frameworks? u Is there enough performance variability to justify measurement? u How much of the measurement variance is accounted for by physicians as opposed to other elements of the system (practice site, network organization, plan)?

5. Sampling Framework Eastern, MACentral, MAWestern, MA Tufts, BCBSMA, HPHC, Medicaid BCBSMA, Fallon, Medicaid BCBSMA, HNE, Medicaid PNO1PNO2PNO3 34 Sites 143 Physicians 23 Sites 35 Physicians PNO6 10 Sites 37 Physicians PNO4 PNO5 Both commercially insured & Medicaid patients sampled Only commercially insured patients sampled

6. Measures from the Ambulatory Care Experiences Survey (ACES) Communication Comprehensiveness · whole-person orientation ·health promotion/ patient empowerment Integration team specialists lab Continuity · longitudinal · visit-based Organizational Access Interpersonal Treatment Trust Primary Care

7. Sample Size Requirements for Varying Physician-Level Reliability Thresholds

8. What is the Risk of Misclassification?  Not simply 1-  MD u Depends on:  Measurement reliability (  MD ) u Proximity of score to the cutpoint u Number of cutpoints in the reporting framework  Not simply 1-  MD u Depends on:  Measurement reliability (  MD ) u Proximity of score to the cutpoint u Number of cutpoints in the reporting framework

9. Risk of Misclassification at Varying Distances from the Benchmark and Varying in Measurement Reliability (  MD )

10. 50 th p’tile 3.26 α MD=0.7 α MD=0.8 α MD=0.9 4.9 6.3 0 65 100 Certainty and Uncertainty in Classification Comparison with a Single Benchmark = area of uncertainty Significantly below Significantly above

11. α MD=0.7 α MD=0.8 α MD=0.9 10 th p’tile 90 th p’tile 0 53 76 100 6.3 4.9 3.26 3.26 Certainty and Uncertainty in Classification Cutpoints at 10 th & 90 th Percentile = area of uncertainty Bottom Tier Middle TierTop Tier

14. Number of Doctors MD Mean Score, % Variability Among Physicians (Communication) ___________________________________________________________________________

15. 25 th -75 th percentile range of group scores Group Mean score Eastern Region Central RegionWestern Region Group Mean Score, % Variability Across Practice Sites (Communication)

16. 25 th -75 th percentile range of site scores Site Mean score Site and MD Mean Score Variability Among Physicians within Sites (Communication) 25 th -75 th percentile range of MD scores MD Mean score Site A-1 Site A-2Site A-3Site A-4

17. Allocation of Explainable Variance: Doctor-Patient Interactions Communication Whole-person orientation Health promotion Interpersonal treatment Patient trust

18. Allocation of Explainable Variance: Organizational/Structural Features of Care

19. Summary and Implications u With sample sizes of 45 patients per physician, most survey- based measures achieved physician-level reliability of.7-.85. u With a 3-level reporting framework, risk of misclassification is low – except at the boundaries, where risk is high irrespective of measurement reliability. u Individual physicians and practice sites accounted for the majority of system-related variance on all measures. u Within sites, variability among physicians was substantial. u With sample sizes of 45 patients per physician, most survey- based measures achieved physician-level reliability of.7-.85. u With a 3-level reporting framework, risk of misclassification is low – except at the boundaries, where risk is high irrespective of measurement reliability. u Individual physicians and practice sites accounted for the majority of system-related variance on all measures. u Within sites, variability among physicians was substantial.

20. Summary and Implications (cont’d) u Feasibility of obtaining highly reliable measures of patients’ experiences with individual physicians and practices has been demonstrated. u The merits and value of moving quality measurement beyond health plans and network organizations is clear. u By adding these aspects of care to our nation’s portfolio of quality measures, we may reverse declines in interpersonal quality of care. u Feasibility of obtaining highly reliable measures of patients’ experiences with individual physicians and practices has been demonstrated. u The merits and value of moving quality measurement beyond health plans and network organizations is clear. u By adding these aspects of care to our nation’s portfolio of quality measures, we may reverse declines in interpersonal quality of care.

22. 10 th p’tile 50 th p’tile 90 th p’tile α MD=0.7 α MD=0.8 α MD=0.9 0 53 65 76 100 6.3 4.9 4.9 4.9 3.26 3.26 3.26 Certainty and Uncertainty in Classification Multiple Cutpoints = area of uncertainty Bottom Tier Top Tier 3rd Tier 2nd Tier