Measurement theory and provider profiling Timothy P. Hofer MD Dept. Medicine, University of Michigan VA Ann Arbor HSR&D Center of Excellence

The measurement problem Quality  i(jk) construct indicator

Site (clinic, hospital) Provider (physician) Patient Levels of care Indicators iii

Implications of the measurement model The indicator is a fallible measure of the construct Some indicators are less precise than others Quality indicators are very imprecise for a variety of reasons You need to account for the measurement error The location of the construct variability can suggest different causes, interventions and measurement procedures

Intra-class correlation(=reliability) Ability to distinguish between physicians (or sites) single observation under a specified set of conditions of measurement.

Vol. 281 No. 22, pp , June 9, 1999

MD laboratory utilization profiles

VA Network 11 Diabetes Care Project Health Services Research Volume 37 Issue 5 Page October 2002 doi: / Whom Should We Profile? Examining Diabetes Care Practice Variation among Primary Care Providers, Provider Groups, and Health Care Facilities Sarah L. Krein, *Timothy P. Hofer, Eve A. Kerr, and Rodney A. Hayward

Resources available VA Diabetes Registry Project ( ) Automated Clinical Databases Data warehouse (VA Healthcare and analysis group) Database Components Encounter records (OPC/PTF ) Outpatient Pharmacy Lab primary care provider database (PCMM () Vitals Cohort identification procedure Data quality and measure validation Kerr EA, et al. Journal on Quality Improvement 2002; 28(10):

Selected Measures: Resource Use Cost of hypoglycemic medications Cost of home glucose monitoring for patients not on insulin Cost of calcium channel blockers Quality Processes OutcomesIntermediate Outcomes

Selected Measures : Intermediate Outcomes Last A1c value A1c  9.5% Last LDL value LDL  3.6 mmol/L (140mg/dl) Quality Processes OutcomesIntermediate Outcomes

Selected Measures: Process Measures Hemoglobin A1c obtained LDL-C obtained Lipid profile obtained Quality Processes OutcomesIntermediate Outcomes

Selected Measures: Mixed or Linked Measure LDL  3.6 mmol/L (140mg/dl) or on a statin

Are there differences between physicians? What are the sources of variation? Noise Unmeasured differences Physician effects Clinic or group effects Health System/payor effects

Outcomes

Intermediate outcomes

Process measures

Physician effect size

Panel size Variance attributable to level of care Median PCP Panel size in study sample Last LDL-C Value (1%) Cost of home glucose monitoring for patients not on Insulin Last LDL-C value <3.6 mmol/L or on a statin (5%) Hemoglobin A1c obtained (8%) PCP Effect Negligible SmallModerate

Implicit chart review – site level Trained physician reviewers 621 records 26 clinical sites

Conclusions Measurement models are fundamentally important to measuring and profiling quality. There is often little reason or capability to profile at the physician level. Profiles that ignore measurement error Misrepresent the variability in quality Are difficult (or impossible) to validate

Example – the imprecise thermometer Budget cuts inspire innovation in the clinic

Observed temperature

Observed vs. true temperature

Strength in numbers Body temperature(F) trueobservedaverage

Scale transformation

Reliability “A person with one watch knows what time it is” “A person with two watches is never quite sure”

Effect of gaming