1. Quality Improvement Research Methods: Issues in Detecting Changes in Clinician Performance Haya R. Rubin, M.D., Ph.D. Lynne Nemeth, R.N., Ph.D. Hoangmai Pham, M.D., M.P.H.

2. Background Given multi-year projects to measure physician performance, it is now possible to examine and compare provider changes in performance on quality indicators, providers want to track if change is occurring Interventions are being applied that require comparing changes in different groups, i.e., physician performance incentives This has made evident methodological issues in determining whether, which, and how big a change has occurred, and whether one change exceeds another.

3. Overview of Issues Risk Adjustment –Patient Characteristics* –Clinician/Practice Characteristics* –Insurance Type Indicator Specifications –Denominators: relationship of indicator lag time to frequency of indicator measurement –Specific indicators: choice and volume –Composites: Scope, weighting –Provider exclusion from denominators

4. Risk Adjustment: What needs to be adjusted for in design or analysis? –Patient characteristics: practices differ from each other, and over time, and interventions groups may differ from each other in the patient populations they serve. Should patient characteristics (i.e., age, gender, race, ethnicity, diagnoses, chronic diseases, comorbidity, health status, attitudes, behavior) be adjusted for in comparing changes over time? –Practice/provider characteristics: Practices differ in size, scope and specialty, state of facilitators/barriers to achieving quality care, and how well they are doing on indicators at baseline. Do we need to adjust for any of these practice characteristics when determining if changes in quality have occurred and in comparing changes to each other?

5. Risk Adjustment: Insurance/Payment Type Payment incentives may be limited to specific payers; can measures for specific payers be generalized to others? Does it matter if measures are applied in capitated payment systems vs. fee-for- service systems? What is the state of other payer incentives/structures to facilitate quality besides the intervention being evaluated?

6. Indicator Specification: Does the nature of the measure affect the determination of change? Indicator specification issues –Specific indicators: Choice, volume issues –Composite indicators: Scope and weighting –Provider exclusion from denominators –Lag time for indicator evaluation vs. frequency of indicator measurement

7. Specific Indicators Choice: Does a specific indicator apply to a practice’s work/role in improving patients’ health? e.g., Should a BP control indicator be used to compare urologists’ care for hypertensive patients? Volume: Does the provider see enough patients with the condition to which the indicator applies to allow an evaluation of change in performance?

8. Composite Indicators Which indicators are included in a composite indicator? –Choice, scope: How well do they apply to each of the practices being compared? How much of the valuable work of each practice in improving health and survival is included in the composite indicator? –Weighting of individual component indicators: Does it approximate that aspect’s impact on patient quality- adjusted survival? How do we determine how to weight each one?

9. Provider Exclusion In some systems, providers can exclude patients from denominators where there are clinical exclusions specified Where possible, do some fail to use this capability and look like they perform worse than they do relative to other practices? Do some use this inappropriately and look like they perform better than they do relative to colleagues?

10. Lag Time vs. Frequency If indicator specifies a period for measurement such as “last 6 months” and is measured quarterly, how do we deal with the overlap? What about replication of patients in each period?

11. Panelists Lynne Nemeth, Ph.D. –Medical University of South Carolina –Will present on impact of baseline performance on changes observed in A-TRIP study on 99 practices Hoangmai Pham, M.D., M.P.H. –Center for Studying Health System Change –Will present on impact of patient demographic characteristics on changes in clinician preventive care performance observed in Medicare