1. 1 EPI235: Epi Methods in HSR May 3, 2007 L10 Outcomes and Effectiveness Research 4: HMO/Network (Dr. Schneeweiss) Methodologic issues in benchmarking physician and hospital performance. Analysis of patient data clustered in physicians and in clinics using hierarchical (multi-level) models. Reporting benchmarking results. Background reading: Austin PC, Goel V, van Walraven C. An introduction to multilevel regression models. Can J Public Health 2001;92:150-154. Austin PC, Tu JV, Alter DA: Comparing hierarchical modeling with traditional logistic regression analysis among patients hospitalized with acute myocardial infarction: Should we be analyzing cardiovascular outcomes data differently? Am Heart J 2003;145:27-35. Carey K: A multilevel modeling approach to analysis of patient costs under managed care. Health Econ 2000;9:435-446.

2. 2

3. 3

4. 4 Pennsylvania Consumer Report www.phc4.org

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11 Quality improvement by quality measurement in NY Hannan ED et al. JAMA 1995

12. 12 Risk factor reporting in the NY Bypass Reporting System (%)

13. 13

14. 14

15. 15 No Clusters Clusters but No Clustering Clusters with much Clustering Extreme Clustering: Pure Lumps

16. 16

17. 17 CABG Mortality among 55 hospitals MLwiN®

18. 18

19. 19

20. 20

21. 21

22. 22

23. 23

24. 24

25. 25

26. 26 What is the best comparator for benchmarking? 1) Average of all institutions 2) Stratified by institution characteristics Austin et al. Am Heart J 2004

27. 27

28. 28 Dealing with clustering  Generalized estimating equations (GEE)  SAS: proc genmod  Random effects modeling (multi-level modeling)

29. 29 Effect of physician specialty on patient outcomes Attending cardiology specialty Austin et al. Am Heart J 2003

30. 30 Teaching hospital

31. 31 GEE vs. random effects model Twisk, Eur J Epi, 2004 Expl.: Risk factors of increased cholesterol levels

32. 32 GEE Rand. intercept

33. 33 GEE Rand. intercept

34. 34 Final Note: Parameter Interpretation  Marginal parameters are often thought to be more appropriate for policy questions, “What is the difference in the rate of illness in the treated population versus the untreated?”  Conditional parameter estimates closer to individual effects, “What is the effect of treatment in an individual person?”  Conditional parameter are more clinically relevant?