1. Term 4, 2006BIO656--Multilevel Models 1 Midterm Open “book” and notes; closed mouth 20-25 minutes to read carefully and answer completely  60 minutes to think 4 problems, with possible topics: –Some debriefing on the role of MLMs –Something on linear MLMs variances, etc. – MLMs and shrinkage –Some basic computations and interpretations of logit-linear models

2. Term 4, 2006BIO656--Multilevel Models 2 PART 6 PROFILING, RANKING “League Tables”

3. Term 4, 2006BIO656--Multilevel Models 3

4. Term 4, 2006BIO656--Multilevel Models 4 RANKING IN THE NEWS

5. Term 4, 2006BIO656--Multilevel Models 5 LETTERMAN’S TOP 10 LIST

6. Term 4, 2006BIO656--Multilevel Models 6 NEW YORK’S MOST DEADLY CARDIAC SURGEONS!!!!

7. Term 4, 2006BIO656--Multilevel Models 7

8. Term 4, 2006BIO656--Multilevel Models 8 THE LEADING SPH IS HARVARD

9. Term 4, 2006BIO656--Multilevel Models 9 HOPKINS IS THE LEADING SPH!!!

10. Term 4, 2006BIO656--Multilevel Models 10 PROFILING (League Tables) The process of comparing “units” on an outcome measure with relative or normative standards –Quality of care, use of services, cost –Educational quality –Disease rates in small areas –Gene expression Developing and implementing performance indices to compare physicians, hospitals, schools, teachers, genes,........

11. Term 4, 2006BIO656--Multilevel Models 11 PROFILING OBJECTIVES PROFILING OBJECTIVES (in health services) Estimate and compare provider-specific performance measures: –Utilization/cost –Process measures –Clinical outcomes –Patient satisfaction/QoL Compare using a normative (external) or a relative (internal) standard

12. Term 4, 2006BIO656--Multilevel Models 12

13. Term 4, 2006BIO656--Multilevel Models 13 RANKING IS EASY Just compute estimates & order them

14. Term 4, 2006BIO656--Multilevel Models 14 MLE ESTIMATED SMRs

15. Term 4, 2006BIO656--Multilevel Models 15 RANKING IS DIFFICULT Need to trade-off the estimates and uncertainties

16. Term 4, 2006BIO656--Multilevel Models 16 MLE ESTIMATED SMRs & 95% CIs

17. Term 4, 2006BIO656--Multilevel Models 17 Sampling variability & Systematic variability Systematic variability Variability among physicians/hospitals that might be explained by hospital-specific characteristics Sampling variability Statistical uncertainty of physician/hospital-specific performance measures Use MLMs that Incorporate patient, physician and hospital-level characteristics Capture all important uncertainties Produce appropriate statistical summaries

18. Term 4, 2006BIO656--Multilevel Models 18 Statistical Challenges Need a valid method of adjusting for case mix and other features Patient, physician and hospital characteristics –But, beware of over adjustment Need a valid model for stochastic properties Account for variation at all levels Account for within-hospital, within-patient correlations Need to Adjust for systematic variation Estimate and account for statistical variation

19. Term 4, 2006BIO656--Multilevel Models 19 PROPER USE OF STATISTICAL SUMMARIES The challenge Differences in standard errors of hospital-specific estimates invalidate direct comparisons In any case, large SEs make comparisons imprecise Consequence Even after valid case mix adjustment, differences in directly estimated performance are due, in part, to sampling variability (Partial) Solution, use: Shrinkage estimates to balance and reduce variability Goal-specific estimates to hit the right target

20. Term 4, 2006BIO656--Multilevel Models 20 Comparing performance measures Ranks/percentiles, of: Direct estimates (MLEs) Shrunken estimates (BLUPs, Posterior Means) Z-scores testing H 0 that a unit is just like others Optimal (best) ranks or percentiles Other measures Probability of a large difference between unit-specific “true” and H 0 -generated event rates Probability of “excess mortality” –For the “typical patient, on average or for a specific patient type Z-score/P-value declarations....

21. Term 4, 2006BIO656--Multilevel Models 21

22. Term 4, 2006BIO656--Multilevel Models 22 USRDS

23. Term 4, 2006BIO656--Multilevel Models 23 USRDS

24. Term 4, 2006BIO656--Multilevel Models 24

25. Term 4, 2006BIO656--Multilevel Models 25

26. Term 4, 2006BIO656--Multilevel Models 26

27. Term 4, 2006BIO656--Multilevel Models 27

28. Term 4, 2006BIO656--Multilevel Models 28 MLE ESTIMATED SMRs & CIs

29. Term 4, 2006BIO656--Multilevel Models 29 Poisson-Normal Model Poisson-Normal Model (N, Y[k], emort[k]) are inputs model { {prec~dgamma(0.00001,0.00001) for (k in 1:N) { 0 logsmr[k]~dnorm(0,prec) smr[k]<-exp(logsmr[k]) rate[k]<-emort[k]*smr[k] Y[k] ~ dpois(rate[k]) } Monitor the SMR[k]

30. Term 4, 2006BIO656--Multilevel Models 30 SE MLE PM MLE, SE & POSTERIOR MEAN SMRs (using a log-normal/Poisson model)

31. Term 4, 2006BIO656--Multilevel Models 31 Posterior Mean: estimated SMRs & CIs using a log-normal/Poisson model (original scale)

32. Term 4, 2006BIO656--Multilevel Models 32 Posterior Mean: estimated SMRs & CIs using a Gamma/Poisson model (expanded scale)

33. Term 4, 2006BIO656--Multilevel Models 33 Estimated relative, physician-specific visit rate and 95% CI Adjusted for patient demographic and case-mix (1.0 is the “typical” rate) Caterpillar Plot (Hofer et al. JAMA 1999)

34. Term 4, 2006BIO656--Multilevel Models 34 Amount that physician-specific, laboratory costs for diabetic patients deviates from the mean for all physicians [$/(pt. yr.)] Lines show the path from the direct estimate (the MLE) to the shrunken estimate (Hofer et al JAMA 1999) DIRECTADJUSTED

35. Term 4, 2006BIO656--Multilevel Models 35 Example using BUGS for hospital performance ranking

36. Term 4, 2006BIO656--Multilevel Models 36

37. Term 4, 2006BIO656--Multilevel Models 37 BUGS Model specification model { for k in 1:K { b[k]~dnorm(0, prec) r[k]~dbin(p[k], n[k]) logit(p[k]) <- mu + b[k] } pop.mean<-exp(mu + bb)/(1+exp(mu + bb)) mu~dnorm(0, 1E-6) prec~dgamma(.0001,.0001) tausq<-1/prec add~dnorm(0, prec) bb<- mu + add } Monitor the p[k] and ask for ranks

38. Term 4, 2006BIO656--Multilevel Models 38 Summary Statistics

39. Term 4, 2006BIO656--Multilevel Models 39 Posterior distributions of the ranks

40. Term 4, 2006BIO656--Multilevel Models 40

41. Term 4, 2006BIO656--Multilevel Models 41

42. Term 4, 2006BIO656--Multilevel Models 42

43. Term 4, 2006BIO656--Multilevel Models 43 X = (Posterior Mean-Based Ranks) – (Optimal Ranks) LOS  = = = =

44. Term 4, 2006BIO656--Multilevel Models 44 LOS

45. Term 4, 2006BIO656--Multilevel Models 45 Relations among percentiling methods 1998 USRDS Percentiles

46. Term 4, 2006BIO656--Multilevel Models 46 BACK TO THE USRDS, SMR EXAMPLE

47. Term 4, 2006BIO656--Multilevel Models 47

48. Term 4, 2006BIO656--Multilevel Models 48

49. Term 4, 2006BIO656--Multilevel Models 49 False detection and non-detection

50. Term 4, 2006BIO656--Multilevel Models 50

51. Term 4, 2006BIO656--Multilevel Models 51 K is large and we can use a completely non-parametric prior

52. Term 4, 2006BIO656--Multilevel Models 52  = (1-B) =  2 /(  2 +  2 ) = ICC

53. Term 4, 2006BIO656--Multilevel Models 53 Probability of being in the upper 10% as a function of true percentile  = intra-class correlation

54. Term 4, 2006BIO656--Multilevel Models 54