1. Term 4, 2006 BIO656--Multilevel Models 1 PART 3

2. Term 4, 2006 BIO656--Multilevel Models 2 NEED TO INCORPORATE ALL UNCERTAINTIES The Z versus t distribution is the basic example Want to produce a CI for a population mean Assume a Gaussian sampling distribution,: Z is t with a large df t 3 is the most different from Z for t-distributions with a finite variance

3. Term 4, 2006 BIO656--Multilevel Models 3

4. Term 4, 2006 BIO656--Multilevel Models 4 ACCOUNTING FOR (explaining) UNEXPLAINED VARIABILITY Including regressors can explain (account for) some of unexplained variability Doing so is always a trade-off in that you need to use degrees of freedom to do the explaining Going too far--adding too many regressors-- inflates residual variability In MLMs there is variance at various levels that can potentially be taken into account

5. Term 4, 2006 BIO656--Multilevel Models 5 TEACHER EXPECTANCY TEACHER EXPECTANCY (data are in “Datasets” ) Data are from a Raudenbush & Bryk meta-analysis of 19 studies (see Cooper and Hedges,1994) Effect size k = distance between treatment and control group means measured in population standard deviation units SE k = the standard error of the effect size Weeks k = estimated weeks of teacher-student contact prior to expectancy induction

6. Term 4, 2006 BIO656--Multilevel Models 6 TEACHER EXPECTANCY TEACHER EXPECTANCY (continued) Each study consisted of either telling teachers that a student had great potential or not All students received a pre-test and a post-test Teachers evaluated progress A positive effect size indicates that the teachers rated students who were “likely to improve” as having improved more than the control group A negative slope on “Weeks” indicates that the more a teacher got to know a student before the experiment,the less the influence of the expectancy intervention

7. Term 4, 2006 BIO656--Multilevel Models 7

8. Term 4, 2006 BIO656--Multilevel Models 8

9. Term 4, 2006 BIO656--Multilevel Models 9 ESTIMATING  2 ESTIMATING  2 (when all  k 2 =  2 ) Compute residuals, r k = (Y k – model k ) Compute the Mean Squared Error: (n – df)MSE =  r 2 k Then, compute:  2 = (MSE -  2 ) model k is either just the intercept or intercept +  weeks k  2 decreases if MSE decreases

10. Term 4, 2006 BIO656--Multilevel Models 10 A CASE STUDY ON VARIANCE ACCOUNTING

11. Term 4, 2006 BIO656--Multilevel Models 11 DIABETES CONTROL STUDY Percentage Variance at the Patient, Physician, and Clinic Levels Patrick J. O’Connor MD MPH, Gestur Davidson PhD A. Lauren Crain PhD Leif I. Solberg MD Robin R. Whitebird PhD Thomas A. Louis PhD HealthPartners Research Foundation University of Minnesota Johns Hopkins Bloomberg SPH

12. Term 4, 2006 BIO656--Multilevel Models 12 Conceptual Model: What Affects Diabetes Care? A Nested Hierarchy Health Plan Medical Group Clinic Physicians Patients Interactions across all levels

13. Term 4, 2006 BIO656--Multilevel Models 13 OUTCOME MEASURE: A1c (HbA1c) Glycohemoglobin, Glycated hemoglobin Known as: Hemoglobin A1c Used to monitor diabetes and to aid in treatment decisions Should be assayed at first diagnosis of diabetes and then 2 to 4 times per year Requires a blood sample Normal values between 4% and 6% High is bad

14. Term 4, 2006 BIO656--Multilevel Models 14 STUDY SITE Health Partners Medical Group, MN 175,000 adults receiving care at 19 clinics in 1995 Medical group centrally administered and clinics have common guidelines, formulary, and culture So, there may be less variance at clinic or physician level than in other contexts

15. Term 4, 2006 BIO656--Multilevel Models 15 STUDY DESIGN Analysis of 2,463 adults with diabetes mellitus in 1994 Follow-up A1c data in 1995, 1996, 997 Patients nested within providers and clinics

16. Term 4, 2006 BIO656--Multilevel Models 16 STUDY PARTICIPANTS 2,463 adults with DM in same clinic and with same primary care physician each year from 1995-1997 DM identification: Sensitivity = 0.91 Pred. Pos. Val. = 0.94 To be included in analysis must have had at least one A1c test each year A1c test rates ranged 80-87% per year 1995-1997

17. Term 4, 2006 BIO656--Multilevel Models 17 Number of Eligible Adults with diabetes in 1995 cohort YearEnrolledWith A1c Test 19955,4324,339 19964,8353,941 19974,4513,767

18. Term 4, 2006 BIO656--Multilevel Models 18 Analytic Sample 19 Clinics 41 Physicians 2,463 Patients 3 years of time

19. Term 4, 2006 BIO656--Multilevel Models 19 MODEL & ANALYSIS Multilevel multivariate hierarchical linear models (using MLwiN) to estimate variance components at each level (time, patient, physician, clinic) Analyzed for A1c in each year, and change in A1c across years

20. Term 4, 2006 BIO656--Multilevel Models 20 STUDY POTENTIAL There is substantial variance in A1c and in change in A1c across all levels of the hierarchy Some of the A1c variance is a “roll-up” from lower levels To develop rational improvement strategies, one must understand where the variance resides and if some can be explained

21. Term 4, 2006 BIO656--Multilevel Models 21 HYPOTHESES After control for patient and physician variance, there will be no clinically significant variance in A1c change at the clinic level At each level that has significant variance in A1c change, we may be able to identify key variables that are related to the variance

22. Term 4, 2006 BIO656--Multilevel Models 22 Characteristics of study participants who had/(did not have)  1 A1c tests during the 36-month study period A1c Measurement Status Comparison Variables MeasuredNot Meas.P-val % Female53% 0.73 Average Age59.560.40.10 Average Charlson1.732.070.01 Female Physician28%25%0.23 Average Physician Age 42.243.00.01 % Fam. Med.33%35%0.39

23. Term 4, 2006 BIO656--Multilevel Models 23 Characteristics of participants who were/(were not) assigned to a primary care physician Physician Assignment Status Comparison Variables AssignedNot Assigned P-val % Female47% 0.60 Average Age60.455.30.01 Average Charlson 1.751.020.01 A1c Value8.298.060.01

24. Term 4, 2006 BIO656--Multilevel Models 24

25. Term 4, 2006 BIO656--Multilevel Models 25

26. Term 4, 2006 BIO656--Multilevel Models 26

27. Term 4, 2006 BIO656--Multilevel Models 27 PERCENT OF VARIANCE ACCOUNTING 1995 A1c Level PERCENT OF VARIANCE ACCOUNTING 1995 A1c Level “Vanilla” model & with Covariates Vanilla Model Full Model Clinic 1.9% 2.7% Physician 2.8% 1.4% Patient95.4%96.0%

28. Term 4, 2006 BIO656--Multilevel Models 28 Covariates for 1995 A1c Level ( R 2 = 0.14 ) VariableCoefficientSE Pt Age < 65 0.0250.010 Insulin Use 0.1590.010 Sulfonyl Use 0.1060.010 Phy. Specialty-0.0150.014 Phy. Gender-0.0160.057 Pt Comorbid=2 0.0250.012

29. Term 4, 2006 BIO656--Multilevel Models 29 The slope on insulin is 0.159 which is > 0 Does this mean that insulin is bad for diabetics? Or, does it represent an association between A1c level and the decision to treat? Hint: It’s the association/selection BEWARE OF SELECTION EFFECTS

30. Term 4, 2006 BIO656--Multilevel Models 30 PERCENT OF VARIANCE ACCOUNTING (1997-1995) A1c Change PERCENT OF VARIANCE ACCOUNTING (1997-1995) A1c Change “Vanilla” model & with Covariates Vanilla Model Full Model Clinic< 0.1% Physician 0.7% 0.8% Patient99.3%*99.2%*

31. Term 4, 2006 BIO656--Multilevel Models 31 Covariates for 1997-1995 A1c change (R 2 = 0.11) VariableCoefficientSE Pt Age < 65 0.0930.044 Drug Intensity-0.4180.056 Doc Specialty 0.0570.104 Pts Per Doc -0.0020.002 Doc Age -0.0010.004 Pt Comorbidity -0.0140.018

32. Term 4, 2006 BIO656--Multilevel Models 32 ANALYZING CHANGE REMOVES THE SELECTION EFFECT The slope on insulin is now –0.418 which is < 0 Warning: such a simple analysis will not always sort things out  “Causal Analysis” is needed

33. Term 4, 2006 BIO656--Multilevel Models 33 RESULTS Models with limited set of covariates explained about 14% of variance in A1c levels in 1995 Models with covariates explained about 35% of variance in change in A1c from 1995-97 Over 90% of variance was at patient level or related to physician-patient interaction Little variance at physician level Little variance at the clinic level

34. Term 4, 2006 BIO656--Multilevel Models 34 Factors associated with change in A1c Time: A1c got better each year Older patients had more improvement Comorbidity was related (complex) Drug Intensification (by drug class) was the variable that was strongest predictor Unidentified Patient Factors are likely

35. Term 4, 2006 BIO656--Multilevel Models 35 Other Clinical Domains A1c Test Rates LDL Test Rates Eye Exam Rates Generally similar results, with the majority of variance at patient level; much less variance at physician and clinic levels

36. Term 4, 2006 BIO656--Multilevel Models 36 STUDY LIMITATIONS Relatively homogeneous medical group, may reduce variance at clinic and doc level Clinic systems already in place Selection effects Paucity of covariates at clinic and provider levels

37. Term 4, 2006 BIO656--Multilevel Models 37 DISCUSSION Interventions may be made at any level, not just levels with significant variance However, a great deal of recent attention is directed to clinic systems Patient behavior and provider behavior and doctor-patient interaction need more attention Factors that impact drug intensification may be key

38. Term 4, 2006 BIO656--Multilevel Models 38 Future Directions Larger set of medical groups and clinics More covariates at each level Model selection effects Estimate power at various levels Strategies to handle missing data Assess other clinical domains