1. Ensuring Consistency in Assessment of Continuing Care Needs: An Application of Differential Item Functioning Analysis R. Prosser, M. Gelin, D. Papineau, B. Zumbo, Presentation at CES Conference June 3, 2003

2. Outline Background –The MDS-HC Assessment Tool & Its Uses –The Challenge Methodology –The IADL Scale –Differential Item Functioning Analysis Results of Our Look at 5 Raters Implications for Practice

3. What is MDS? Standardized assessment of functional, psychological, social and environmental needs, administered by clinician raters Target groups include older adults and people with disabilities or chronic diseases Minimum Data Set: a tool for collecting the minimum amount of data needed for producing a comprehensive and accurate profile Our focus is on MDS for Home Care (HC) clients Mandated for use in B.C.

4. A Word About InterRAI International group of 40+ researchers and clinicians –Registered as not-for-profit corporation and owns international copyright on RAI instruments –Conducts multinational collaborative research to develop, implement and evaluate the instruments and their related applications –Tools available in 12 languages –Lead Canadian researcher: Dr. John Hirdes (University of Waterloo)

5. Uses of MDS Assessment Care Planning Outcome MeasuresQuality Indicators Resource Allocation

6. Components of the MDS System –Minimum Data Set assessment form –Assessment protocols (trigger issues to review in developing client’s care plan) –Quality indicators (standards and targets) –Health outcome scales (effectiveness) –Resource utilization groups: case mix system of groups with homogeneous resource requirements

7. Key Outcome Scales Cognitive Performance Scale (CPS) Depression Rating Scale (DRS) Activities of Daily Living (ADL) Performance Scales Independent Activities of Daily Living (IADL) Scales

8. Uses of Outcome Scales Provide additional information to determine service needs and to plan care Evaluate the effects of treatments Compare results in different settings for client groups with similar health issues and needs

9. Uses of MDS in Planning & Implementing a New Service Model Establish / refine service inclusion and exclusion criteria using RAI-HC Permits more reliable and equitable service allocation Determine number of clients on current caseloads who could benefit from this service Estimate future demand for this service

10. The Challenge Need to have an objective basis for evaluating and demonstrating assessor comparability and improving the MDS rating process –To ensure that service allocation decisions based on assessments are made fairly—and transparently so (i.e., can be explained and justified) –To provide reliable information for care planning and program evaluation Time pressure (long instrument) Small samples

11. The IADL Scale 7 items –Meal Preparation –Ordinary Housework –Managing Finance –Managing Medications –Phone Use –Shopping –Transportation Four point scale for each item –Independent, Some Help, Full Help, By Others

12. Differential Item Functioning (DIF) Analytic Foundation: Item Response Theory of measurement Answers the question “Do item response curves differ appreciably for subgroups of a population, e.g., males & females or blacks & whites?” Often used to provide evidence in relation to item bias in achievement testing In our case, subgroups are seniors rated by DIFFERENT RATERS Our Question: Do different raters use an item the same way?

13. Item Characteristic Curves & DIF

14. Our Method Began by looking at DIF via logistic regression Decided that graphical approach is much more informative  J. Ramsay’s Testgraf software

15. Our Sample Seniors from one of the four subregions within North & West Vancouver Selected the five raters for that area who had the largest caseloads Included only cases with complete data (i.e., no ratings omitted due to the target activity not being performed within the time frame)  N = 335

16. Raters’ Subgroups

17. The IADL Scale

18. Curve for Meal Preparation Item

20. Curves for Housework Item (5 raters)

21. Housework item (Option curves)

22. Curves for Transportation Item (Item curve)

23. Transportation Item (Option curves)

24. Summary of Key Findings The five raters behaved similarly for most items Rater 5 appears to be using items differently than the other raters, particularly item 7

25. Implications for Practice Consider elaboration / clarification of scoring instructions provided to raters Conduct periodic reviews / monitoring with raters to ensure that consistency is maintained Provide additional training to particular raters on particular items as needed May want to have raters rate random samples of clients or stratify the analyses

26. Benefits of this Approach to DIF Analysis Can be used with relatively small samples of the population being rated Raters can rate different seniors provided the samples are equivalent  random assignment Short scales (small numbers of items) are OK Graphical approach is easy to interpret yet reveals details needed for making changes to instructions / training of raters

27. Areas for Future Work Testing with raters –How well does the feedback work in practice? –Is transparency increased? Determining rules of thumb for use –Minimum numbers of cases to rate –Scale features, e.g., minimum length needed

28. For Further Information InterRAI Web Site: http://nt8380.hrca.harvard.edu/ http://nt8380.hrca.harvard.edu/ Differential Item Functioning: http://www.educ.ubc.ca/faculty/zumbo/DIF/ TestGraf Web Site: ftp://ego.psych.mcgill.ca/pub/ramsay/testgraf/ ftp://ego.psych.mcgill.ca/pub/ramsay/testgraf/ Author e-mail: bprosser@vrhb.bc.ca