Measuring Quality Outcomes John N. Morris, PhD, MSW Director Emeritus Institute for Aging Research Hebrew SeniorLife, Boston November 2011

Key Question: How does care at your nursing facility stack up against external standards?

Agenda interRAI and interRAI Suite Application of interRAI data – the QIs Conclusions

Agenda interRAI and interRAI Suite Application of interRAI data – the QIs Conclusions

Who are We? A non-profit, international group of researchers and clinicians Key interests: –Science (e.g., cross-national comparisons) –Assessment instrument development –Support of implementation in countries across the globe Holds copyright to RAI assessment instruments Grants royalty-free licenses to governments and care providers License software vendors around the world

interRAI Suite of Assessment Instruments interRAI has created a multi-domain suite of assessment instruments, providing a common language to describe persons across sites of care The instruments support assessment and care planning in aged care, mental health, and disability services Each assessment instrument has: –Items common to other instruments – e.g., ADLs, cognitive function, mood, pain, falls, –As well as specialized items that are exclusive to that setting __ e.g., suicide and criminality items in the mental health tool

InterRAI Assessment Instruments First what we do not do interRAI instruments are more than just a cobbled together set of measures

Why are interRAI Assessments Different? Developed by international panel of experts: geriatrics, gerontology, assessment, and health services research Carefully tested psychometric properties New design: assessment drives decision-making at all levels, from clinical to policy –Collect data once, use many ways Develop compatible systems across health care sectors

Why are interRAI Assessments Different? cover all relevant domains –individuals’ strengths and weaknesses –include items on caregivers, environment –tradeoff of breadth and length not only self-report –use all possible sources of information include full definitions, time delimiters, examples, exclusions training manuals

Examples of Domains Covered Intake/Initial History Cognition Communication/Vision Mood/Behavior Psychosocial Well-being Functional Status Continence Disease Diagnoses Health Conditions Oral/Nutritional Status Skin Condition Medications Treatment/Procedures Responsibility Social Supports Environmental Assessment Discharge Potential/Status Discharge

Field Test – Can we trust the items interRAI has looked at user acceptability – the results are most encouraging Completed reliability in many countries Used Kappa statistic – below.4 poor,.41 to 60 moderate, higher scores considered substantial agreement

Average Weighted Kappa, by Instrument and Type of Item Good Acceptable

Agenda interRAI and interRAI Suite Applications of interRAI data – the QIs Conclusions

It’s not enough just to measure…..

Applications of interRAI Data ASSESSMENT Care Planning ScreeningQuality Case-MixPolicy

Nursing Home Quality Indicators: So What is a “Good” Home?

Different Ways to See Quality Tender loving care -- a focus on daily life, social engagement, and happiness –Focus on an ideal image of what happens in a home setting. Standards set by the service industry –Nature of facility -- Friendliness of staff, clean and comfortable rooms, the quality of the food –Qualifications of staff –Staffing levels –Adherence to standard care guidelines Resident satisfaction self reports Maximize resident outcomes–“Quality Indicators”

ALL ARE IMPORTANT! Unfortunately, the correlation among these measures is low This holds for: –Survey results generated by government agencies –Person-based satisfaction surveys –Staffing levels –Person change measures

Different Types of Quality Indicators -- Examples Problem prevalence Proportion of residents physically restrained Proportion with daily pain Problem Incidence – “Occur over time” Falls over prior 90 days Urinary tract infection Change in Status over time Rate of Decline in Physical Functioning Rate of Improvement in Mood

Defining Quality Process standards –Tender loving care – a mirror of life at home –Adherence to specified care protocols Environmental standards –Cleanliness, quality of food –Caregiver credentials –Caregiver staffing levels –Physical environment Person’s satisfaction Person’s status and how changes over time - Quality Indicators

Thus We Have To Make a Decision on How to Assess Quality For interRAI how the person changes over time matters – what happens with respect to functional, clinical status, and social aspects of daily life Such outcomes are measured with program-based Quality Indicators – QIs

Our Goal is Clear Once in such a program the person has the right to expect staff to take every step possible to: –Solve immediate problems –Maximize the person’s functional potential –Enhance the person’s quality of life

But the Challenges are Many What specific quality measures should we use – what are the most appropriate yardsticks? What can we reasonably expect a facility or program to achieve? Looking at nursing homes, for example, we have to comprehend the person’s underlying potential for change

High Decline Rates for Persons in NHs Are Not Inevitable !

Inter-Program Variability Inter-program variability in outcome rates can be enormous The next two graphs depict mean (or average) mobility change rates for nursing facilities in Canada and the United States

Technical Issues Access the computerized interRAI longitudinal assessments of the program participants Transform these person-level data into program level QIs -- –introduce risk adjustment –decide how “different” is different” –Ultimately identify the “bad” and “good” providers Establish the relationship among quality indicators Create a summary Quality Composite

Risk Adjustment of QIs Necessary to compare “apples to apples” Account for bias due to variations in program admission practices Three ways to adjust –Stratification –Exclusions –Regression-based adjustment Individual risk factors (covariates) Direct stratification

ADL Adjustment Example The country is Japan The comparison is across three types of facilities The outcome is the proportion of residents who decline in “late loss” ADLs over a 90- day period

Example Risk Adjustment : ADL Decline in the Three Types of Japanese LTC facilities

History of QIs Development goes back two decades We have worked to define what matters – it is something the industry in each country must face In our cross-national work, interRAI created standardized versions of 80+ NF QIs Through analysis and deliberations, the set has been reduced to 19 measures

Functional QI Items QI Measure ADLs X Locomotion X Falls X Cognition X Communication X

Clinical QI Items QI Measure Continence X Behavior X Infection X Pain X Mood X Behavior X Pressure Ulcer X

Utilization QI Items QI Measure Catheter X Feeding tube X Restraint X

Let us Look at Rates of Change In Nursing Facilities First we will look at the individual QIs We contrast Michigan facilities against the interRAI cross-national standard With exceptions – Detroit facilities look pretty good

Multi-Dimensionality of QIs –Facilities that perform well in one dimension may perform at a more average or even poor level in other selected dimensions of quality –How do we identify the “best” facilities or programs? If a facility does poorly in any one dimension is the facility poor? If it does superbly in multiple dimensions, is it best?

A NH Example Facilities with the lowest rates of falls are no more likely than other facilities to have low rates UTIs, antipsychotic use, cognitive change, and pain change. On the other hand, they are somewhat more likely to experience an improvement in early loss ADLs and incontinence

Creating The QI Composite 8 functional change QIs -- employing both decline and improvement variants) 10 clinical complexity QIs – e.g., incontinence, pain 80% of composite score from functional measures 20% of composite score is from clinical complexity measures

Distributional Properties of NH QI Composite The next table is based on data from 2 Canadian provinces and 5 US states Data are weighted

An International Standard Given the cross-country relative equivalency in rates, interRAI has adopted this North American Distribution as its International Standard And we show that the distribution differs from one jurisdiction to another

Several Use Examples for Quality Indicators The type of report provided to a facility Assessing inter-state variability Measuring change in facility performance over time – an example from Massachusetts

Key points The position of your QI score – are you better, worse, or about average of other facilities in the comparison group (e.g., a state) The range of scores across facilities in the state – some are in similar ranges others are quite diverse If you are doing poorly in an area – is this a key issue that must be reversed? If you are doing well in an area – do you understand why, does it make sense? If it does not, how can you expect to maintain this QI area as positive over time? 50

Example: Mid-Loss ADL Improvement TECHNICAL DEFINITION Percent of residents who improve in Mid-Loss ADL functioning (transfer, locomotion) or remain completely independent in these areas NUMERATOR – two types of persons: Residents at the follow-up assessment who are more independent than they were at the prior assessment Residents who were independent at BOTH the baseline and follow-up assessments DENOMINATOR: All residents with a valid follow-up assessment and a valid prior baseline assessment 51

Explanation of Example Higher scores on this measure imply better quality of care. The adjusted QI score over the reporting period is trending downward. The amount of change is trivial. –Regardless, performance is suffering over the reporting period. Performance is above the reference sample median at the end of the period. 52

Proportion of residents who improve in Mid-Loss ADL functioning (transfer, locomotion) or remain completely independent in Mid-Loss ADLs – A HIGH Score is Good Percentile Rank Adjusted QI Score 2010Q32010Q42011Q12011Q2 Quarter 53

More Detailed Data – Gives you the count of the number of residents who experience the condition referenced by the QIs QuarterAdjusted QI Score Raw QI Score # of Residents in Numerator # of Residents at risk 2010 Q Q Q Q

Inter-State Comparison

Summary NH Quality Indicator Index – States and Provinces

Changes in Quality Over Time A 21 quarter comparison of quality in Massachusetts Nursing Homes – there is real improvement 57

Agenda interRAI and interRAI Suite Applications of interRAI data -- QIs Conclusions

Final Comments interRAI Quality Indicators represent –A large set of measures –They cross multiple dimensions –They challenge providers to improve functional outcomes and relevant clinical complexities Choices have been made and as one considers the local situation on the ground you can select from among these measures – setting local standards and comparisons to international standards

Final Comment Choices have been made and one can select from among these measures to assess program performance – setting local standards and comparisons to international standards