Presentation is loading. Please wait.

Presentation is loading. Please wait.

Total Survey Error in Disability Assessments Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS) Brad Edwards.

Similar presentations

Presentation on theme: "Total Survey Error in Disability Assessments Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS) Brad Edwards."— Presentation transcript:

1 Total Survey Error in Disability Assessments Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS) Brad Edwards and Tamara Bruce, Westat Presented at the International Total Survey Error Workshop Quebec, Canada June 2011

2 Overview NHATS and capacity measures Self report and performance NHATS assessments Strategies for error reduction -design, training, standardization, parsing out nonresponse Pretest results National experience Future research 2

3 NHATS Westat working with a team led by Johns Hopkins Bloomberg School of Public Health Funded through cooperative agreement from NIA CAPI panel study with 9,000 Medicare beneficiaries, annual interviews starting 2011 New and improved measures of disability for producing data on trends and trajectories; self- reports, mental and physical assessments, eventually biomarkers, links with admin data 3

4 Conceptual Framework Blend of Nagis model and WHO perspective Distinguish explicitly between –Capacity to do something – basic building blocks –What people actually do in actual environment – activities Accommodations may fill gap 4

5 Capacity Is Critical Measures of capacity over time are key elements in understanding individual patterns of progression to activity limitations. They allow us to track trends in function that are independent of environmental changes or accommodations, for understanding the disablement process, and as targets for interventions to prevent or slow disability. 5

6 Reducing Error in Capacity Measures Questionnaire design Importance of performance measures Challenges for quality management –New to surveys –Complex –Expect high level of interviewer variance 6

7 Physical Capacity: Upper Extremity Self report Able to –Put book on shelf/reach overhead –Open jar/grasp small object Performance Grip strength 7

8 Physical Capacity: Lower Extremity Self Report Able to –Walk 6/3 blocks? –Kneel/bend over? –Lift and carry 20/10 pounds? –Walk up 20/10 stairs? Performance Walking speed Balance stands –Side by side –Semi-tandem –Full-tandem –One leg eyes open –One leg eyes closed Chair stands 8

9 Cognitive Capacity: Memory Self report At present time? Memory problems interfere with activities? Memory compared to one year ago? Performance 10 word recall –Immediate –Delayed 9

10 Other Cognitive Performance Measures Orientation –Day of week, date, naming President and Vice President Overall cognitive screening/executive function –Clock drawing Attention & interference/executive function –Stroop test (computerized) 10

11 Standardizing Administration 11 Performance measures of capacity can be hard to standardize in large-scale surveys with lay interviewers NHATS uses multiple methods to improve consistency & accuracy: –Activities Booklet design & use within CAPI framework –Standardized presentation of cognitive tasks using flash displays –Survey design & interviewer feedback using CARI coding –Formal certification process using live respondents –Follow-up web-based recertification

12 Training Approaches Broader scope of demands on NHATS interviewers for performance measures –tests require using a variety of equipment –navigation of unfamiliar environments –potentially uncomfortable instructions to respondents Integration of video components –recruitment –in-person interviewer training modules such as walking course –certification & re-certification processes

13 Walking Course Timing 1

14 Walking Course Timing 2

15 Walking Course Timing 3 15

16 Training Video Demo 16

17 Item Nonresponse Performance measures used in depth set of exclusion criteria to minimize burden High completion rate for both cognitive & performance tasks for those eligible Comprehensive list of reasons why test was not conducted allow detailed analyses Majority of unit nonresponse due to inability to complete easier task & safety concerns Performance related to age, health rating, care setting, memory 17

18 Respondent Reactions Avoiding test and performance Training interviewers how to respond to requests (e.g., How did I do?) Physical activities: 7 th inning stretch Positive feedback at the end Interest in assessing respondent satisfaction and rapport in relation to nonresponse in later rounds 18

19 Comparison of Performance and Self Report High functioning performance group (attempting one-leg balance stands, meeting accuracy thresholds for Stroop) reported better self-reported health and memory and (for the Stroop) had higher performance on two memory tests 19

20 Comparison with Other Surveys NLTCS screening questions in NHATS form a bridge from previous trend data to detailed NHATS self reports and assessments Possibility of future bridge to 6 disability questions developed by NCHS and used on the American Community Survey 20

21 Conclusions Reducing error in capacity measures based on performance assessment –sharpens understanding of individual trajectories and accommodations –is expected to improve predictive ability for health outcomes NHATS protocol –standardizes administration –focuses on reasons for nonresponse 21

22 Conclusions (2) As a new survey, NHATS has been able to give consideration to the implications of doing these types of assessments from the outset, starting with interviewer recruitment and training. Experience from pretesting has led to refinements of these procedures (e.g., use of videos in recruitment; certification procedures in training) and to the NHATS Activities Booklet. Training materials and data collection instruments will be available later this year at 22

23 Future Research Interviewer variance study Examination of –interviewer and respondent conditioning effects –drift trajectories –item nonresponse as predictor of unit nonresponse Error comparison/tradeoffs between self-reports and performance Analysis of cognitive performance and data quality More development –distance learning –SPC charts, integration with paradata for management –re-certification via WebEX, 2-way video 23

24 24 For more info, contact: Brad Edwards or Tamara Bruce

Download ppt "Total Survey Error in Disability Assessments Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS) Brad Edwards."

Similar presentations

Ads by Google