Washington Group Cognitive Test Kristen Miller Questionnaire Design Research Lab National Center for Health Statistics, USA 1) Purpose of cognitive test 2) Structure of protocol 3) Types of cognitive test data & possible analyses 4) Things to keep in mind

Purpose of Cognitive Testing Interviews: Designed to examine the stages of the question response process – Comprehension, retrieval, judgment, response Analysis: Identifies… – potential response errors: (e.g. vague concepts, recall problems, inadequate response options) – patterns of interpretation

Traditional Cognitive Testing Protocol Semi-structured Data collected are qualitative Qualitative methodologies for analysis Small samples: approx. 10 – 30 participants

Washington Group Protocol Structured Questionnaire Designed to illustrate: 1. Administration ease 2. Interpretations 3. Factors considered by respondent 4. Degree of consistency with physical abilities 5. Differences between subgroups

Cognitive Interview Protocol I. Demographic Section: Country, language, gender, age, SES II. Question Testing Sections A. Core Question B. Interviewer Coding C. Open-ended Follow-up Probe D. Cognitive Follow-up Probe E. Functioning Follow-up Probe III. Health Questions: subjective health, chronic condition list IV. Interviewer Debriefing

Interviewer Code: Did the respondent… need you to repeat any part of the question? Yes/No have any difficulty using the response options? Yes/No ask for clarification or qualify their answer? Yes/No

Percent of respondents with no response problems Vision83.89% Hearing88.47% Cognitive85.72% Mobility91.03% Self-care93.65% Communication86.70% N=1,287

Number of response problems by age 01 or 23+n %12.56%29.15% Chi-square=19.74, p<.05

Open-ended Follow-up Probe 1. Can you tell me what you were thinking? Why did you answer [fill with respondent’s answer to the core question]?

Qualitative Data Varied in quality A lot missing Very brief Provides some insight – E.g. “Because I wear glasses”

Cognitive Follow-up Probes: 1. How often they had difficulty 2. The amount of effort they needed to put into the activity 3. The degree of concern or worry they had regarding the difficulty 4. Other domain-specific questions 1. Communication: Shyness question 2. Cognitive: Reason question

Cognition Cognitive Probe Do you believe that your memory or concentration difficulties are… [Mark all that apply]  because you have too many things to do?  because you getting older?  or, because of something else? Only busy: 28.78% Aging:25.84% Something else:43.07%

Reason by degree of problem SometimesA lotCan’t don Only busy89.05%10.95%0.00%137 Aging Something Else

Functioning Follow-up Probes 1. Presence of a condition 2. Use of an assistive device 3. Degree of difficulty doing related activities

Analysis of Functioning Data Examine the consistency between: – Washington Group Question responses and – Follow-up functioning questions responses Goal: explain the discrepancies – Misunderstood word? – Cultural difference? – Translation problem? – Interviewer error? – Error in the follow-up questions? Other data assist in explanation

Response patterns: Vision PatternWears Glasses Functioning Problem Reports Disability ANo BYesNo (corrected)No C YesYes, disability DYesYes (not corrected)Yes, disability EYesNo (corrected)Yes, disability FNo Yes, disability GYesYes (not corrected)No H YesNo

Response patterns: Hearing PatternAidDisabilityMissed WordsFunctioning Problem ANo B Yes CNo YesNo D Yes ENoYesNoYes FNoYes No G Yes HNoYesNo IYesNoYes J K No LYes NoYes M No NYesNo Yes O NoYesNo PYes No