1. Methods to Detect and Control Bias in Comparative Analyses of Disability Statistics 5 th Annual Meeting of the Washington Group on Disability Statistics Rio de Janeiro, Sept 21-23 2005 Bedirhan Üstün & Nenad Kostanjsek Classifications Assessment Terminology Measurement and Health Information Systems www.who.int/classifications/icf

2. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION …

3. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Temperature: Hot, Mild or Cold ??? Measured Real temperature 20 C: Iceland - Hot Germany-Mild Brazil- Cold

4. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Big Mac Index University Professor in Turkey gets 3000 USD. University Professor in the Switzerland gets 8000 USD. Big Mac in TR = 1.5 USD Big Mac in CH = 4 USD Salary = 2000 Big Macs in both

5. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Health & Disability Survey Data prove useful to complement National Health & Disability Information Systems –more representative of entire populations –direct information from individuals on their own experiences rather than professionals or surveillance information can be tailored to specific management and evaluation purposes in mind: –provide empirical data for system performance assessment More like to be standard and accurate –gathered more carefully

6. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Numerous National and International Health & Disability Surveys exist and they differ in: Content Questions Response scales Samples Timing DOWNSIDE of Surveys

7. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Problem of Comparability in International Health Data Mortality statistics 3 x Health Survey data 6 – 8 x Disability Survey data 30 – 60 x Health Interventions 40 – 300 x

8. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 1. Same Concepts Examples: –Disability = Impairment, Activity limitation & Participation restriction (ICF definition) –Disability = Impairments & simple activities –Hearing = body function –Communicating = activity and participation domain –The self-care category includes the sub- categories: washing, dressing, toileting….

9. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 2. Same Domains & Questions

10. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 2. Same Domains & Questions (cont.)

11. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 3. Same response scale (cont.) Different response categories for severity scale –none, mild, moderate, severe… –Numerical scales (0,1,2,3,4) (1,2,3,4,) –Likert visual –Percentage visual scale Different response categories for frequency scales –always, usually, sometimes, never or all the time, –once, a few times, sometimes, many times

12. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 3. Same response scale (cont.)

13. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 4. Translation protocols Translation/interpretation of questions Translation/interpretation of response scales The meaning of the underlying construct may vary The relation between the questions and the underlying construct may vary

14. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 4. Translation protocols (cont.) The item is difficult or impossible to translate because there is no equivalent term in the local language. The item, when translated, has a meaning that is more narrow than the original term. Only part of the connotations of the original term carries over into the local language. The item, when translated, has a meaning that is expanded beyond the meaning found in the original. The translated items has additional meanings that would change the interpretation of the item. The item is translated into exactly the same word that has been used for another item in the questionnaire. Words that are different concepts in the questionnaire become merged into the same concept in the local language. The item can be translated, but there is a cultural applicability problem with the item, or with the definition, or with the examples used for the item.

15. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 5. Same application mode – same stimulus Respondent Input Output

16. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Comparability 6. Measuring and adjusting for shifts in cut points “Different Rulers” Do response categories mean the same in different populations? Very good Very bad Good Bad Moderate True MobilityResponse ScaleA B C

17. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Systematically inconsistent reporting in different surveys men and women age groups education groups economic level education levels cultural and ethnic groups occupations levels of health insurance.

18. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for adjusting for shifts in cut points Calibration Tests: Vision used extensively in international studies Distant Vision standard Snellen’s chart Near vision standard card for use in illiterate populations used extensively in international studies Distant Vision standard Snellen’s chart Near vision standard card for use in illiterate populations

19. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for adjusting for shifts in cut points Calibration Tests: Cognition Verbal fluency names of animals in one minute record correct number & errors Simple Memory Immediate recall delayed recall (20 min.) of word list Colour cancellation test time taken number of correct and wrong cancellations selected from WHO international studies on Culture-fair Cognitive Battery

20. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for adjusting for shifts in cut points Calibration Tests: Mobility Posturo-Locomotor-Manual test Time to stand up Time taken to complete task 3 trials stand up from sitting position walk one step lift up weight of 2 kgs walk 6 metres place weight at shoulder weight lift up weight walk back towards chair place weight back return to chair and sit down

21. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Vignettes Methods for adjusting for shifts in cut points Vignettes Mobility [Mary] has no problems with walking, running or using her hands, arms and legs. She jogs 4 kilometres twice a week. [Anton] does not exercise. He cannot climb stairs or do other physical activities because he is obese. He is able to carry the groceries and do some light household work. [David] is paralyzed from the neck down. He is unable to move his arms and legs or to shift body position. He is confined to bed.

22. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … “bias” Differential Item Functioning a distortion (possibly systematic and non-random) of the results due to a measurement error affects all measurements in the same way violates the fair comparison of results across different groups –substantive implications? –statistically significant? –random or systematic?

23. CLASSIFICATIONS … BUILDING BLOCKS OF HEALTH & Disability INFORMATION … Methods for Cross-Population Comparability  Same concepts  Same domains & questions  Same response scales  Translation protocols  Same application mode – same stimulus  Correction for confounders (education, income…) using known calibrators  Use modern Item Response Theory (IRT) approaches