What information to collect? Jan Smit Statistics Division, ESCAP With thanks to Wei Liu What information to collect? 29/07/2019

Outline Overview of ESCAP/WHO project Disability question set testing What questions were asked? Sampling & sample characteristics What info should be collected? Comparison between Washington Group and WHO questions What information to collect? 29/07/2019

ESCAP/WHO project ESCAP/WHO Project on Improving Disability Data 2004-2006 Goal: improve national disability statistics by applying the ICF framework and using international measurement standards Timeframe: this is one of the various last activities, although an extension of the project is currently negotiated with the donor (ROK) What information to collect? 29/07/2019

ESCAP/WHO project (cont.) Target group: statisticians and experts in disability from 20 countries Expected outcome: capacity to produce better disability data that meet policy needs Partners: Australian Bureau of Statistics, Washington Group What information to collect? 29/07/2019

Project countries What information to collect? 29/07/2019 Afghanistan Brunei Darussalam Cambodia China DPR Korea Islamic Republic of Iran Japan Hong Kong, China Laos Macao, China Republic of Korea Singapore Sri Lanka Thailand Vietnam Taking part in pilot test Fiji Indonesia India Mongolia Philippines What information to collect? 29/07/2019

Motivation for project Biwako Millennium Framework for Action (BMF) Principle 6 to “Strengthen national capacity in data collection and analysis concerning disability statistics” However, many data issues within ESCAP region What information to collect? 29/07/2019

Regional disability data issues Data not available to inform policy Disability prevalence is underestimated Traditional view of disability prevails at every layer of societies, incl. government No comparability of data; need for use of international standards and guidelines Lack of even basic information on persons with disabilities at the national level Data that is available is not comparable across sources Under-reporting of true incidence of disability Data does not sufficiently reflect the lived experience and needs of persons with disabilities The current standard classification system (ICF) not fully understood or implemented in most countries in the region What information to collect? 29/07/2019

Lack of data comparability What information to collect? 29/07/2019

Project components Phase 1: Training in the use of ICF for disability collection, in standard methodologies and in design of collection tools National action plans for improving disability information systems Preparation of disability statistics training manual & material What information to collect? 29/07/2019

Project components (cont.) Phase 2: Field testing of standard sets of disability questions through identical protocols Evaluation of results for application to real data collections Building a network of experts What information to collect? 29/07/2019

Main project activities 4 regional workshops for advocating and training on the ICF approach and its implementation Pilot testing of disability questionnaires for use in census and surveys and results analysis and dissemination The ESCAP/WHO Disability Statistics Training Manual What information to collect? 29/07/2019

Disability question set testing Study 1: Sensitivity and specificity testing (and more) Study 2: Test-retest reliability Study 3: Cognitive testing Study 2: 100 respondents re-visited and asked same questions again. Level of agreement expressed in kappa statistic generally OK. Study 3: 100 willing participants were asked about their understanding of the questions. What information to collect? 29/07/2019

Pilot test: Study 1 Through NSOs of five countries Fiji (FJI) Indonesia (IDN) India (IND) Mongolia (MNG) Philippines (PHL) Data collection period: May-July 2005 In local languages What information to collect? 29/07/2019

Questions Demographic characteristics 6 Washington Group questions (W) 53 WHO questions (B and D) 17 body structure/function questions from World Health Survey 36 activity & participation questions from Disability Assessment Scale (DAS II) 4 Australian Bureau of Statistics questions on need for assistance (E) What information to collect? 29/07/2019

Activity & participation quest. Understanding & communicating (D1) Getting around (D2) Self care (D3) Getting along with people (D4) Life activities (D5) Participation in society (D6) Incl. 2 questions (D6.2-3) that might be considered as addressing environmental factors What information to collect? 29/07/2019

Sampling Each NSO was requested to administer the questionnaire to a sample of at least 1,000 respondents of 18 years and older representative of the national population in terms of age, sex and educational attainment Interpreted by the collaborating NSOs in different ways Sample not representative for whole population What information to collect? 29/07/2019

Sample characteristics: Sex Country Sex IND IDN PHL FJI MNG Total Female 245 449 606 527 596 2423 Male 345 351 433 468 466 2063 Total 590 800 1039 995 1062 4486 Col % Col % Col % Col % Col % Col % Female 41.5 56.1 58.3 53 56.1 54 Male 58.5 43.9 41.7 47 43.9 46 Total 100 100 100 100 100 100 What information to collect? 29/07/2019

Age What information to collect? 29/07/2019

Education What information to collect? 29/07/2019

Marital status What information to collect? 29/07/2019

Employment status What information to collect? 29/07/2019

What info should be collected? Depends on question(s) to be answered (already discussed) What domains are most important? No question about respondents overall judgement of their own health (or perhaps happiness, quality of life) What are the main underlying ‘factors’? (Exploratory) factor analysis What information to collect? 29/07/2019

Factors Day-to-day life (D1.3, D5.1, D5.2, D5.3, D5.4, D5.5, D5.6, D5.7, D5.8 and D6.8) Self-care (W5, D3.1, D3.2, E1 and E2) Resources and emotion (D6.4, D6.5, D6.6, D6.7 and D6.8) Getting along with people (D4.1, D4.2, D4.3 and D4.4) Mobility (W3, B1.13, B2.1, B2.2 and B2.5) What information to collect? 29/07/2019

Factors (cont.) Hearing (W2, B1.8 and B1.9) Learning and understanding (D1.4 and D1.5) Seeing (W1, B1.6 and B1.7) Pain and discomfort (B1.1 and B1.2) Sadness and anxiety (B1.16 and B1.17) What information to collect? 29/07/2019

Prevalence What information to collect? 29/07/2019

Sensitivity & specificity Base of comparison + _ Test questions True positives (a) False positives (b) False negatives (c) True negatives (d) Sensitivity: __a___ a + c Specificity: __d__ b + d What information to collect? 29/07/2019

Sensitivity, specificity, gamma WHO question Sensitivity Specificity Gamma Seeing B1.6 0.60 0.91 0.83 B1.7 0.63 0.90 0.84 B1.6 and B1.7 0.58 0.94 0.85 Hearing B1.8 0.39 0.98 0.91 B1.9 0.38 0.97 0.88 B1.8 and B1.9 0.34 0.98 0.90 Mobility B1.13 0.47 0.93 0.80 D2.5 0.51 0.91 0.79 B1.13 and D2.5 0.43 0.96 0.82 What information to collect? 29/07/2019

Sensitivity, specificity, gamma WHO question Sensitivity Specificity Gamma Cognition D1.1 0.46 0.88 0.70 D1.2 0.48 0.92 0.79 D1.1 and D1.2 0.43 0.94 0.79 Self care D3.1 0.51 0.98 0.96 D3.2 0.58 0.98 0.96 D3.1 and D3.2 0.49 0.99 0.96 Communication D1.5 0.31 0.96 0.80 D1.6 0.31 0.96 0.80 D1.5 and D1.6 0.28 0.97 0.80 What information to collect? 29/07/2019

Ongoing research… Can info on some ‘important’ domains be recovered from a limited number of questions/variables? E.g. can D5.5 be explained by the 6 WG questions (or WHO counterpart questions) + demographic info? Initial results in terms of model fit promising Difficulty is accuracy of prediction What information to collect? 29/07/2019

Not only collect data… But document and preserve data collection exercise Instrument, scope, coverage, etc. Response options, labels, etc. Microdata Management Toolkit www.internationalsurveynetwork.org And preferably disseminate micro data (unit records) Possibly anonymized What information to collect? 29/07/2019