Field test options instrument, instructions, non-response/ refusals, and Interviewer debriefing Washington Group Regional Training Workshop Rio de Janeiro 19 – 20 September 2005 Margie Schneider HSRC, South Africa

Recap What we have covered so far: –Purpose of the questions and where they came from –Core set of questions and extended set and Q X Q specifications –Objectives of the test –Translation protocol What we still need to go through: –How to design the test and related issues –Enumerator training –Plan for analysis and report writing

Different components of the testing process Pre-testing and expert review Testing internal validity on people with known disability status (quota sampling and linked to cognitive testing) Pilot testing – as for full Census or survey Full field test Different sampling approaches will be discussed in relation to different testing protocols

Field Test options 2 questions to be answered: 1.What set of questions should be used in field test? 2.What field test approach should be used? Consider your own context, need, funding, skills, etc. in deciding

Question sets 1.Core set – (6 domains of functioning): Compulsory 2.Core + extended set on core domains : highly recommended 3.Core + extended set + further additional questions: nice to have for analysis 2 additional domains (learning and interpersonal interactions) Psychological distress 4.Country specific question set : nice to have for comparison with prior data

What approach to testing? What factors to consider in deciding: –Country needs –Capacity to undertake test –Funds available –Time available –Planned activities, e.g. other surveys –?

Census or national survey Use of question set in planned Census or national survey Advantages: –Large sample (5000 – or whole pop) –Minimal additional cost for disability questions Disadvantages: –Not sufficient space for extended set –Need to administer extended set and cognitive test to sub-sample (costs of time and additional training)

Special study – small sample Select sample on basis of known ‘disability’ status Allows for construction of 2X2 table for sensitivity and specificity calculations and identifying true and false positives and negatives – use of ‘gold standard’ Suggest around 200 true positives and 200 true negatives Interviewer does not know status of respondent (unless visible and obvious)

Special study (contd.) Advantages: –Larger set of questions –Close observation of interview –Cost is not exorbitant Disadvantages: –Might not get full population representation –Cost of establishing true status is high –What counts as the true positive? (beyond the easily observable) –What is the ‘gold standard’?

Special study – large sample Uses population based sample Sample size based on expected prevalence of the different types of activity limitations Prevalence rates are usually low and so need large sample to yield enough ‘disabled’ respondents

Special study – large sample ( contd) Advantages: –Large data set –Detailed set of questions - more than extended set –Provides an indication of prevalence using core and extended sets Disadvantages: –Costly and time consuming –Requires capacity to run and analyse

Field test and cognitive test All respondents have core set One sub sample of respondents have cognitive test (core + extended are embedded) A second sub-sample of respondents do –Core + extended –Core + extended + further Qs –Core + extended + country specific questions Some do all ?

Non-responses and refusals How do you define these? How do you deal with these? Aim to: –Reduce non-response and refusals –Manage them when they do arise

Reducing NR and refusals Use up to date sampling frame Clearly defined reasons for data collection Preparing the way – prior contact, letters, etc. Adequate interviewer training Allow budget for call-backs Plan for follow up of NR and refusals Separate refusals, part refusals, non- contacts and sample loss (e.g. vacant dwellings)

Effect of each on analysis Sample loss: does not create bias but reduces sample size; need large enough initial sample to take these into account Refusals and non-contact: bias where these respondents might be different to those respondents reached Need to understand reasons for refusals and non-contacts – during pre- testing and pilot stage

Interviewer debriefing Interviewers are well placed to evaluate process and content at an early stage Interviewers should note comments in margins of questionnaires Different approaches: –Group discussion (focus group technique) –Interviewer rating forms –Standardised interviewer questionnaires –Combination of all three

Enumerator training Washington Group Regional Training Workshop Rio de Janeiro 19 – 20 September 2005 Margie Schneider HSRC, South Africa

General points Select some disabled interviewers Importance of all interviewers having a good understanding of what disability is and is not and how it relates to the questions asked Avoid using term ‘disabled’ or ‘with disabilities’ All interviews are face to face Translation training

General points (contd.) Confidentiality and understanding what this means Getting informed consent (ethics clearance?) Interviewers must be able to explain purpose of survey Read questions in set order and with set wording Editing in field of completed questionnaires Submitting of completed interviews to head office

Types of interviews Direct: respondent answers for him or herself Interpreted: an interpreter ‘translates’ and respondent answers directly Facilitated: a third party assists in explaining (e.g. intellectually disabled person) Proxy: a person responds for another (e.g. child)

Interviewing disabled people Show respect and treat the person like anyone else Don’t use first names unless permitted Address the person directly (not their attendant) Ask how you can adapt your presentation to make it easier (no need to ask what is wrong with person)

Hearing difficulties Lip reading Lighting Face the person Get person’s attention before speaking Reduce background noise Set the context – especially when changing topics Use written communication (literate)

Physical difficulties Accessibility of building where conducting interviews Presence of attendant and confidentiality issues Get to same level (e.g. sitting for person using wheelchair) Person to be seated comfortably Address person directly Pointing may be difficult Person may need breaks to move around

Visual difficulties Large print and small print for cue cards Braille versions of cue cards Good contrast printing for pictures and print (black on white or yellow) Identify yourself and others in the room verbally

Communication difficulties Clarify preferred mode of communication Repeat what you think was said to clarify unclear speech Limit to yes / no questions

Specific learning difficulties Manage problems in spatial orientation, hand-eye coordination Limit auditory, visual and tactile distractions Avoid written text Explain carefully (if verbal language skills are affected)

Intellectual difficulties Be careful with informed consent Explain terms simply Listen carefully Have familiar person (friend or relative) close by Use pictures or role play with little human or animal figures

Emotional or mental health difficulties Side effects of medication Break up interview if too fatigued Give encouragement and support Manage expressions of frustration Manage stress

Hidden difficulties Might not come forward with information because of fear of stigma Effect of medication May need to break up interview