Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray

Presentation outline Background to study Aims of research Methodology Designing a behaviour code frame Using the behaviour code frame Analysing the data Lessons learned

Background to study English Longitudinal Study of Ageing (ELSA) Dependent Interviewing (DI) Two types of data item  Feed Forward (DI)  Non-Feed Forward (non DI) Little evaluation of the impact of DI on data quality conducted to date

Aims of research Research aims:  To assess how DI affects data quality  To explore how Rs react to feed-forward phrases  To find whether this varies by nature and sensitivity of topic Methodological aim:  To explore the combination of CARI and Behaviour Coding as methodological tools

Methodology Computer Assisted Recorded Interviewing (CARI)  Computer acts as a sophisticated tape recorder  Unobtrusively records interaction Behaviour Coding  Codes systematically applied to interviewer- respondent behaviours  Uncover and assess problems with questions Two methods combined for this study

Designing the code frame

Principles for good design Code frame adapted from Cannell et al (1989) Short and straightforward Few, easy to apply codes Discrete Broad rather than specific

Behaviours coded Question asking behaviour for interviewers Immediate response behaviour for respondents Whether partner intervened (concurrent interviews) Final outcome of the entire exchange

Two behaviour code frames Two code frames designed:  DI (feed-forward) items  non DI (non feed-forward) items First level exchange (initial utterance) Code what occured before other person speaks

Code frame

Behaviours coded Interviewer/Interviewer feed-forward Respondent/Respondent feed-forward Whether partner intervened Final outcome One code per behaviour

Interviewer codes Exact Wording/Slight Change01 *Major change02 *Omission03 *Question became a statement04 *Inaudible Interviewer/Other05 Not applicable 99 *denotes where notes must be made

Interviewer feed-forward codes FF item read as worded/slight change01 *FF statement became a question02 *FF question became a statement03 *Other major change04 *Omission05 *Inaudible Interviewer/Other06 Not applicable 99 *denotes where notes must be made

Respondent codes Adequate Answer 01 *Inadequate Answer/Elaboration 02 *Clarification 03 Question Re-Read 04 Don't Know 05 Refusal 06 *Inaudible Respondent/Other 07 Not applicable99 *denotes where notes must be made

Respondent feed-forward codes *Affirmed FF item - adequate 01 *Disputed FF item - adequate 02 *Inadequate Answer/Elaboration 03 *Clarification 04 Question Re-Read 05 Don't Know 06 Refusal 07 *Inaudible Respondent/Other 08 Not applicable 99 *denotes where notes must be made

Partner intervention codes *Yes 01 No 02 Not applicable (no partner present)99 *denotes where notes must be made Code used for where the respondents partner intervened and subsequently answered for the respondent

Final outcome codes Adequate Answer 01 *Inadequate Answer 02 Don't Know 03 Refusal 04 *Inaudible/Other99 *denotes where notes must be made Coding whether the final answer meet the objective of the question

Technical details

CARI equipment Equipment testing  External microphones CARI built into Blaise program Recording switched on and off at relevant items Sound files automatically generated and saved Sound files removed from interviewer laptops  Macro run  Data sticks (USB)

Behaviour coding system Conducted within Blaise Coding program designed for this purpose  Weststat testnote software Three windows displayed simultaneously  Blaise interviewing screen  Coding entry screen  Sound file (.wav) Automatically routed through interview Tags to skip to relevant data items

Using the code frame

Sound file

Blaise interviewing screen

Coding program

Data preparation and analysis

Organising the data Two types of data  Behaviour codes (quantitative)  Coder notes on non-standard behaviours (qualitative) All data automatically stored in Excel tab delimited file One Excel file produced for each coder Excel files amalgamated Exported into SPSS

Data preparation More cleaning than expected Two main problems:  Duplicate files (limitations of system used)  Incorrect code frame used at interviewer and respondent behaviours (DI and non DI items)

Analysing the data SPSS Frequencies and crosstabulations Coder notes provided additional context Very small base sizes at some items due to routing

Advantages and disadvantages of our approach and lessons learned

What worked CARI  Unobtrusive in nature  Minimal impact on interviewers and respondents Behaviour coding  Able to run statistical analyses  Able to draw conclusions  Method of coding easier than paper (routing)

What didn’t worked CARI  High number of inaudible or hard to hear cases (1/3 of respondents)  Purchased speakers to help Next time…  Fully re-test microphones  Probe respondents reasons for not giving consent to being recorded

What didn’t worked Behaviour coding  Lengthy and costly process Coding (approximately 45 mins per interview) Data cleaning  Over complex code frame  Coding method found cumbersome, limited and error prone  Coder judgement not measured

Next time... One code frame only Build in sufficient time for each stage Clear rationale for behaviour coding Inter-coder reliability test (Kappa score) Adequate sample for uncommon questions Create more sophisticated, less error prone coding system

Discussion & Questions...