What is meant by mode effect on measurement? A research study to identify causes of mode effects Gerry Nicolaas

Background Increasing use of mixed modes  Falling response rates  Rising costs of data collection Risk of reduced data comparability  Coverage error  Non-response error  Measurement error Need for practical advice to inform decisions about when to mix modes and how

Mixed Modes and Measurement Error Funded under the ESRC Survey Design and Measurement Initiative 3-year contract starting 1 Oct 2007 Collaboration between NatCen, ISER and independent survey methods consultant

Research Team National Centre for Social Research (NatCen)  Gerry Nicolaas  Steven Hope  David Hussey Institute for Social & Economic Research (ISER)  Peter Lynn  Annette Jäckle  Alita Nandi  Nayantara Dutt Independent Survey Methods Consultant  Pam Campanelli

Main Objective Practical advice on how to improve portability of questions across modes  Which mode combinations are likely to produce comparable responses?  Which types of questions are more susceptible to mode effects?

Research Design A literature review & framework of mixed modes  develop a conceptual framework  identify gaps in evidence base and formulate hypotheses to address gaps Quantitative data analysis  test hypotheses using existing datasets and new experimental data Cognitive interviewing  explore how respondents process questions in different modes

Mode is defined by: Interviewer presence (face-to-face, phone, none) Oral and/or visual transmission of information  Question delivery  Delivery of response options  Recording of response

Face-to-Face Interview ACASIWritten (CAI)‏Visual (CAI)‏Aural (CAI)‏ ACASIWritten (CAI)‏Aural (CAI)‏ SAQ in-int’wWritten (paper)‏ Visual (paper)‏ CASIWritten (CAI)‏Visual (CAI)‏ FTF (card)‏OralVisualAural FTF (no card)‏OralAural DescriptionResponseResponse Options Question

Telephone interview IVROral (CAI)‏Aural (rec)‏ Phone with showcards OralVisualAural TDEWritten (CAI)‏Aural TelephoneOralAural DescriptionResponseResponse Options Question

Self-completion A-WebWritten (CAI)‏Aur & Vis (CAI)‏ Web / Written (CAI)‏Visual (CAI)‏ SAQ (e.g. mail)‏ Written (paper)‏ Visual (paper)‏ DescriptionResponseResponse Options Question

Causality How, when and why to mix/choose modes Need to better understand causal mechanisms Development and testing of behavioural theory

ComprehensionRetrievalJudgementResponse Depth of cognitive processing: Sufficient Effort? Social norms: Willingness to disclose? Cognitive demands Aural/visual Interviewer presence Privacy/legitimacy I-R interaction Context information: Influence on processing? Time pressure R distraction R motivation Add. explanations I characteristics Sequential/ Simultaneous Qs Control Qaire Social desirability bias Satisficing

Hypotheses Short versus long response lists  Effects of interviewer presence on satisficing Agree-Disagree scales  Is acquiescence caused by satisficing, cognitive ability, social desirability ? Ranking versus Rating  Effects of interviewer presence Fully-labelled versus End-labelled scales  Effects of visual stimulus and interviewer presence Showcards versus No Showcards (face-to-face interview)  Effects of visual stimulus “Branched” versus “Non-branched” questions  Effects of item design (no mode differences expected) “Yes/No” versus “Code all that apply”  Effects of item design (no mode differences expected)

Question selection Questions were designed to vary by: Task difficulty Sensitivity Question type  Satisfaction  Other attitudinal  Behavioural  Other factual

Mixed Modes experiment Follow-up surveys to NatCen Omnibus (& BHPS)  Face-to-face, telephone and web comparisons  Experimental design with random allocation Limitation  Restricted to respondents with web access

Acquiescence (1) Hypothesis  Acquiescence is a form of satisficing  More satisficing in web than f2f & tel (no interviewer to motivate, explain, probe, etc) 12 Agree/Disagree questions  5-point agree/disagree scale  Use of opposite statements

Acquiescence (2) Initial results from the experiment  More acquiescence in f2f & tel compared to web Results from the cognitive interviews  Only 2 out of 23 cases of agreeing to opposite statements due to acquiescence  Justifiable explanations given for other 21 cases

Acquiescence (3) Example of justifiable agreement with opposite statements  N36: Compared to other neighbourhoods, this neighbourhood has more properties that are in a poor statement of repair.  N38: Compared to other neighbourhoods, this neighbourhood has more properties that are well kept.  Respondent: In this village, … it’s like half and half…”

Acquiescence (4) Interim conclusions  Use of opposite statements to detect acquiescence bias brought into question  Why higher rate of acquiescence in f2f & tel compared to web?

Other forms of satisficing (1) Hypotheses:  More satisficing in web than f2f & tel (no interviewer to motivate, explain, probe, etc)  More satisficing in tel than f2f (lack of physical presence of interviewer, lack of non-verbal communication, distractions, etc) Indicators of satisficing  Primacy effects in visual modes, recency effects in aural modes, middle category effects, item non-response

Other forms of satisficing (2) Initial results from the experiment  Primacy & recency effects: inconsistent patterns  Item non-response: no mode differences  Middle category effects: web respondents more likely to select middle categories than f2f & tel resps (also for agree/disagree scales – see acquiescence results) Results from cognitive interviews  Results suggest more satisficing in web & tel than in f2f interview

Mode effect or question design effect? Example:  “Code all that apply” frequently used in f2f interviews and self-completion surveys  Alternative format for tel interviews tends to be a series of Y/N questions Hypothesis  No mode effect  i.e. No differences if series of Y/N questions used across all modes

Mode effect or question design effect? Initial results from the experiment  F2F & tel resps more likely than web resps to say “Yes” in series of Y/N questions  Similar mode effect not found for f2f and web respondents in “code all that apply” format (nb this format not used in tel mode) Results from cognitive interviews  Questions raised about validity of “Yes” answers in Y/N series that may differ by mode

Continue with analysis of experimental data Papers  Causes of mode effects on survey measurement  The role of the interviewer in producing mode effects  The role of visual/aural stimuli in producing mode effects  The role of question format in producing ‘mode’ effects  Using cognitive interviews to explore mode effects Training course  Questionnaire design for mixed mode surveys What next?

For more details on this project, contact: