Improved Questionnaire Design Yields Better Data: Experiences from the UKs Annual Survey of Hours and Earnings Jacqui Jones, Pete Brodie, Sarah Williams & Jane Carter UK Office for National Statistics

Overview Data Collection Methodology at ONS Methods standards Design standards Case study: The Annual Survey of Hours and Earnings Slide 1

Background to DCM at ONS Mid 1990s 1 st QTU set-up for Census Then expanded to include social surveys and business surveys Slide 2

Methods Standards Aim was to develop a framework of processes to be used for all DCM question(naire) development work Outcome was the Data Collection Methodology Improvement and Measurement Framework (DCM IMF) Slide 3

The DCM IMF Identification of data requirements + Expert Review + Evaluation Review Stage 2 Redesign & Development Stage 3 Pre-field Testing Stage 4 Stage 1 Stage 5 Field Implementation Identify and Disseminate Best Practice Stage 7 Stage 6 Field Testing Post-implementation Evaluation Slide 4

The DCM IMF Stage 1 Stage 2 Stage 3 Stage 4 Literature review Expert review Evaluation review Instrument development Expert review Qualitative testing Expert review Re- development Quantitative analysis Qualitative follow-up Slide 5

Design Standards Aim was to develop tested standards for the design of questions and questionnaires Outcome is draft standards – with constraints – colour and cross boxes Slide 6

Example of Design Standards

Case Study: ASHE ASHE collects data for earning statistics e.g. minimum wage ASHE questionnaires are sent out to employers to complete on behalf of selected employees 250,000 sample of employees Slide 9

ASHE: Need for Change National Statistics Quality Review New data requirements e.g. Pay for other reasons Pension data Existing data problems e.g. Possible inclusion of pay elements in the wrong responses Slide 10

Overview of the Old Questionnaire Very cramped 2 sided questionnaire which collected 26 data items 4 sides of additional notes Respondent coding Slide 11

Old Questionnaire Slide 12

Old Questionnaire Slide 14

Overview of the Old Questionnaire Unclear data requirements Inconsistent terminology Double-barrelled questions Slide 15

Old Questionnaire

ASHE Development Followed DCM IMF processes: Focus group (8 large businesses) Pre-field testing 30 cognitive interviews (5 waves of testing) Waves 1-3 concurrent interviewing Waves 4-5 retrospective interviewing Slide 17

ASHE Development 2004 Field tested 6 sided questionnaire: Split sample design (98% old, 2% new) Quantitative analysis Non-response Respondent burden (actual & perceived) Processing errors Slide 18

Results of ASHE Field Test Response (no statistical difference) Errors (no statistical difference) Respondent feedback (positive) Respondent burden (reduced per question) Some problems with question wording Costs (increased paper, printing, postage, processing) Time (increased processing time) Slide 19

6 sided to 4 sided compromises Minimum standard – based on testing - keep question & response category wording + instructions to stay with questions Challenge was to promote positive respondent perceptions and clear navigational path No time for further testing – redesign by DCM experts & quick perception telephone interviews Slide 20

Old Questionnaire Slide 21

ASHE 6 Sided Questionnaire

ASHE 4 Sided Questionnaire

2006 ASHE 4 Sided Questionnaire Slide 24

2005 Qualitative Follow-Up Looked at differences between responses to the 2004 old questionnaire and the sided questionnaire. Focussed on 5 key areas e.g. the affect on basic pay of a new other pay question 31 in-depth telephone interviews Slide 25

Old Basic Pay Question Slide 26

New Basic Pay and Other Pay Questions Slide 27

Other Pay Results Qualitative study found: Better comprehension of new questions Improved reporting of other pay Respondents preferred new layout Quantitative analysis found: Old questionnaire - other pay had been excluded by some Correction by imputation added 0.8% growth to basic pay at the all employee level Slide 28

What did we learn? Must have: clear & agreed data requirements agreement on the maximum number of pages DCM experts involved in redesign & testing method standards design standards improved methods for measuring data quality Slide 29

Thank You