Download presentation
Presentation is loading. Please wait.
Published byLily Dowd Modified over 10 years ago
1
Improved Questionnaire Design Yields Better Data: Experiences from the UKs Annual Survey of Hours and Earnings Jacqui Jones, Pete Brodie, Sarah Williams & Jane Carter UK Office for National Statistics
2
Overview Data Collection Methodology at ONS Methods standards Design standards Case study: The Annual Survey of Hours and Earnings Slide 1
3
Background to DCM at ONS Mid 1990s 1 st QTU set-up for Census Then expanded to include social surveys and business surveys Slide 2
4
Methods Standards Aim was to develop a framework of processes to be used for all DCM question(naire) development work Outcome was the Data Collection Methodology Improvement and Measurement Framework (DCM IMF) Slide 3
5
The DCM IMF Identification of data requirements + Expert Review + Evaluation Review Stage 2 Redesign & Development Stage 3 Pre-field Testing Stage 4 Stage 1 Stage 5 Field Implementation Identify and Disseminate Best Practice Stage 7 Stage 6 Field Testing Post-implementation Evaluation Slide 4
6
The DCM IMF Stage 1 Stage 2 Stage 3 Stage 4 Literature review Expert review Evaluation review Instrument development Expert review Qualitative testing Expert review Re- development Quantitative analysis Qualitative follow-up Slide 5
7
Design Standards Aim was to develop tested standards for the design of questions and questionnaires Outcome is draft standards – with constraints – colour and cross boxes Slide 6
8
Example of Design Standards
10
Case Study: ASHE ASHE collects data for earning statistics e.g. minimum wage ASHE questionnaires are sent out to employers to complete on behalf of selected employees 250,000 sample of employees Slide 9
11
ASHE: Need for Change National Statistics Quality Review New data requirements e.g. Pay for other reasons Pension data Existing data problems e.g. Possible inclusion of pay elements in the wrong responses Slide 10
12
Overview of the Old Questionnaire Very cramped 2 sided questionnaire which collected 26 data items 4 sides of additional notes Respondent coding Slide 11
13
Old Questionnaire Slide 12
15
Old Questionnaire Slide 14
16
Overview of the Old Questionnaire Unclear data requirements Inconsistent terminology Double-barrelled questions Slide 15
17
Old Questionnaire
18
ASHE Development Followed DCM IMF processes: Focus group (8 large businesses) Pre-field testing 30 cognitive interviews (5 waves of testing) Waves 1-3 concurrent interviewing Waves 4-5 retrospective interviewing Slide 17
19
ASHE Development 2004 Field tested 6 sided questionnaire: Split sample design (98% old, 2% new) Quantitative analysis Non-response Respondent burden (actual & perceived) Processing errors Slide 18
20
Results of ASHE Field Test Response (no statistical difference) Errors (no statistical difference) Respondent feedback (positive) Respondent burden (reduced per question) Some problems with question wording Costs (increased paper, printing, postage, processing) Time (increased processing time) Slide 19
21
6 sided to 4 sided compromises Minimum standard – based on testing - keep question & response category wording + instructions to stay with questions Challenge was to promote positive respondent perceptions and clear navigational path No time for further testing – redesign by DCM experts & quick perception telephone interviews Slide 20
22
Old Questionnaire Slide 21
23
ASHE 6 Sided Questionnaire
24
ASHE 4 Sided Questionnaire
25
2006 ASHE 4 Sided Questionnaire Slide 24
26
2005 Qualitative Follow-Up Looked at differences between responses to the 2004 old questionnaire and the 2005 4 sided questionnaire. Focussed on 5 key areas e.g. the affect on basic pay of a new other pay question 31 in-depth telephone interviews Slide 25
27
Old Basic Pay Question Slide 26
28
New Basic Pay and Other Pay Questions Slide 27
29
Other Pay Results Qualitative study found: Better comprehension of new questions Improved reporting of other pay Respondents preferred new layout Quantitative analysis found: Old questionnaire - other pay had been excluded by some Correction by imputation added 0.8% growth to basic pay at the all employee level Slide 28
30
What did we learn? Must have: clear & agreed data requirements agreement on the maximum number of pages DCM experts involved in redesign & testing method standards design standards improved methods for measuring data quality Slide 29
31
Thank You Jacqui.Jones@ons.gov.uk
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.