1. Measuring Respondent Burden to Statistical Surveys
John-Mark Frost, Sarah Green, Jacqui Jones & Denise Williams
UK Office for National Statistics

2. Aim of Presentation To provide an overview of the:
Drivers for the UK pilot of using the Standard Cost Model to measure respondent burden from statistical surveys
Issues highlighted by this work

3. Overview of Presentation
Measuring respondent burden from statistical surveys
Drivers
Measurement
Standard Cost Model for measuring respondent burden from statistical surveys
Pilot
Issues
The way forward

4. Drivers for Measuring Respondent Burden from Statistical Surveys
PMIs
Prime Ministers’ Instructions on the Control of Statistical Surveys
UK Code of Practice for Official Statistics
Report annually the estimated costs (for example, on businesses, service providers, or the public) of responding to statistical surveys …
Code of
Practice

5. Measurement of Respondent Burden from Statistical Surveys
PMIs
Simple method:
- used during survey control reviews
- 20% of survey sample
- postal self-completion questionnaire
Calculated by:
(No. of questionnaires (100% response)) *
(estimated completion time) *
(appropriate hourly rate) +
Up-rating for respondent re-contact
Recognised weakness in method:
“questionnaire did not adequately capture all activities involved in complying with the regulation”
Code of
Practice

6. SCM to Measure Total Respondent Burden in Europe
Measurement
PMIs
Code of
Practice
Statistics Netherlands Standard Cost Model
European approach to measure administrative
burdens for all government information requests
- Including survey participation
Standard Cost Model approach:
- Highlights impact of international legislation
- Transparent measurement
- Measures administrative burden
- Data collected in face-to-face interviews
chosen as the common approach by the European member states to measure administrative burdens for all government information requests, such as tax returns and value added tax, including survey participation.
EU initiative to enable comparisons across member states.
Standard Cost Model was developed by the Netherlands
Data collected in face-to-face interviews.
This involves small sample sizes.
In interviews they are asked to specify how much time and money they spend, performing each administrative activity that is required when fulfilling a given information obligation. Often distinguish between small and large businesses.
Collected data is standardised in terms of time and money spent performing each administrative activity.
The standardisation gives a representative figure of the costs incurred by a normally effective business within each segment.
SCM

7. SCM to Measure Total Respondent Burden in the UK
Measurement
PM’s
Code of
Practice
Administrative Burden Reduction Project
- Total UK administrative burden
- Includes survey participation
- Burden associated with statistical surveys
forms only a small part of the total
- Use SCM
SCM
ABRP

8. Drivers for Measuring Respondent Burden from Statistical Surveys
Measurement
PM’s
Need to change method
Need consistency
= Use SCM
Code of
Practice !
SCM
ABRP

9. Introduction of the SCM to Measure Respondent Burden from Statistical Surveys
ABRP used face-to-face interviews with very small sample sizes
Not feasible or robust enough for measuring compliance on a regular basis
A paper-based version of the SCM developed to measure costs of complying with statistical surveys
Implement quantitative rather than qualitative approach
The paper-based version was piloted in ONS

10. SCM to Measure Respondent Burden from Statistical Surveys
Questionnaire developed
10 questions
Breakdown of responding activity
Information available from ‘business-as-usual’
External costs
Survey irritants
For example: unclear definitions or available information did not match information requested
Standard Cost Model formula:
mean weighted cost per questionnaire + re-contact uplift *
number of questionnaires in survey sample *
survey frequency
Breakdown of responding activity
Time taken to become familiar with the questionnaire
The time of everyone who helped you to complete the questionnaire
Time spent extracting and preparing information from your systems
Any other time spent in relation to completing this survey
External costs
External accountant or bookkeeper
Survey irritants
A lot of this data is already available in ONS eg. complaints
Cost per questionnaire
[(internal cost + overhead – adjustment for BAU) + external cost]

11. Pilot highlighted large differences in estimates when compared to:
Pilot for Measuring Respondent Burden from Statistical Surveys Using the SCM
Pilot highlighted large differences in estimates when compared to:
Face-to-face interview methodology
Previous ONS methodology
Differences to previous ONS methodology are driven by:
Different information in models
Change in pay rates
Different uplifts for overheads
Adjustment to internal costs for business-as-usual

12. Issues with the SCM for Measuring Respondent Burden from Statistical Surveys
Figures surrounded by confidence intervals of the same magnitude as previous methodology
Method is hugely resource intensive
Respondents
Data producers
Difficulties in estimating accurately
Breakdown of activities
Associating costs with respondents

13. Questions Raised from Pilot Results
Statistical surveys form only small fraction of total administrative burden
- Does measurement need to be consistent?
Are such resource costs beneficial?
- The figures are purely estimates
Would a simpler method be:
Fit-for-purpose?
As robust?

14. Way Forward for Measuring Respondent Burden from Statistical Surveys
Recommendation to use a simpler model:
More proportionate
Is concerned with consistently measuring change over time rather than accurate levels
ONS has worked with producers of UK official statistics to develop such a model
Currently proposals are being finalised to take to the National Statistician for approval

15. Proposed methodology Still under discussion:
[{(# responses * median time) +
(# validated * median time)} * (hourly wage)] +
(# responses * median external costs)

16. Thank you