A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without Reducing Output Quality Gareth James Methodology Directorate UK Office for National Statistics

Outline of presentation Introduction response-chasing in ONS business surveys Understanding non-response effects, patterns and reasons Strategy for response-chasing scoring methods – current investigations and future strategies 2

Introduction Non-response … the failure of a business to respond in part or full to a survey. Effect on: –bias and standard error, –perception of output quality, –business behaviour Improve response rates by: –better questionnaire design, sample rotation rates, … –response-chasing - necessary, but expensive Quality improvements and efficiency targets –effective targeting needed 3

Current practice at ONS Use of % targets (mainly counts, occasionally other variables) Written reminders to all. Then targeted phone calls, … could lead to enforcement Businesses identified as ‘key’ (by survey area) chased intensively first After ‘keys’, principle to chase large-employment businesses next Methods differ between surveys 4

Current practice at ONS Areas for improvement: –Methods for ‘key’ businesses: make more consistent, transparent, scientific –Effective use of response-chasing tools –Team structure and knowledge (Area undergoing restructure) Efficiency initiatives –save resources: some changes already implemented –effects being monitored; evaluation needed 5

Efficiency initiatives – removal of second reminders 6

UNDERSTANDING NON-RESPONSE

Patterns of non-response Industrial sector - identified those with lower response rates (e.g. catering, hotels) High correlation between industry response rates at early and final results Size of business – larger businesses take longer to respond. Chasing strategy ensures responses are received later though 8

Intensive Follow-Up (IFU) exercise Dual aims: –to estimate non-response bias (work in progress – see final paper) –to establish reasons for non-response and (later) cost response- chasing Used the Monthly Inquiry into the Distribution and Services Sector (MIDSS): –dedicated team for the IFU –contacted c.600 non-responders per month in chosen industries –businesses to receive up to 5 phone calls –reason for initial non-response; nature of call; length of call 9

IFU results – returned data c.80% of all businesses selected for IFU returned questionnaire, but many businesses returned questionnaire just after deadline – no call needed! Only c.60% of those contacted returned questionnaire 10

IFU results – reasons for non-response Reason for initial non- response Number who gave a reason Returned data after IFU calls Still didn’t return data after IFU calls Forgot, missed date 66777%23% Too busy, too low priority 36167%33% Actively decided not to 6733%67% 11

BUILDING A RESPONSE-CHASING STRATEGY

Dealing with businesses that don’t respond Aim to make response-chasing more efficient Create a scoring system to prioritise/categorise non- responders Focus on reducing non-response bias 13

Estimation in ONS business surveys We impute/construct where there is non-response. Then estimate totals as where 14

Bias in ONS business surveys Total potential non-response bias (= total imputation error) given by We will concentrate on (i.e. the absolute error of imputation for each business) 15

Scoring - principles Reduce imputation error by attempting to predict (Large value means increased risk if business is imputed – therefore target these) May also wish to score to encourage good response behaviour from businesses – e.g. new-to-sample Need a system that is easy to use and justify. 16

Scoring methods (McKenzie) Calculate imputation error from previous returns; then rank into deciles: 0, 1, …, 9. (Smallest – Largest) New-to-sample or long-term non-responders = 10 Tested on MIDSS in ; implementation issues (Daoust) Calculate weighted contribution to estimates – categorise into 3 groups for follow-up New investigations with adapted methods 17

Current investigations in MIDSS Predict imputation error in monthly turnover (= y) –Various predictors available –Rank businesses then group –No imputation score? Use stratum average. Assess actual error against predicted. 18

Results (5 groups) Actual ScoreImputation error << 1 Percentage of within each priority score group 19

Results ActualWeighted prediction ScoreImputation error Previous imp. error << 12 Percentage of within each priority score group 19

Results ActualWeighted prediction ScoreImputation error Previous imp. error Register turnover << 124 Percentage of within each priority score group 19

Results ActualWeighted prediction Unweighted prediction ScoreImputation error Previous imp. error Register turnover Register employment << Percentage of within each priority score group 19

Conclusions Significant gains available in response chasing Future plans: Refinements to scores: –optimum predictor –individual adjustments (e.g. long-term non-responders) –overall or by separate industry groups? –multivariate surveys Dynamic updating of scores Live testing 20

References Daoust, P., (2006), 'Prioritizing Follow-Up of Non-respondents Using Scores for the Canadian Quarterly Survey of Financial Statistics for Enterprises', Conference of European Statisticians McKenzie, R., (2000) 'A Framework for Priority Contact of Non Respondents', Proceedings of the Second International Conference of Establishment Surveys 21