IAOS 2014 Da Nang: An agile approach to question testing and satisfying a new data requirement Pete Brodie ONS, UK

Overview Background Drivers Questionnaire requirements Delivery dilemma Solution and outcome New output requirement “Agile” methodology Outcome Results Next steps Conclusion

Background ONS delivers a very broad spectrum of statistics Financial pressures on all NSIs Difficult to react quickly to new requirements Scarcity of specialist skills

Drivers Changes due to ESA 10 and BPM6 - updated regulations for producing National Accounts Changes in the labour market – the way people are employed

Questionnaire requirements Early in 2014 legislative changes required substantial changes to most ONS business statistics First tranch of 28 Finance Surveys Total redesign of questionnaires Additional questions Covering new sectors of the economy

Delivery dilemma Standard approach for DCM uses multiple waves of cognitive interviews with responders Transcription of interviews Analysis using thematic framework Detailed report of issues and recommendations Time and available resource – impossible!

Solution and outcome I Brand new approach needed trialled for two surveys Condensed usual waves of testing into one week “stints” Three stages to each stint MondayTuesdayWednesdayThursdayFriday Stage 1Stage 2Stage 3 FieldworkCollaborative meetings – DCM, Clients, Stakeholders Changes agreed

Solution and outcome II Traffic light document

Solution and outcome III Extensive changes implemented for two surveys – agreed between all stakeholders Real improvements made All questions rated “green” at end Client feedback very positive Re-use methodology with some tweaks

New output requirements Autumn 2013 demand for measurement of the number of employees on “Zero Hours” Contracts Definition unclear No information from businesses Promise to produce estimates early in 2014 Conventional approach of qualitative and quantitative research likely to take around three years

“Agile” methodology I Staged approach – pilot followed by live survey Combined qualitative and quantitative work Novel (for ONS) approach of telephone survey of businesses Used flexible “Statistical Computing” for all quantitative production

“Agile” methodology II Pilot survey of around 1000 businesses (30 strata) Supplemented frame information with contact numbers Collaborative working between different parts of the office In field for four weeks achieved 60% response Qualitative interviews with selected responders Bespoke code for editing, outlier detection and estimation

“Agile” methodology III Quantitative estimates from pilot used to optimise survey design for main survey – aim for 10% cv at top level Questionnaires updated using qualitative research Main survey of 5000 businesses included all pilot sample and 4000 new members In field for six weeks achieved over 60% response

Outcome Published results on 30 th April – first business survey estimate of the number of live contracts of 1.4 million (c.v. 8.5%) Greater understanding of term -“contracts with no guaranteed minimum hours” Comparison with estimates from Labour Force Survey Published some more detailed estimates Some data available for longitudinal analysis

Results I Proportion of business using ZHC by size

Results II Proportion by broad industry

Next Steps The LFS questionnaire was updated in the light of findings and to answer detailed demographic questions Follow up business survey of 5000 now in the field Considering longitudinal design Further embedded qualitative work Considering longer term strategy

Conclusions These two examples illustrate what is possible when using an agile approach. Too often constrained by artificial boundaries. Opportunities for development of staff and methods through collaborative working. Necessity is the mother of invention!

Thank you! Any questions ?