1. Measuring the quality of regional estimates from the ABS Jennie Davies and Daniel Ayoubkhani

2. Overview ABS User needs ABS estimation Methods Results Recommendations Outcomes

3. Annual Business Survey (ABS) ONS’s largest business survey Sample of ~73,000 UK businesses Covers agriculture (part), production, construction, distribution and service (part) sectors Variables (collected and derived) include: Turnover Purchases of goods and services Approximate Gross Value Added Net capital expenditure

4. Annual Business Survey (ABS) National publication (November and June) 4-digit SIC breakdown Regional publication (July) 12 UK regions, 2-digit SIC breakdown Special Analysis system Ad-hoc requests for lower level estimates (eg local authority, 3-digit SIC by employment sizeband etc) Standard errors and CVs provided with national estimates but not regional

5. User needs Large range of ABS users Central government, local government and devolved administrations, Eurostat, National Accounts, academia, consultancy firms, media, public Quality measures therefore important for sound decision making Lack of standard errors for regional estimates makes it difficult for users to assess accuracy

6. User needs Recent UKSA Assessment of ABS: “there is insufficient information about methods and quality” “there is no information about the resulting quality of the statistics and no caveats around their use” QIF project therefore undertaken to develop methodology for calculating standard errors of published regional ABS estimates Aim to provide users with information about the quality of regional estimates

7. Reporting unit data National estimates Modelled local unit data Regional estimates ABS estimation Regional apportionment model Ratio estimation Ratio estimation National standard errors GES Regional standard errors ??? Small area estimation for small domains 7

8. Standard error estimation (1) Assume that the apportioned values are “true” returns Single stage cluster sampling Use GES to calculate standard errors

9. Standard error estimation (2) The regional apportionment model parameters depend on the sample data so are variable Use bootstrapping to capture the use of the regional apportionment model

10. Bootstrapping Standard errors capture sampling variability ie how much estimates vary under different possible samples Bootstrapping re-samples from the original sample to create a new sample Carry out estimation on new sample Repeat lots of times Calculate standard error of the resulting estimates

11. Bootstrapping Fix the model parameters based on the full sample data quality assure method against GES Re-fit the regional apportionment model in each iteration include (possible) additional variance from the model

12. Results Compared the methods in terms of: Differences in standard errors Practical considerations Results for turnover presented Other variables produced similar results

13. GES vs bootstrap without re-fitting the model Bootstrapping without re-fitting the regional apportionment model in each iteration should be comparable with estimates from GES

15. Bootstrapping with and without re- fitting the model Comparing these to see if the regional apportionment model increases variances Compared to bootstrap rather than GES to remove additional differences seen before

17. Practical implications Bootstrapping took ~30 hours Relied on exact replication of the regional apportionment model Problem for derived variables such as aGVA

18. Recommendations Use GES to produce standard errors With caveats that the regional apportionment model is assumed to be fixed

19. Outcomes Report published on ONS website (Feb 2014) Methodology approved by ABS Survey Management Board Standard errors of regional estimates to be published for first time in July 2014 Another QIF bid submitted to investigate standard errors of small area ABS estimates Would allow for quality measures to be published alongside majority of Special Analysis requests

20. Summary ABS User needs ABS estimation Methods Results Recommendations Outcomes