1. Continuous improvement of EU-SILC quality: standard error estimation and new quality reporting system Emilio Di Meglio and Emanuela Di Falco (EUROSTAT) Q2014 Conference Vienna1

2. Why variance estimation? Requested by regulation Quality report Compliance Requested by users Policy relevance of indicators Requested by researchers Q2014 Conference Vienna 2

3. Current legal requirements According to Reg.1982/2003, the X and L (initial sample) data are to be based on a nationally representative probability sample of the population residing in private households. Representative probability samples shall be achieved both for households and for individual persons in the target population. The sampling frame and methods of sample selection should ensure that every individual and household in the target population is assigned a known and non-zero probability of selection. Reg. 1177/2003 defines the minimum effective sample sizes to be achieved. Q2014 Conference Vienna 3

4. Main challenges for EU SILC Difficulty to find the « best » possible method for variance estimation at Eurostat level –Different designs (flexibility) –Missing information –Debate on methods ongoing Differentiate the needs: accuracy estimates for policy usage and accuracy estimates for researchers. Q2014 Conference Vienna 4

5. Sampling design by country (2012) Q2014 Conference Vienna 5

6. Our objective Resampling taking into account all the possible elements coming from 32 countries would be extremely computationally and resource intensive Variance estimation methods balancing between scientific accuracy and administrative considerations (time, cost, simplicity) are the only viable solution Aim: to quickly provide to users and policy makers standard errors for the SILC-based indicators, particularly the AROPE (At-Risk-Of-Poverty or social Exclusion), its components and its main breakdowns. Q2014 Conference Vienna 6

7. The method (synthesis) Linearization is a technique based on the use of linear approximation to reduce non-linear statistics to a linear form, justified by asymptotic properties of the estimator (Särndal et al, 1992 ; Deville, 1999 ; Wolter, 2006 ; Osier, 2009) The "ultimate cluster" approach (Särndal et al, 1992) is a simplification consisting in calculating the variance taking into account only variation among Primary Sampling Unit (PSU) totals This method requires first stage sampling fractions to be small which is nearly always the case. This method allows a great flexibility and simplifies the calculations of variances. It can also be generalized to calculate variance of the differences of one year to another (Berger, 2004, 2010 ). Applicable with the main statistical packages (SAS, R, STATA) Q2014 Conference Vienna 7

8. Results on AROPE For 6 countries 95% Confidence Interval for AROPE equal or smaller that ±1.0% (CZ, IT, SI, DE, FI, NO) For 9 countries 95% Confidence Interval for AROPE between ± 1% and ±1.5% (ES, PL, UK, EE, AT, SK, CH, SE, IS) For 8 countries 95% Confidence Interval for AROPE between ±1.5% and ±2% (BE, DK, HR, HU, NL, PT, CY, MT) For 6 countries 95% Confidence Interval for AROPE larger than ±2% (BG, EL, IE, RO, LT, LV) Complete results in EU-SILC quality report Q2014 Conference Vienna 8

9. Measurement of net changes To measure the significance of the evolution of social indicators Example: When the At-risk-of-poverty or social exclusion rate for Poland goes from 27.2% in 2011 to 26.7% in 2012, are we able to say that this change is significant? Exercise already done for AROPE and other main EU-SILC indicators Q2014 Conference Vienna 9

10. Output Q2014 Conference Vienna 10 Country AROPE (2011) % AROPE (2012) % Difference 2012 – 2011 (% points) Standard error (% points) Significanc e of change HU31.032.41.50.7 Y MT21.422.20.80.4 Y NL 15.715.0-0.80.2 Y PL27.226.7-0.50.3 N

11. EU-SILC Quality reports(Reg. No 1777/2003) At national level, Member States have to produce: An Intermediate QR (by the end of the year N+1) Based on cross-sectional data of year N A Final QR (by the end of the year N+2) Based on longitudinal and cross-sectional data year N At European level, EUROSTAT has to produce : EU Comparative Intermediate QR (by June of the year N+2) Based on the national Intermediate QRs EU Comparative Final QR (by June of the year N+3) Based on the national Final QRs Q2014 Conference Vienna 11

12. Quality reporting Revision process New template (ESQRS) Revision of the Contents Introduction of annexes and questionnaire ESS Metadata Handler (old NRME) Q2014 Conference Vienna 12

13. EU SILC key quality dimensions Accuracy Comparability Coherence  National ESQRS Cost and burden Statistical processing Timeliness and punctuality Relevance  EU ESQRS Q2014 Conference Vienna 13

14. Availability of quality metadata Quality reports Questionnaires Methodological papers Further action: integrate more information in a wiki platform Q2014 Conference Vienna 14

15. Conclusion and future plans The variance estimation methodology is of relatively simple application It can be considered as a good compromise between scientific soundness and feasibility under current constraints. The next steps consist in still improving these calculations by asking Member States to provide the necessary information where missing. Dissemination of further information to users. Better disseminate quality reports Q2014 Conference Vienna 15