1. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 1 Evaluating the Total Non-Response Errors in the European-Union Survey on Income and Living Conditions (EU-SILC): A Territorial Quality Profile Claudio QUINTANO Rosalia CASTELLANO Gennaro PUNZO European Conference on Quality in Official Statistics 2008 Rome, Italy – July 8-11, 2008

2. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 2 AIM OF THE WORK Evaluating the ACCURACY of the Italian Section of EU-SILC data with focus on the NON-SAMPLING ERRORS deriving from the several components of TOTAL NON-RESPONSE STEPS  A SET OF AD HOC BASIC QUALITY INDICATORS (HIERARCHICAL FRAMEWORK)  CLASSES OF SYNTHETIC QUALITY INDICATORS  BROKEN DOWN BY DIFFERENT TERRITORIAL LEVELS  ONE-WAY RANDOM EFFECTS ANOVA MODEL (Singer, 1998)  RANKING ANALYSIS TERRITORIAL QUALITY PROFILE

3. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 3 EU-SILC: BACKGROUND AND MAIN FEATURES OF THE ITALIAN SEGMENT PRIMARY SAMPLING UNITS (PSU) SECONDARY SAMPLING UNITS(SSU)HOUSEHOLDS A two-stages sampling design… …drawn from municipality-registers by a systematic sampling MUNICIPALITIES …STRATIFIED by demographic size, inside each NUTS2 region 288 STRATA Self-Representing Non Self-Representing First order Second order

4. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 4 WAVESLONGITUDINAL SUB-SAMPLES 2004 2005 2006 2007 2008 2009 INTEGRATED DESIGN BASED ON FOUR YEARLY ROTATIONAL PANELS A (4) B (3) C (2) D1D1 B (4) D2D2 C (3) E1E1 C (4) D3D3 E2E2 F1F1 D4D4 E3E3 F2F2 G1G1 E4E4 F3F3 G2G2 H1H1 F4F4 G3G3 H2H2 I1I1 CROSS-SECTIONAL WHOLE PANEL

5. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 5 Source: EU-SILC Quality Reports (2004 and 2005) WavesABCDETotal 20047.9567.9937.9988.045–31.992 2005–6.1796.1856.4148.00826.786 EU-SILC SAMPLING DESIGN... Table 1 – PSU stratification... Table 2 – Longitudinal replications in terms of SSU... STRATAPSUSelf-Representing Non Self-Representing of First order Non Self-Representing of Second order 110 30 30148110 60 60592 Total288762 Source: Author’s elaborations on Istat data

6. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 6 SAMPLINGUNITS A THEORETICAL FRAMEWORK IN-SCOPE RESPONDENTSNONRESPONDENTS NONACHIEVEMENTNONCONTACTEDREFUSED UNABLE TO RESPONSE INCORRECTADDRESS NOT LOCATED ADDRESS UNABLE TO ACCESS OUT-OF-SCOPE TEMPORARILYNOT-AT-HOME

7. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 7 EU-SILC BASIC QUALITY INDICATORS IN-SCOPE RATE NON-ACHIEVEMENT RATE REFUSAL RATE UNABLE-TO- RESPONSE RATE INCORRECT ADDRESS RATE NOT-LOCATED ADDRESS RATE UNABLE-TO- ACCESS RATE OUT-OF-SCOPE RATE TEMPORARILY NOT-AT-HOME RATE

8. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 8 A THEORETICAL FRAMEWORK IN-SCOPE RATE NON-ACHIEVEMENT RATE REFUSAL RATE UNABLE-TO- RESPONSE RATE INCORRECT ADDRESS RATE NOT-LOCATED ADDRESS RATE UNABLE-TO- ACCESS RATE OUT-OF-SCOPE RATE TEMPORARILY NOT-AT-HOME RATE

9. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 9 THE PROCESS OF CONTACTING THE EU-SILC SAMPLE HOUSEHOLDS... cross-sectional sample... 20042005 In-Scope Rate Out-of-Scope Rate 99.36 0.64 0.6498.85 1.15 1.15 Source: Author’s elaborations on Istat data Table 3 – Eligibility Rates Table 4 – Non-Contact Rates 20042005 Not-Located Address Rate Unable-to-Access Rate Incorrect Address Rate 0.850.244.060.530.322.30 Source: Author’s elaborations on Istat data 0.53% 0.32% 0.30% DECEASED INSTITUTIONALIZED TRANSFERRED Table 5 – Frame Error Rates 20042005 Frame Error Rate 4.673.42 Source: Author’s elaborations on Istat data -0.32 -1.76 -1.25

10. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 10... AND THEIR ACTUAL INVOLVEMENT EU-SILC cross-sectional sample... 20042005 Refusal Rate Unable-to-Response Rate Non-Achievement Rate Temporarily Not-at-Home Rate Temporarily Not-at-Home Rate8.621.099.782.737.740.915.443.71 Table 6 – Non-Participation Rates Source: Author’s elaborations on Istat data -4.34 NON-ACHIEVEMENT RATE also includes a residual share of non-participant households, though successfully contacted, whose reasons are not specified -0.88 +0.98

11. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 11 COOPERATIONRATE NON COOPERATION RATE NON ACHIEVEMENT RATE REFUSAL RATE UNABLE TO RESPONSE RATE... A SYNTHESIS OF THE BASIC QUALITY INDICATORS: A HIERACHICAL APPROACH COOPERATION RATE

12. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 12 NON CONTACT RATECONTACTRATE INCORRECT ADDRESS RATE NOT LOCATED ADDRESS RATE UNABLE TO ACCESS RATE... A SYNTHESIS OF THE BASIC QUALITY INDICATORS: A HIERACHICAL APPROACH CONTACT RATE

13. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 13 RESPONSE RATE CONTACT RATE COOPERATION RATE COMPLETION RATE... A SYNTHESIS OF THE BASIC QUALITY INDICATORS: A HIERACHICAL APPROACH x

14. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 14 IN-SCOPE RESPONDENTSNONRESPONDENTS NONCONTACTED CONTACTED INCORRECTADDRESS NOT LOCATED ADDRESS UNABLE TO ACCESS OUT-OF-SCOPE COOPERATING NONCOOPERATING NON ACHIEVEMENT REFUSED UNABLE TO RESPONSE SAMPLINGUNITS

15. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 15 EVALUATING THE ACTUAL SURVEY PARTICIPATION OVER TIME EU-SILC cross-sectional and longitudinal samples... wave 2004 ABCDTotal Cooperation Rate Contact Rate Response Rate 79.3095.2175.5079.8095.2375.9980.1794.3175.6182.7194.6778.3180.5094.8576.35 wave 2005 BCDETotal Cooperation Rate Contact Rate Response Rate 87.0097.4484.7787.9797.3585.6386.8997.2284.4782.6195.7179.0685.9196.8583.21 Table 7 – Synthetic Participation Rates Source: Author’s elaborations on Istat data COMPLETION RATE 75.86% 82.25% wave 2004 wave 2005 +6.86

16. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 16 DOES TOTAL NON-PARTICIPATION DIFFER ACROSS NATIONAL TERRITORY? Fig. 3 – Non–Response RatesFig. 4 – Non–Completion Rates Fig. 2 – Non–Contact RatesFig. 1 – Non–Cooperation Rates wave 2004 wave 2005

17. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 17 DOES TOTAL NON-PARTICIPATION DIFFER ACROSS NATIONAL TERRITORY? Fig. 5 – Frame Error RatesFig. 6 – Refusal Rates wave 2004 wave 2005 In order to investigate in-depth the territorial perspective, its role and significance, as well as the effects on the total non-response process... ONE-WAY RANDOM EFFECTS ANOVA MODEL (NULL MODEL)

18. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 18 with i = 1, 2,...,m and j = 1, 2,..., 21 municipalities ONE-WAY RANDOM EFFECTS ANOVA MODEL (NULL MODEL) NUTS2 regions fixedeffect randomeffects of belonging to the j th NUTS2 region difference between the i th municipality and the mean within the j th region measuring the effect representing the intraclasscorrelation degree of homogeneity within NUTS2 regions

19. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 19 Outcome Rate (2004) Non-ContactNon-CooperationNon-Response Fixed Effect: Intercept Random Effect: Region variance Residual variance 0.0373 (.00340) 0.0006 (.00009) 0.0035 (.00018) 0.1575 (.00812) 0.0004 (.00039)* 0.0004 (.00039)* 0.0463 (.00240) 0.1882 (.00880) 0.0062 (.00053) 0.0474 (.00247) Outcome Rate (2005) Non-ContactNon-CooperationNon-Response Fixed Effect: Intercept Random Effect: Region variance Residual variance 0.0207 (.00322) 0.0007 (.00008) 0.0027 (.00013) 0. 1448 (.01308) 0.0085 (.00107) 0.0558 (.00265) 0.1627 (.01366) 0.0098 (.00122) 0.0568 (.00270) Table 8 – One-way random effects ANOVA model: main results Source: Author’s elaborations on Istat data REGIONEFFECT

20. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 20 Outcome Rate (2004) Frame Error RefusalNon-Completion Fixed Effect: Intercept Random Effect: Region variance Residual variance 0.0348 (.00317) 0.0005 (.00008) 0.0031 (.00016) 0.0625 (.00463) 0.0011 (.00013) 0.0011 (.00013) 0.0061 (.00032) 0.1925 (.00900) 0.0071 (.00056) 0.0470 (.00245) Outcome Rate (2005) Frame Error RefusalNon-Completion Fixed Effect: Intercept Random Effect: Region variance Residual variance 0.0247 (.00315) 0.0006 (.00008) 0.0024 (.00011) 0. 0740 (.00668) 0.0012 (.00027) 0.0243 (.00116) 0.1703 (.01373) 0.0113 (.00123) 0.0564 (.00269) Source: Author’s elaborations on Istat data Table 9 – One-way random effects ANOVA model: main results REGIONEFFECT

21. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 21 In order to inspect how far is from each Italian NUTS2 region or geographical macro area to the entire national territory in terms of dissimilarities of data production process quality... SYNTHETIC QUALITY INDICATORS EXPLORING THE MAIN DIFFERENCES IN EU-SILC PARTICIPATION ACROSS NATIONAL TERRITORY SPATIALINDICES RANKING ANALYSIS BY NUTS2 REGIONS BY MACRO AREAS COGRADUATION KENDALL’S RANK CORRELATION

22. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 22 RANKING ANALYSIS MacroAreaNonContactMacroAreaNonCooperat.MacroAreaNonResponse CenterNorth-WestNorth-EastIslandsSouth124.78100.13 97.35 97.35 81.89 81.89 74.86 74.86North-WestCenterNorth-EastIslandsSouth114.63110.24 89.22 89.22 85.96 85.96 83.93 83.93CenterNorth-WestNorth-EastIslandsSouth112.25111.46 91.09 91.09 85.73 85.73 82.85 82.85 MacroAreaNonContactMacroAreaNonCooperat.MacroAreaNonResponse CenterNorth-WestIslandsNorth-EastSouth146.96111.90 95.18 95.18 75.57 75.57 54.12 54.12North-WestCenterNorth-EastIslandsSouth125.36110.47102.53 71.48 71.48 59.10 59.10North-WestCenterNorth-EastIslandsSouth122.47115.90 98.17 98.17 75.98 75.98 58.88 58.88 Table 10 – Ranking of Italian geographical macro areas (Italy=100) - waves 2004 and 2005 waves 2004 and 2005 Source: Author’s elaborations on Istat data Kendall’s rank correlation

23. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 23  NON-CONTACT, OVERALL NON-RESPONSE AND, OBVIOUSLY, NON-COMPLETION RATES SIGNIFICANTLY DIFFER BY ITALIAN NUTS2 REGIONS ON BOTH THE WAVES (2004 AND 2005) CONCLUDING REMARKS...  DIFFERENTIALS ACROSS NUTS2 REGIONS ALSO CONCERN THE FRAME ERRORS AND REFUSALS AS THE TWO CRUCIAL SOURCES OF NON-CONTACTS AND NON-COOPERATION, RESPECTIVELY  NON-COOPERATION RATES SEEM TO BE STATISTICALLY DIFFERENT ACROSS NATIONAL TERRITORY ONLY IN 2005  A DOWNWARD TREND OVER WAVE IS REVEALED FOR THE OUTCOME QUALITY INDICATORS CONSIDERED ALSO DUE TO THE PANEL FRAMEWORK OF THE EU-SILC SURVEY  STAYING PUT OF THE ITALIAN REGIONS IN THE SAME (OR IN A SIMILAR) RANKING FOR THE NON-CONTACT AND NON- COOPERATION RATES AS WELL AS FOR THE FRAME ERROR AND REFUSAL RATES

24. 11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 24 … AND FURTHER DEVELOPMENTS  URBANICITY  POPULATION DENSITY  CRIME RATES  LACK OF SOCIAL COHESION  AND SO ON... At a sub-national level, a variety of contextual factors may influence survey participation, such as... REALLY... We deliberately neglected these aspects but we intend to examine them closely afterwards SOCIO-ECONOMIC ENVIRONMENT INFLUENCES ON SURVEY PARTICIPATION (Groves and Couper, 1998)