1 C. ARRIBAS, D. LORCA, A. SALINERO & A. COLMENERO Measuring statistical quality at the Spanish National Statistical Institute

2 SUMMARY 1.-Global Quality assessment of all surveys at NSI 2.-Implementation of systematic quality management 3.-Identification & calculation of SQIs of outputs & processes 4.-Production of SQRs of outputs(user´s oriented) and processes (internal uses) 5.-Conclusions

3 1. Global Q assesment of all Surveys 1.-Calculation of synthetic Indicator of Global Q of the survey 2.-Calculation of synthetic Indicator of Global Q for survey output and process 3.-Calculation of synthetic Indicator of Q criteria of outputs or processes phases 4.-Calculation of SQIs of outputs and processes

4 2. Implementation of systematic quality management I.-Objectives: monitoring in a continuous basis the quality assessment of the most important statistical surveys  Review the statistical processes and outputs  Identify and Evaluate Standard Quality Indicators  Produce Standard Quality Reports

5 II.- Procedures followed:  Self-assessment by Survey Managers (using DESAP, european checklist)  Audit teams for checking: Information provided by Survey managers Evaluation results of SQIs of processes & outputs Content and presentation of SQRs of processes & outputs in order to harmonize the format 2. Implementation of systematic quality management

6 3. Identification & Calculation of SQI’s of output & processes I.- Identification –SQIs of outputs Selected set of Indicators, defined by Eurostat and based on Q criteria –SQIs of processes Selected set of Indicators, defined by NSI and based on processes phases

7 –Relevance –Accuracy –Timeliness & Punctuality –Accessibility & Clarity –Comparability –Coherence EUROSTAT Q criteria:

8 Quality component Indicator RelevanceR1. User satisfaction index. R2. Rate of available statistics. AccuracyA1. Coefficient of variation. A2. Unit response rate (un-weighted/ weighted). A3. Item response rate (un-weighted/ weighted). A4. Imputation rate and ratio. A5. Over-coverage and misclassification rates. A6. Geographical under-coverage ratio. A7. Average size of revisions. Timeliness and Punctuality T1. Punctuality of time schedule of effective publication. T2. Time lag between the end of reference period and the date of first results. T3. Time lag between the end of reference period and the date of the final results. Accessibility and clarity AC1. Number of publications disseminated and/or sold. AC2. Number of accesses to databases. AC3. Rate of completeness of metadata information for released statistics. ComparabilityC1. Length of comparable time-series. C2. Number of comparable time-series. C3. Rate of differences in concepts and measurement from European norms. C4. Asymmetries for statistics mirror flows. CoherenceCH1. Rate of statistics that satisfies the requirements for the main secondary use. TABLE 1.- STANDARD QUALITY INDICATORS FOR STATISTICAL OUTPUTS

9 –Sampling Frame –Coverage –Data Collection –Response Burden –Editing & Imputation –Sample & Estimation –Documentation NSI set of selected process phases

10 Processes Phases Indicator Sampling Frame SM1. Time gap between the date reference period and the last update of the sampling frame CoverageCV1. Rate of overcoverage CV2. Rate of undercoverage CV3. Rate of misclassification CV4. Assessment of the coverage Data Collection DC1. Technique used DC2. Training of interviewers DC3. Unit non response rate DC4. Item non response rate Response Burden RB1. Sample rotation scheme RB2. Time for completion questionnaire RB3. Questionnaire design and interview Editing and Imputation EI1. Editing procedures EI2. Imputation procedures Sample and Estimation SE1. Target and achieved sample size SE2. Sampling errors SE3. Efficiency of the survey design SE4. Calibration or re-weighted DocumentationDO1. Documentation on the process DO2. Documentation on courses / manuals TABLE 2.- STANDARD QUALITY INDICATORS FOR STATISTICAL PROCESSES

11 II.- Calculation 1.-DESAP checklist, was filled for all surveys by Survey managers and the answers were audited 2.-Link each SQI to one or several DESAP questions 3.-Associate the answers to a measurement scale: –for each SQI weighted average of selected questions is obtained –for each Q criteria or process phase weighted average of selected indicators is obtained –Global Q for output or statistical process is obtained as the weighted average values of Q criteria or process phases 3. Identification & Calculation of SQI’s of output & processes

12 II.- Calculation (cont) 4.-Measurement scale: represent Q degrees for outputs and processes 5.-Transformation rules: transform original answers into values or scores of the measurement scale –the scale is the same to all questions of the Indicator –the transformation rule of questions into scale scores must be coherent, so they can be comparable 3. Identification & Calculation of SQI’s of output & processes

13 4. Production of SQRs of outputs & processes Grouped 33 main surveys: I. Short-term statistics and indicators II.- Structural Surveys III.-Household Surveys Survey managers produced: SQRs of outputs: user´s oriented I.-Description of the Survey II.-Q Evaluation (based on Q criteria) SQRs of processes: internal uses for top management and improvement actions I.-Description of the Survey II.-Q Evaluation (based on process phases)

14 5. Conclusions  Implementation of systematic Q management using the same tools & methods to assess Q of surveys  Survey analysis was obtained: –Numerical values of SQIs of outputs & processes: Permit compare Q between different surveys Permit compare Q changes over time Calculation system permits introduce Q improvements and obtain the new numerical value of SQI automatically –The SQRs approach the NSI to users enable users a better understanding and use of survey data

15  Possibility to repeat the exercise (review the whole process) in some years time  Selection of particular domains for improvements: –Standardization of the documentation of the surveys –Improve and introduce generalized systems for data editing and imputation –Improve dissemination of survey data –Identify best practices and disseminate them among staff  Concluding remark: Statisticians should quantify social and economic features of a society.  Our first duty should be to try to measure, in numerical values, the Q of our own work. 5. Conclusions