1. Marina Signore Head of Service “Audit for Quality Istat Assessing Quality through Auditing and Self-Assessment Signore M., Carbini R., D’Orazio M., Brancato G., Simeoni G.

2. Auditing and Self-assessment Presentation Describe the new procedure for assessing quality launched by Istat and how it fits into Istat Quality Policy The new procedure was launched in October 2009 and is now in the testing phase

3. Users Requirements Measurements External Environment DatQAM map Standards Preconditions Documentation Process Variables Tools Quality indicators User SurveysQuality Reports Self-Assessment Audit Labelling Certification (ISO 20252)

4. Auditing and Self-assessment Istat Quality Policy Harmonized with international quality frameworks European Statistics Code of Practice (since 2005) Eurostat Quality dimensions Centralized Structure for Quality Assurance (management and assessment)

5. Auditing and Self-assessment Preconditions Istat Framework for Quality Measurements Assessment Actions Quality Guidelines Documentation system Standard Quality Indicators Survey Assurance- Quality Control Systems Internal Audit Self-Assessment Assessment of Standard Quality Indicators Improvement Actions Communication on Quality to Users

6. Auditing and Self-assessment Quality Assessment Current surveys Analyses of SQI managed in SIDI In-depth analyses of survey process self-assessment internal auditing New Surveys Procedure for quality labeling developed and tested on 2 surveys adherence to few principles (relevance, questionnaire design and test, response burden reduction, quality assurance system) not yet launched

7. Auditing and Self-assessment Regular evaluation for most statistical processes but not a specific tool for promoting quality improvements Quality Assessment for current processes analyses of trends (response rates, timeliness,…) evaluation of Istat policies reports prepared by central service for Quality Analyses of SQI managed in SIDI

8. Auditing and Self-assessment Development (October 2009) of a strategy that combines audit and self-assessment way for highlighting strengths and weaknesses tool for improving weaknesses tool for collecting and disseminating good practices Quality Assessment for current processes Concrete way of improving quality

9. Auditing and Self-assessment Quality Assessment: main features The audits are carried out by internal auditors (face-to-face interview) The output of the assessment process consists in the identification of improvement actions, resources and timetable The improvement actions resulting from self-assessment are reviewed by a team of reviewers Additional resources or methodological support in order to implement the improvement actions User consultation on the improvement actions and for setting priorities Establishment of a Quality Committee in order to coordinate and sponsor the assessment activity

10. Auditing and Self-assessment Quality Committee Established in February 2010 Composition Chaired by the Head of Department of Statistical Production Co-chaired by the Head of Service “Audit for Quality” Members: Head of services

11. Auditing and Self-assessment Quality Committee Mandate To coordinate audit and self-assessment activity: Evaluation of final reports Approval of improvement actions Monitoring of improvement actions To develop procedures for monitoring the quality of statistical processes To coordinate and sponsor quality assessment in the ISS To promote the adoption of the CoP in the ISS

12. Auditing and Self-assessment The tools Quality guidelines Audit questionnaire Self-assessment questionnaire (electronic version) Operative manual

13. Auditing and Self-assessment Quality Guidelines Part I: Process quality Survey purposes Survey design Registers Data collection Data processing Data dissemination and documentation Reference guide to design, carry out and control a statistical process Part II: Product quality Components of Eurostat Quality definition

14. Auditing and Self-assessment Quality Guidelines Definition of the Principle to be followed Text describing what should be done in order to fulfill the principle Quite general descriptions and references to technical documents Light document less than 50 pages For each section

15. Auditing and Self-assessment Audit and Self-assessment questionnaires Part I: Process quality Part II: Product quality Part III: Process management and quality management Structure For each section The Principle to be followed is stated Some questions in order to evaluate the degree of compliance to the Principle Slight differences in the two questionnaires Input: DESAP checklist and OECD questionnaire

16. Auditing and Self-assessment The testing phase To test the tools and the procedure To evaluate the impact and sustainability on a voluntary basis Audit: 2 processes (2 teams of 3 experts each) Self-Assessment : 5 processes (3 teams of 3 reviewers each) Features

17. Auditing and Self-assessment Timetable of the new procedure Development of the procedure and tools (October - December 2009) Internal discussion on the tools (January 2010) Testing phase (February- June 2010)

18. Auditing and Self-assessment Final remarks Great opportunity for improving the quality of Istat statistical processes Analyses of the testing phase but first results seem to be very positive Further developments will concern processes based on administrative sources and secondary studies (e.g. National Accounts)