Workshop on Energy Statistics, China September 2012 Data Quality Assurance and Data Dissemination 1

2 Introduction  IRES Chapter 9: deals with Data Quality Assurance and Meta Data  Prerequisites of quality – institutional and organizational conditions, including:  Legal basis for compilation of data  Adequate data-sharing and coordination between partners  Assurance of confidentiality and security of data  Adequacy of resources – human, financial, technical  Efficient management of resources  Quality awareness

3 Overview of Quality Assurance  Under IRES, countries are encouraged to: Develop national quality assurance programs Document these programs Develop measures of data quality Make these available to users

4 What is a Quality Assurance Framework?  All planned activities to ensure data produced are adequate for their intended use  Includes: standards, practices, measures  Allows for: Comparisons with other countries Self-assessment Technical assistance Reviews by international and other users

5 Quality Assurance Framework (Statistics Canada)  Six Dimensions of Data Quality, based on ensuring “fitness for use” 1.Relevance 2.Accuracy 3.Timeliness 4.Accessibility 5.Interpretability 6.Coherence

6 Quality Measures and Indicators  Should cover all elements of the Quality Assurance Framework  Methodology should be well-established, credible  Must be easy to interpret and use  Should be practical – reasonable, not an over- burden  For Key Indicators, see IRES Table 9.2

7 Promoting Data Quality at Statistics Canada  Quality is a priority of senior management  Key quality indicators are tracked  Quality assurance reviews are conducted for major surveys  Data quality secretariat established  Questionnaire Design Resource Centre established  Quality assurance training delivered  Mandatory training provided to new employees

Quality assurance must be built into all stages of the survey process Survey Stages: 1.Identification of needs 2.Survey design 3.Building the survey 4.Data collection 5.Data processing 6.Analysis 7.Dissemination 8.Archiving 9.evaluation 8 Quality Assurance Framework

1. Identification of Needs Activities:  Define objectives, uses, users  Identify concepts, variables  Identify data sources and availability Quality Assurance  Consult with users and key stakeholders  Check sources for quality, comparability  Gather input and support from respondents  Establish quality targets 9

2. Survey Design Activities:  Design outputs  Define variables  Design data collection methodology  Determine frame & sampling strategy  Design production processes Quality Assurance  Consult users on outputs  Select & test frame  Design & test questionnaire  Test workflows  Develop checklists  Develop processes for error detection 10

3. Building the Survey Activities:  Build collection instrument  Build processing system  Design workflows  Finalize production systems Quality Assurance  Focus test questionnaire with respondents  Test systems for functionality  Test workflows  Document 11

4. Data Collection Activities:  Select sample  Set up collection  Run collection  Finalize collection Quality Assurance  Maintain frame  Train collection staff  Use technology with built in edits  Implement verification procedures  Monitor response rates, error rates, follow-up rates, reasons for non- response 12

5. Data Processing Activities:  Integrate data from all sources  Classify and code data  Review, validate and edit  Impute for missing or problematic data  Derive variables  Calculate weights Quality Assurance  Monitor edits  Implement follow-ups  Focus of most important respondents  Analyze and correct outliers 13

6. Data Analysis Activities:  Transform data to outputs  Validate data  Scrutinize and explain data  Apply disclosure controls  Finalize outputs Quality Assurance  Track all indicators  Calculate quality indicators  Compare data with previous cycles  Do coherence analysis 14

15 Sample Quality Indicators  From IRES Table 9.2, linked to QA Framework  Relevance: user feedback on satisfaction, utility of products and data  Accuracy: response rate, weighted response rate, number and size of revisions  Timeliness: time lag between reference period and release of data  Accessibility: number of hits, number of requests  Coherence: validation of data from other sources

7. Data Dissemination Activities:  Load data into output systems  Release products  Link to meta data  Provide user support Quality Assurance  Format, review, test outputs  Produce and follow dissemination checklists  Ensure all meta data is available  Provide contact names for user support 16

8. Archiving Activities:  Create rules and procedures for archiving and disposal  Maintain catalogues, formats, systems Quality Assurance  Periodic testing of processes and systems  Ensure meta data is attached 17

9. Evaluation Activities:  Conduct post mortem reviews to assess performance, identify issues Quality Assurance  Consult with clients about needs, concerns  Monitor key quality indicators  Periodic data quality reviews  Ongoing coherence analysis  Investments 18

19 Meta Data  Important for assessing “fitness for use” and ensuring interpretability  Required at every step of the survey process  Critical for enabling comparisons with other data  Should include results of data quality reviews  IRES table 9.3: generic set of meta data requirements

20 Dissemination  IRES Chapter 10 – Dissemination  Countries should have a Dissemination policy:  Scope of data available  Reference period and timetable  Data revision policy  Dissemination of meta data and quality reports  Data collected should not be withheld  Users must be aware of the availability  Data must be accessible – barriers must be reduced (e.g. format, cost, complexity)

21 Ensuring Confidentiality  Individual data must be kept confidential  Complicating factors: small numbers of respondents, dominance of a respondent  Methods of protecting confidentiality:  Aggregation  Suppression  Other (e.g. rounding)

22 Balancing Confidentiality & Disclosure  An ongoing challenge and trade-off (relevance)  Strategies to maximize utility: Raise the level of aggregation Data which are publically available are fully used Request permission to disseminate from respondents Employ passive confidentiality Publish confidentiality rules where data can be disseminated provided “excessive damage” is not caused to the respondent

23 Reference Periods and Timetable  Users must be aware of availability & release dates  Reporting should be based on calendar year (Gregorian)  Release targets recommended by IRES:  Monthly data within 2 months after reference period  Quarterly data within 3 months after reference period  Annual data within 15 months after end of reference period  Key indicators should be released even faster  Ongoing challenge: the trade-off between timeliness, quality

24 Revisions Policy  Countries should develop a revisions policy  Provisional data should be revised when new or more accurate data become available  Two main types of revisions:  Routine revisions (e.g. for late reporters, corrections)  Major revisions (e.g. changes in concepts, definitions, classifications, data sources, sample restratification)  All meta data should be provided to support users in understanding the revisions

25 Dissemination Formats  Formats should be chosen to meet user needs  Can be a combination of paper or electronic formats  Should always include meta data  Should minimize barriers to access (e.g. cost, technology, awareness, complexity)

26 Dissemination of Energy Data in Canada  Statistics Canada – primary data All data are announced in the Statistics Canada Daily Aggregate series available (free) on CANSIM Major publications:  Report on Energy Supply and Demand in Canada (Energy Balances)  Quarterly Energy Statistics Handbook Move from paper to electronic publications  Other major sources of energy information Natural Resources Canada – energy efficiency indicators Environment Canada – greenhouse gas emissions National Energy Board – energy reserves, forecasts, trade

27 Thank you! Andy Kohut, Director Manufacturing and Energy Division Statistics Canada Section B-8, 11 th Floor, Jean Talon Building Ottawa, Ontario Canada K1A 0T6 Telephone: