1 Quality Assurance In moving information from statistical programs into the hands of users we have to guard against the introduction of error. Quality assurance systems that minimize the possibility of errors are a necessary component of catalogue and data delivery systems.

2 Quality Assurance in Assembling and Delivering Data Products Activities related to the assembly and delivery of data products can be managed through individual programs at the NSO, or through a centralized delivery office Each activity has specific quality considerations that need to be addressed

3 Assembling the Product Assembling products for users might include changing formats of tabulations run during the processing stage. This introduces the opportunity for error. Data review and verification procedures are absolutely necessary to maintain accuracy NSO might choose to use a Table Retrieval System (TRS) in which all possible tabulations are programmed and retrievable by subject matter and geography. This facilitates the assembly of tailored products Where possible, it is best to use automated conversion of formats The programming step is one of the most demanding in How can ACCURACY be safeguarded when assembling a data product? Are the data products RELEVENT to the users?

4 Delivering the Product Product delivery is a complex process that might involve the final review of products, tabulation and packaging for the customer, and formal release by the Statistical Office. Tabulations from raw outputs must undergo their own series of quality control checks, electronic files prepared for release on the Internet must also meet standards for Internet products. Is delivery of the product TIMELY? How can users find the data, can users INTERPRET the data? Is the data COHERENT with international standards?

5 Quality Controls in Assembly and Delivery of Data Products Accuracy Timeliness Accessibility Interpretability Coherence

6 Accuracy One objective of the quality assurance system during the production and delivery stage is to prevent the introduction of error in this stage of the cycle. Assess accuracy by keeping documentation of errors. Often small errors reflect system wide problems Are error control systems in place? It is possible that errors are detected during the production and delivery stage, are errata policies in place?

7 Timeliness Assuring timely delivery of the product requires planning for the time required to get necessary approvals for release Assess timeliness by: Comparing scheduled delivery with actual delivery date

8 Accessibility The ease with which users can learn of the existence of data, locate it, and import it into their own working environment. This can be accomplished via: Agency-wide catalogue system Agency or program based delivery system Proper management of catalogue and delivery systems Constant improvement based on usage and user satisfaction feedback

9 Accessibility Other points of access to newly available statistical data: Official release mechanism (announcement, bulletin, etc.) Statistical agency web site A statistical agency has to ensure that the information needs of the general public continue to be met whether through the media, libraries, or the internet. To maximize accessibility, a statistical agency must be open to opportunities for partnership with external public and private organizations, but must also ensure: Identification as the source of data remains visible Where appropriate, encourages linkages back the original data sources held by the statistical agency

10 Pricing Policy A pricing policy needs to balance the desire to make certain basic information freely accessible in the public domain, while recovering the costs of providing specific products, more detailed information, and special requests.

11 Improving Accessibility: Data Delivery Service Center All calls for product purchases and inquiries managed through the Center Publishes product catalogues and updates Produces special order CDs and print on demand products Publishes Internet and bi-weekly newsletters Prices, sells, and ships products Maintains financial records

12 User Feedback Documenting and tracking user feedback can help improve delivery systems. This can be done by: Automated usage statistics for the various components of these systems Surveys of user satisfaction with particular products, services, or delivery systems Voluntary user feedback in the form of comments, suggestions, complaints, or praise

13 Interpretability The information needed to understand statistical data must be written in simple terms and includes: The concepts and classifications that underlie the data The methodology used to collect and compile the data Measures of data accuracy

14 Management of Interpretability Interpretability is perhaps the one dimension of quality where the statistical organization should aim to do more than the user is asking. To properly manage interpretability you need: A policy on informing users of the basic information they need to interpret data An integrated base of metadata that contains the information needed to describe each of the statistical agency’s data holdings Direct interpretation and commentary on the data by the statistical organization

15 Assessing Interpretability Measuring compliance with the statistical organization’s policy on basic information to be supplied with all data Documentation and tracking user feedback on the usefulness and adequacy of the metadata and analysis provided Correct use and interpretation of data by decision makers and media

16 Coherence In order for your data to be coherent between different data items, between different points in time, and internationally coherent, you need to: Develop and use standard frameworks, concepts, variables, and classifications for all the subject-matter topics that you measure Is the target of measurement consistent across programs? Is consistent terminology used across programs? Do the quantities being estimated bear known relationships to each other?

17 Coherence You also need to ensure that the process of measurement does not introduce inconsistency between data sources even when the quantities being measured are designed in a consistent way. The development and use of common frames, methodologies, and systems for data collection and processing for ongoing surveys as well as across different surveys contribute to this aim The use of a common business register across all business surveys The use of commonly formulated questions when the same variables are being collected in different surveys The use of common methodology and systems for the various processing steps of a survey

18 Coherence The third step in building coherence into your data involves the comparison and integration of data from different sources. For example: The integration of data in the national accounts Seasonal adjustment of data Confrontation of data from different sources as part of pre-release review or certification of data to be published Feedback from external users and analysts of data that point out coherence problems with current data Some incoherence issues only become apparent with the passage of time and may lead to historical revisions of data

19 Assessing Coherence Three broad measures can help you assess the coherence of your data - the documentation, monitoring and analysis of: The existence and degree of use of standard frameworks, variables and classification systems The existence and degree of use of common tools and methodologies for survey design and implementation The incidence and size of inconsistencies in published data

20 Planning Guidelines for Data Delivery Build in time for final review Recheck major findings of the reviews in product manufacturing, especially any unusual corrections or adjustments Check that public access tabulations are consistent with raw data files. Maintain sign-off procedures for all responsible parties on publications, press releases, and files posted on the Internet and in catalogues Provide an adequate description of survey methods and limitations along with the data Develop procedures for informing product users of any revisions or corrections after releasing the products Use only supported software to put files on the Internet Review discussions/ presentations/ reports for technical accuracy, including handling of sampling and non-sampling error