Joseph Lukhwareni Statistics South Africa Reengineering projects focusing on metadata and the statistical cycle Statistics South Africa, South Africa 3-5 April 2006 Joseph Lukhwareni and Ashwell Jenneker

Page 2 Topics Background Technology solution Metadata Management Metadata definition Metadata Categories Metadata Store Metadata standards Metadata: Statistical Value Chain Conclusions

Page 3 Background(1) Vision: Preferred supplier of quality statistics “Improve quality of, and access to, authoritative and reliable statistical information” Challenges: – Data and metadata structured and stored according to different standards and procedures – Inadequate analytical capacity

Page 4 Background(2) Data Warehousing under discussion since 1997 December 2003: Establish Data Management and Information Delivery (DMID) project 2004: Stats SA approves business case for the development of a data warehouse Centrally managed store for data and metadata Enabling environment – Standardisation – Policy development

Page 5 Change in focus from data warehouse to solution that supports complete statistical value chain “ End-to-end Statistical Data Management Facility (ESDMF)”End-to-end Statistical Data Management Facility (ESDMF)” Metadata drives workflow and is generated by workflow (has governing and descriptive role): Controlled input environment: Data capture included Central data store Two separate analytical environments Controlled publication mechanism Access control Technology solution

Page 6 Technology solution(ESDMF)

Page 7 Metadata Management Defining metadata Developing/adopting relevant metadata standards and policies Determining appropriate technical storage environments Establishing metadata owners, maintainers and regulators Registration of metadata items Capturing/maintaining metadata Ensuring that data users have controlled and easy access to relevant metadata in a timely manner

Page 8 Metadata definition Data about data Data that defines and describes other data and processes Refers to the definitions, descriptions of procedures, system parameters, and operational results which characterise statistical programs. Metadata may be passive (descriptive) or active (prescriptive)

Page 9 Metadata categories Definitional metadata Operational metadata Systems metadata Dataset metadata Procedural/methodological metadata

Page 10 Metadata Store Manage the definition and maintenance of metadata needed throughout the statistical value chain Metadata store must be centrally available Provide a single source of approved metadata for metadata creators and users Allow new metadata to be defined to support the growth of the metadata store Include other systems such as the Classifications and Related Standards (CaRS)

Page 11 A metadata standard enables producers to describe datasets fully and coherently The standard facilitates data discovery, retrieval and use DMID team is in the process of investigating standards, for example: Formulation of data definitions e.g. ISO/IEC part 4 Naming and identification principles e.g. ISO/IEC part 5 Registration of metadata items e.g. ISO/IEC part 6 Identification of metadata elements – SANS 1878, South African spatial metadata standard Documentation basis needed for the different statistical production purposes – Statistics Sweden metadata system (SCBDOK) Metadata Standards

Page 12 Need proposal statement, user needs, budget plan, etc Design survey methodology, tabulation plan, questionnaire, etc Build design specification, test results, printed manuals, etc Collect enumeration, proxy response, response rates, etc Process processing methods, editing rates, imputation rate, etc. Analyse seasonal adjustment, sampling error, weighting, etc. Dissemination concepts and definitions, classifications, quality indicators,etc. Metadata: Statistical Value Chain

Page 13 Metadata management is not just a technical issue Continuous capturing and updating of metadata is necessary for effective metadata management Registration is an important element of metadata management Metadata should not be an afterthought Conclusions

Page 14 Thank you Mr T.J Lukhwareni & Mr A.C. Jenneker Data management and Information Delivery Project Statistics South Africa Private Bag X44, Pretoria 0001, South Africa