Implementation of GSBPM, DDI and SDMX reference metadata at Statistics Denmark UNECE workshop 5-7 May 2015 Mogens Grosen Nielsen

Agenda 1.History and strategy on quality and metadata 2.Definitions, models, solution and lessons learned on quality reporting 3.Towards more integrated metadata 4.Information models 5.Challenges

History  January 2010: Taskforce: integration variables, classifications, statistical concepts and quality  October 2011: Test DDI based on input from UNECE meeting (DDI provides integration)  January 2012: EU grant initiated. SDMX quality concepts and quality reporting, using DDI, SDMX and GSBPM. Colectica as tool  January 2015: Quality declarations for all statistics (300+) in Colectica  February 2015: Strategy on quality and metadata approved

Vision and aims 1.Statistical information must guide users in the “turbulent information- sea” 2.Metadata about content and quality must – help users in their knowledge processes – Help users find the right statistics – give users precise information about the products 3.International standards and standard software must enable: – Cost efficient solution – Gradual implementation with few ressources – Sustainable long term solution 4

 Business Process Management (End-to-End Processes)  Stepwise implementation  Code of practice and Qaulity Assurance Framework  Principles on metadata  Metadata must fulfill user needs  Metadata and metadata flow integrated into GSBPM  As much reuse as possible  Active use of metadata in IT-systems (auto-generation of code / metadata driven production) Principles 5

Metadata definition and how to communicate this term to statisticians A. SDMX definition as the short and easy-to- understand definition: Reference metadata: Conceptual metadata Methodological and processing metadata Quality metadata Structural metadata: Metadata act as identifiers and descriptors of the data B. Generic Statistical Information Model (GSIM) describes all information objects

”Classical” metadata using Data Documenation initiative (DDI), SDMX and Colectica - The Diamond Model Hvad betyder Quality declaration Variable/dataset Concept Variable database Klassifikationsdatabase Classifications Methods/ ”Survey” StatBankMethods papers Class database Concepts database Implemented in

SDMX Standard for Quality Single Integrated Metadata Structure (SIMS) 8  Content (population, concepts, reference time etc)  Statistical processing (sources and methods)  Information on 5 quality dimensions 1. Relevance 2. Accuracy and reliability 3. Timeliness and punctuality 4. Comparability 5. Accessibility and Clarity

SIMS and reporting formats Euro-SDMX Metadata Structure (ESMS) and ESS Standard for Quality Reports Structure (ESQRS) 9

GSBPM and work processes with focus on quality declarations 10 Needs Prepare user needs etc. Analyse : Fill in accuracy etc.

METADATA IN COLECTICA Enter Quality Information Publish at Dst.dk Quality eports to Eurostat Publish at the Intranet Existing metadata The solution

METADATA IN COLECTICA Enter Quality Information Publish at Dst.dk Quality eports to Eurostat Publish at the Intranet Existing metadata 300 surveys implemented January

Business perspective: Business Process Management (BPM) and metadata-driven approach GSIM compliant model in DDI and Colectica (concepts, variables and classification) Metadata management Harmonisation of statistical concepts Integration of metadata in publishing systems with focus on users access to data and metadata Towards more integration of metadata

Users User needs /orders General Environment: Political/legal context, Technology/standards Business Process Management ? Respondents/ registers etc Ressources:staff IT-systems etc

General Environment context: Political/legal, Technology/standards Respondents/ registers etc Ressources:staff IT-systems etc Management-, core- and support- processes Management processes Support processes: Quality, metadata, methods & IT Users User needs /orders

Business processes and metadata driven approach 16

Business processes and metadata driven approach 17 Reference metadata recorded: needs, purpose etc.

Business processes and metadata driven approach 18 Structural metadata: DDI on questionnaire, variables and cubes etc Reference metadata: concepts, population etc

Business processes and metadata driven approach 19 Metadata used to create survey system, databases and output systems etc

Business processes and metadata driven approach 20 Auto-generated survey system used

Reference metadata on quality etc Business processes and metadata driven approach 21

Structural and reference metadata used for dissemination Autogenerated code used in dissemination systems Business processes and metadata driven approach 22

Business processes and metadata driven approach 23 All metadata used for evaluation

Levels and what we are doing at Stat DK Models: conceptual, logical and physical Model / levelWhat are we doing at Stat DK ConceptualSelection of variable, concept etc from GSIM (the concept corner) LogicalGSIM compliant DDI model (3.2) PhysicalGSIM compliant DDI model implemented in Colectica

Need for improving content in order to make the quality declarations more uniform and to make the quality declarations more compliant with common guidelines Need for analysis of reuse of quality-concepts in order to report to Eurostat, publish at national web-site and to other international organsations. Need for more analysis and improved dissemination at Need for improved focus on change management and communication Clear organisational roles and a creative work- environment are needed in order to benefit from international standards Lessons learned

Big potential in international cooperation on metadata including cooperation on the use of Colectica Areas for cooperation: -Use of common information-model -Sharing of models and sharing of code for input, processing and output systems -Versioning / historical information in DDI 3.2 and in Colectica -Common metadata and centralized metadata management -How to organize metadata in DDI 3.2 and in Colectica (concepts, variables etc) -How to handle changes e.g. change in a common codelist on civil status -Use of group, schemas etc in DDI 3.2 and in Colectica Opportunities

Thanks for your attention Remember: DDI conference in Copenhagen 2-3 December 2015

The End!