1. SDMX IT Tools Introduction
Jean-Francois LEBLANC
Christian SEBASTIAN
November 2015
Eurostat Unit B3 – IT and standards for data and metadata exchange

2. Table of contents Where are we? Standardization
Why do we need a model?
GSBPM Generic Statistical Business Process Model
Phases
Key features
Other uses
Standards – Relations
GSIM Generic Statistical Information Model

3. Table of contents SDMX & DDI SDMX Summary Why? Benefits Costs
Opportunities
Impacts
From 1.0 to 2.1
The SDMX components
SDMX in practice
Summary

4. Where are we?
Dramatic changes in the environment of official statistics producers (e.g. data deluge)
Modernization of statistical information system seen as a question of survival for the sector of official statistics
Standardization viewed as a key enabler for modernization
"Standards-based” industrialization of statistical production

5. 2. Standardization Why is it necessary? What does it imply?
Harmonization
Reusability and interoperability
Shared solutions across statistical institutes
What does it imply?
Common processes
Common tools
Common methodologies
Why is it necessary?
Harmonization of statistical data (leading to more comparable data)
Reusable, interoperable data for users
Economies of scale across statistical institutes internationally - shared solutions
Good vendor support for the industry

6. 2. Standardization Industry Standards Other major standards
GSBPM - Generic Statistical Business Process Model
GSIM - Generic Statistical Information Model
SDMX - Statistical Data and Metadata eXchange
DDI - Data Documentation Initiative
Other major standards
RDF - Resource Description Framework
LOD - Linked Open Data
JSON - JavaScript Object Notation
XBRL - eXtensible Business Reporting Language
GSBPM
GSIM
SDMX
DDI
SDMX
Preferred standard for exchange and sharing of data and metadata in the global statistical community (UNSC, 2008) – Widely used in the ESS for aggregated data
DDI
Standard for the documentation of data, initially focused on archiving micro-data in the area of social sciences – widely used in national data archives – extended to support the full life-cycle of data
RDF
W3C standard for web-based discovery, dissemination, and linking – an alternative to XML
Allows for after-the-fact linking of any types of resources on the Web, e.g. linking press releases and speeches with relevant data from a statistical organization
Powerful querying language for distributed searches on the web
Very popular with “open data” and “open government” initiatives
JSON
Web-developer-friendly alternative to XML. JSON version of SDMX
XBRL
Standard for reporting accounting information and banking supervision data
XML-based standard
No formal model
Communities standardize the taxonomies to support their needs
Mapping to other models requires an understanding of the implied model of the community
Good tools are required to hide this complexity

7. 3. Why do we need a model?
To define and describe statistical processes in a coherent way
To standardize process terminology
To compare and benchmark processes within and between organisations
To identify synergies between processes
To inform decisions on systems architectures and organisation of resources

8. 4. GSBPM Generic Statistical Business Process Model
Applicable to all activities undertaken by producers of official statistics -> data outputs
Used by National and international statistical organisations
Independent of data source, can be used for:
Surveys / censuses
Administrative sources / register-based statistics
Mixed sources

9. 4.1 GSBPM - Phases

10. Not a linear model 4.2 GSBPM – Key features
Sub-processes do not have to be followed in a strict order
It is a matrix with many possible paths, including iterative loops within and between phases
Some iterations of a regular process may skip certain sub-processes

11. 4.3 GSBPM – Other uses Harmonizing statistical computing systems
Facilitating sharing of statistical software
Framework for process quality management
Structure for storage of documents
Measuring operational costs

12. 5. Standards - Relations Statistical Information concepts concepts
Statistics
production
GSBPM
GSIM
Technology
Methods
Conceptual
Practical
Statistical
how-to
Production
how-to
SDMX, DDI, RDF, ISO-11179, …

13. 6. GSIM Generic Statistical Information Model
Other standards
DDI
SDMX
Implementation
standards
Conceptual
model

14. 7. SDMX & DDI
DDI offers a very rich model for the documentation of micro-data
SDMX offers a very integrated exchange platform for statistical outputs (IT architectures, tools, web services)
integration of the complete production process
The combined use of both standards could allow a higher level of

15. 8. SDMX Statistical Data and Metadata eXchange
UNSD
World Bank

16. SDMX is the global answer to this.
8.1 SDMX – Why?
The exchange of statistical data and metadata is complex, resource intensive and expensive
In the past, national and international organisations had developed specific approaches and solutions
Opportunities and challenges related to new technologies for machine to machine exchange were coming up, e.g. XML, web services.
SDMX is the global answer to this.

17. 8.2 SDMX - Benefits Efficiency Reduced burden after low investment
Consistent and comparable data and metadata messages produced by different organizations
Harmonized statistical processes, offering new ways of data and metadata exchange (such as data hubs)
Web-based dissemination formats are provided that are computer “readable” and easier to update.

18. 8.3 SDMX - Costs
Development/maintenance of the SDMX standards and guidelines done by the international sponsoring institutions (supported by NSIs)
Standards are public and open source
IT tools are created by sponsoring or other organizations and made freely available
Capacity building by individual sponsoring institutions
User community input by means of open process
Low investment cost – gradual implementation

19. 8.4 SDMX - Opportunities Streamline data flows Across domains
Central management (SDMX Registry)
Across domains
Across organizations
Simplification
Standardization
Harmonization
Software tools
Data sharing
Data structures
Concepts
Code lists

20. 8.5 SDMX - Impacts
Reduced reporting burden via common formats adopted by international organizations for data and metadata exchange
User-friendly access when publishing national data and metadata on the web via global standards for data formats, catalogs/registries and associated services
Improved management and analysis of data via global guidelines for metadata vocabularies and repositories in common formats
Replicable models and tools for statistical information systems at national levels

21. SDMX recognised and supported as the preferred standard
8.6 SDMX – From 1.0 to 2.1
Version 2.0
SDMX-EDI
SDMX-ML
SDMX Registry
2008
SDMX accepted at UN level
Version 1.0
GESMES/TS
SDMX recognised and supported as the preferred standard
Version 1.0
Version 2.0
Version 2.1
September 2004
November 2005
February 2008
April2011

22. 8.7 The SDMX Components Describe statistics in a standard way
Objects and their relationships
Data Structure Definition (DSD), Concepts, Code List
Central management and standard access
SDMX Registry, SDMX Web Services
Cross Domain Concepts
Cross Domain Code Lists
Statistical Domains
Metadata Common Vocabulary
Push
Provider generates and sends file to receiver
Pull
Provider opens web service to data
Receiver downloads regularly
Hub
Special case of pull: receiver downloads on end user request 22 22

23. 9. Summary
To enable a modernized statistical production, standards are the key
Standards at different levels are being used in an increasingly coherent way
GSBPM and GSIM provide conceptual models and facilitate communication
SDMX, DDI and other standards provide implementation models which can be used in a coordinated way
There are now more technologies than just GESMES and XML: a coherent overall model is critical

24. Introduction