1. WP.5 - DDI-SDMX Integration
METIS Work Session
6-8 May 2013
WP.5 - DDI-SDMX Integration
E.S.S. cross-cutting project on Information Models and Standards
Marco Pellegrino, Denis Grofils
Eurostat

2. Outline ESS VIP programme
Cross-cutting project on Information Models and Standards - Scope - Phases - Deliverables
SDMX-DDI Integration: open points for discussion

3. ESS.VIP programme
Transformation programme for the modernisation of the production systems in the European Statistical System (ESS) through:
moving towards more common solutions and shared services and environment
economies of scale and efficiency gains, sharing costs
Context : Production of European Statistics
European statistical system : a complex system with many production lines (NGOs and Eurostat)
Long tradition of output harmonisation (legal framework)
Tradition of cooperative developments (ESSnets)
ICT changes and IT rationalisation in many organisations

4. Infrastructure transformation
Improve communication/network among ESS partners
Provide common information repositories
Foster process interoperabilty
Foster process agility and reuse of components

5. ESS.VIP Programme components
Projects in statistical domains
Technical cross-cutting projects
Frameworks and administrative mechanisms
ADMIN
Information models and standards
Communication
NAPS
Network for information exchange
Governance
PRIX
Data warehouses
Human resources
ESBRs
Shared services
Sharing costs Financial resources
SIMSTAT
Legal framework
ICT
Programme management
Common data validation policy
Several dependencies

6. ESS.VIP business and information principles
Maximum reuse of existing process components and segments
Metadata driven processes allowing adaptation/parameterisation and extension to other contexts
New business process built as a sequence of modular process steps / services
Information objects structured according to available information models and stored in corporate registries/repositories in view of reuse
Adherence to industry and open standards as available (e.g. Plug & Play)

7. Information Models Standards
Objectives:
To ensure that ESS.VIP have access to a set of agreed-upon standards supporting the modernisation of statistical production processes.
To increase coherence between standards, at the same time ensuring that these are consistent with best practices and recommendations from the international community.
To define information models that can be used across the ESS to model structural metadata for micro-data and aggregated data.
To set up guidelines for designing and documenting business processes.
To provide support mechanisms (e.g., capacity-building and training).

8. The four meta-modelling levels
Models?
The four meta-modelling levels
SDMX metamodel
Data model, concepts, DSD
The M0 layer is the real system. A model represents this system at level M1. This model conforms to its metamodel
defined at level M2. The metamodel itself conforms to the meta-metamodel at level M3. The meta-metamodel conforms to itself.
A model is a partial analogy of a system.
The analogy between the model and the represented reality is partial.
The properties of the model are not identical to the properties of the reality.
Real data (e.g. BOP, ESA)
A model represents a system and conforms to a metamodel.

9. Integration of methods and techniques
DATA
Software Services
Administrator
DEFINITIONS
User
Information Model
Users would insert definitions in the information system using a subject matter language (instead of giving them as requirements to EDP people) and definitions would drive software, so the desired result could be obtained sooner and cheaper ! Moreover, definitions would be proper and accurate elsewhere the process would fail and definition would remain in the information system, so they could be inquired to remember what the system contains and does, instead of having the behaviour of the information system hidden in computer programs !
To obtain such a goal, the system must contain also information needed to operate it, like the one related to controls, calculations, estimates and other functions to be executed, together with the order and the conditions of execution (the process).
Design
Build
Collect
Process
Disseminate
Use

10. Which standards and models?
Re-use existing resources
Link to new initiatives (e.g. Sponsorship on Standardisation, GSIM)

11. DDI and SDMX SDMX: Statistical Data and Metadata eXchange
Standard for the exchange of statistical data and metadata
“the preferred standard for exchange and sharing of data and metadata in the global statistical community”
Widely used in the ESS and at UN level
Extended to support some registers as well
DDI: The Data Documentation Initiative
Standard for the documentation of data
Initially focused on archiving micro-data in the area of social sciences – Widely used in data archives
Extended to support the full life-cycle of data

12. The SDMX-DDI approach
Informal meetings ( ) between members of the SDMX and DDI communities, looking for ways in which the standards could be used together effectively
Initiative of the SDMX Secretariat through its Technical Working Group
Approach to using SDMX and DDI interchangeably Now, we are at the stage where implementations are being investigated and prototyped
Not “if”, but “how”
Most often, this is done in the context of the Generic Statistical Business Process Model (GSBPM)
Idea of “industrialised” statistical production
Strong emphasis on process management

13. Generic Statistical Business Process Model
DDI
DDI
SDMX

14. GSBPM, DDI and SDMX: towards a complete system?

15. DDI and SDMX
DDI offers a very rich model for the documentation of micro-data
SDMX offers a very integrated exchange platform for statistical outputs (IT architectures, tools, web services)
When people think about using SDMX and DDI together, they make assumptions
Microdata (and tabulations) can be described using DDI
A transformation could be applied to produce SDMX to describe the aggregates/tables
There is a straight mapping from DDI to SDMX
Interestingly, this conceptual model is not how the use of DDI and SDMX together is being approached in reality
The Devil is in the details!
The combined use of both standards could allow a higher level of integration of the complete production process
But: The devil is in the detail!

16. Characterizing the Standards: DDI
DDI Lifecycle can provide a very detailed set of metadata, covering:
The study or series of studies
Many aspects of data collection, including surveys and processing of microdata
The structure of data files, including hierarchical files and those with complex relationships
The lifecycle events and archiving of data files and their metadata
The tabulation and processing of data into tables (Ncubes)
Allows for a link between the microdata variables and the resulting aggregates

17. Characterizing the Standards: SDMX
Describes the structure of aggregate/dimensional data (“structural metadata”)
Provides formats for the dimensional data
Provides a model of data reporting and dissemination
Provides a way of describing and formatting stand-alone metadata sets (“reference metadata”)
Provides standard registry interfaces, providing a catalogue of resources
Provides guidelines for deploying standard web services for SDMX resources
Provides a way of describing statistical processes

18. SDMX Process Metadata Data validation and editing, SDMX Registry,
DSD and data set,
MSD, metadata set,
Web services
Process Metadata
DDI has much more detailed metadata at the level of the study, because it is intended to describe the full process of data production (the data lifecycle)
DDI provides more complete descriptions of the processing of data
SDMX provides more architectural components, to support reporting/collecting and exchange
SDMX provides generic mechanisms to support foreseen and non-foreseen use cases (categorisation, HCL, MSD)
Similarities:
Both standards use a similar mechanism for structuring URN identifiers
Both standards use a similar model for identifiable, versionable, and maintainable things
Both have a concept of an owning agency
There is a very similar set of rules about versioning and maintenance
Both standards use “schemes” as packages for lists of like items
Both standards are designed to support reuse, and have similar referencing models

19. Analysis of use cases
The SDMX TWG has been defining a set of relevant use cases where the two standards could be compared and, if possible, used together:
Survey data collection
Administrative and register data
Combined use of DDI and SDMX
Micro-data access and on-demand tabulation of micro-data
Metadata and quality reporting

20. Survey
A Survey is targeted at a specific Population and comprises Questions
Question may be linked to a Variable which specifies
- conceptual meaning (Concept)
valid set of responses that are allowed (Category Scheme and contained Category)
Output from the Survey is a Unit Record Data Set

21. The Proposed Approach The full set of information includes:
The unit record data
Structural information about the variables and representations
Additional information about how the data has been generated/collected/processed
In DDI, this set of information can be expressed as a DDI instance and a data file
Both the structural and processing metadata can be expressed as a single DDI instance

22. Data Process and Cleaning
Editing Process can include
Validation
Outlier Trimming
Recodes
Editing for Non Response
Editing Process consists of
Description of the process
(Process Description)
Software environment (Executable Code)

23. Tabulation The result of a Tabulation is an Aggregate Data Set
Structured according to a Dimensional Structure Definition (SDMX DSD)
Comprising Dimensions, Attributes, Measures
Each take their semantic and representation from a Variable
Data Set comprises statistical series
Key
Attributes
Observations

24. Output Tables

25. Concepts

26. Metadata Set Unit Record Data DDI Instance ASCII Data File SDMX Data
SDMX Structural
Metadata
SDMX Metadata
Report
In SDMX, we have three XML files:
A file holding the data, expressed as dimensional microdata
The unit identifier is a dimension
The variable identifier is a dimension
There are dimensions related to time
A reference metadata report with all other metadata describing the process/collection/generation of the administrative data
A file describing the concepts, data structure, and codelists (“structural metadata”) for the data, and also the structure of the metadata report

27. The challenge
Its not about which flavor of XML we use (XML doesn’t really matter)
It’s about data and metadata!
If I want to use DDI to describe my data, and you want to use SDMX, how can we ensure that we are getting the same data and metadata?

28. The challenge (2)
If I am using SDMX, but I am sent DDI, a simple transformation must give me the same payload of data and metadata
Vice-versa for SDMX users
Conventions will need to be established regarding identifiers and the way the unit record files are structured
There will need to be agreed models for each business case

29. Combined DDI-SDMX approaches
Mixing the two standards within an implementation, allowing for the expression of the same metadata set in both standards, so that the information could be transformed from one format to the other.
This way, it would become possible to select either DDI or SDMX for a particular operation, similar to what we discussed above regarding application functionality.
Metadata stored and indexed in such a fashion that it can be expressed either as SDMX or DDI on an as-needed basis.
Metadata Repository and Registry project at ABS.
The actual format used for metadata storage may be neither SDMX nor DDI, so long as it can be expressed using both standards.
GSIM to be implemented through a combination of SDMX and DDI?

30. Generic Statistical Information Model (GSIM)
SDMX
DDI
ISO 11179
Etc.

31. Feedback is welcome
Thank you!
Marco Pellegrino
Denis Grofils