1. SISAI STATISTICAL INFORMATION SYSTEMS ARCHITECTURE AND INTEGRATION
WORKING GROUP
3rdMEETING
13-14 MAY 2013
ITEM 2.6
Cross-cutting project on standards and information models including the integrated use of DDI and SDMX

2. E.S.S. cross-cutting project on Information Models and Standards
SISAI 3 Meeting
14th of May 2013
Marco PELLEGRINO (Eurostat)
Denis GROFILS (Eurostat)

3. ESS.VIP programme
Transformation programme for the modernisation of the production systems in the European Statistical System (ESS) through:
moving towards more common solutions and shared services and environment
economies of scale and efficiency gains, sharing costs
WORK Vs CHANGE?
Context : Production of European Statistics
European statistical system : a complex system with many production lines (NGOs and Eurostat)
Long tradition of output harmonisation (legal framework)
Tradition of cooperative developments (ESSnets)
ICT changes and IT rationalisation in many organisations

4. ESS.VIP business and information principles
Maximum reuse of existing process components and segments
Metadata driven processes allowing adaptation and extension to other contexts
New business process built as a sequence of modular process steps / services
Information objects structured according to available information models and stored in corporate registries/repositories in view of reuse
Adherence to industry and open standards as available (e.g. Plug & Play)

5. ESS.VIP Programme components
Projects in statistical domains
Technical cross-cutting projects
Frameworks and administrative mechanisms
ADMIN
Information models and standards
Communication
NAPS
Network for information exchange
Governance
PRIX
Data warehouses
Human resources
ESBRs
Shared services
Sharing costs Financial resources
SIMSTAT
Legal framework
ICT
Programme management
Common data validation policy
Several dependencies

6. Information Models Standards
Objectives:
To ensure that ESS.VIP have access to a set of agreed-upon standards supporting the modernisation of statistical production processes.
To increase coherence between standards, at the same time ensuring that these are consistent with best practices and recommendations from the international community.
To define information models that can be used across the ESS to model structural metadata for micro-data and aggregated data.
To set up guidelines for designing and documenting business processes.
To provide support mechanisms (e.g., capacity-building and training).

7. Phases
2013
2014
2015+
Q1-2
Q3-4
1.1. Review of ESS.VIP project requirement
1.2. Recommendations to ESS.VIP
2. Analysis of standards and models
3. Proposal for an integrated standard to handle micro-data and aggregated data
4. Business process description
5. Maintenance and further enhancement of standards
6. Capacity building

8. Which standards and models?
Re-use existing resources
Link to on-going initiatives (e.g. Sponsorship on Standardisation, ESSnets, GSIM and DDI-SDMX mapping)

9. The SDMX-DDI approach SDMX-DDI dialogue (2010-2013)
Initiative of the SDMX Secretariat through its Technical Working Group
HLG project: Frameworks and Standards for Statistical Modernisation
Approach to using SDMX and DDI interchangeably Implementations being investigated and prototyped (not “if”, but “how”)
Most often, this is done in the context of the Generic Statistical Business Process Model (GSBPM)
Idea of “industrialised” statistical production
Strong emphasis on process management
Informal meetings ( ) between members of SDMX and DDI communities, looking for ways in which the standards could be used together effectively
Initiative of the SDMX Secretariat through its Technical Working Group

10. Generic Statistical Business Process Model
DDI
DDI
SDMX

11. GSBPM, DDI and SDMX: towards a complete picture?

12. Characterizing the Standards: DDI
DDI Lifecycle can provide a very detailed set of metadata, covering:
The study or series of studies
Many aspects of data collection, including surveys and processing of microdata
The structure of data files, including hierarchical files and those with complex relationships
The lifecycle events and archiving of data files and their metadata
The tabulation and processing of data into tables (Ncubes)
It allows for a link between microdata variables and the resulting aggregates

13. Characterizing the Standards: SDMX
Describes the structure of aggregate/dimensional data (“structural metadata”)
Provides formats for the dimensional data
Provides a model of data reporting and dissemination
Provides a way of describing and formatting stand-alone metadata sets (“reference metadata”)
Provides standard registry interfaces, providing a catalogue of resources
Provides guidelines for deploying standard web services for SDMX resources
Provides a way of describing statistical processes

14. SDMX Process Metadata Data validation and editing, SDMX Registry,
DSD and data set,
MSD, metadata set,
Web services
Process Metadata
DDI has much more detailed metadata at the level of the study, because it is intended to describe the full process of data production (the data lifecycle)
DDI provides more complete descriptions of the processing of data
SDMX provides more architectural components, to support reporting/collecting and exchange
SDMX provides generic mechanisms to support foreseen and non-foreseen use cases (categorisation, HCL, MSD)
Similarities:
Both standards use a similar mechanism for structuring URN identifiers
Both standards use a similar model for identifiable, versionable, and maintainable things
Both have a concept of an owning agency
There is a very similar set of rules about versioning and maintenance
Both standards use “schemes” as packages for lists of like items
Both standards are designed to support reuse, and have similar referencing models

15. DDI and SDMX
DDI offers a very rich model for the documentation of micro-data
SDMX offers a very integrated exchange platform for statistical outputs (IT architectures, tools, web services)
When people think about using SDMX and DDI together, they make assumptions
Microdata (and tabulations) can be described using DDI
A transformation could be applied to produce SDMX to describe the aggregates/tables
There is a straight mapping from DDI to SDMX
Interestingly, this conceptual model is not how the use of DDI and SDMX together is being approached in reality
The Devil is in the details!
The combined use of both standards could allow a higher level of integration of the complete production process
No «one size fits all» solution
But: The devil is in the detail!

16. Analysis of use cases
The SDMX TWG has been defining a set of relevant use cases where the two standards could be compared and, if possible, used together:
Survey data collection
Administrative and register data
Combined use of DDI and SDMX
Micro-data access and on-demand tabulation of micro-data
Metadata and quality reporting

17. Survey data: the proposed approach
Process a (CSV) micro-data file accompanied by a DDI instance with both structural and processing metadata
Import data file based on structural info
Derive new variables
Aggregate data
Output aggregated SDMX data files
Structured according to a Data Structure Definition (+MSD?)
Comprising dimensions, attributes and measures
Each taking their semantic and representation from a Variable
Without loss of metadata!

18. The challenge
Is not about which "flavour" of XML we use (XML doesn’t really matter)
It’s about data and metadata!
If I am using SDMX, but I am sent DDI, a simple transformation could give me the same payload of data and metadata. Vice-versa for SDMX users
Conventions will need to be established regarding identifiers and the way the unit record files are structured
There will need to be agreed models for each business case
At the end, users would not really care whether DDI or SDMX is used, as long as the system supports the use cases!

19. The vision: combining DDI and SDMX
Mixing the two standards within an implementation, allowing for the expression of the same metadata in both standards, so that the information could be transformed from one format to the other
Metadata stored and indexed in such a fashion that it can be expressed either as SDMX or DDI on an as-needed basis
The actual format used for metadata storage may be neither SDMX nor DDI, so long as it can be expressed using both standards
Standards-agnostic Metadata Repository and Registry
Portal for data and metadata discovery
GSIM to be implemented through a combination of SDMX and DDI?

20. Generic Statistical Information Model (GSIM)
Dedided on the conceptual framework, now we can focus on “how to” implement (methods, technoology)
We have decided on the conceptual framework (GSBPM, GSIM). Now we can focus on “how to” implement (SDMX, DDI, RDF, etc.)

21. GSBPM, DDI and SDMX: towards a complete picture?

22. Marco.Pellegrino@ec.europa.eu Denis.Grofils@ec.europa.eu
Feedback is welcome
Comments on the IMS cross-cutting project: - on relevance and timeline - on open issues (IT infrastructure) - on how to move forward (task-force, sprint?)
Expressions of interest in participating