1. The Euro SDMX Registry & SDMX Global Registry
Bengt-Åke Lindblad (Eurostat B3)
Emmanuel Clement (Eurostat B5)
SISAI Meeting – Luxembourg – June 2012

2. Presentation overview
What is the Euro SDMX Registry
Role of the Euro SDMX Registry
Euro SDMX Registry in figures
Evolution and Roadmap 2012
What is the SDMX Global Registry
Early version of the SDMX Global Registry
Tentative roadmap for the SDMX Global Registry 2

3. What is the Euro SDMX Registry?
This metadata registry implements the SDMX registry specifications
It is a repository of SDMX artefacts (DSDs, standard code list…) used for SDMX based data/metadata exchange by Eurostat and Member States
It has a web-based user interface as well as web services for interacting with SDMX structural metadata objects in use within Eurostat, the ESS and beyond
It is accessible to other programs over the Internet (or an Intranet or Extranet) to provide information needed to facilitate the reporting, collection and dissemination of European statistics
It is important to stress that registry services are not concerned with the storage of data or reference metadata sets. Data and reference metadata sets are stored elsewhere. The registry is only concerned with providing information needed to access the data and reference metadata, including information on the structure of the data and reference metadata.
In addition, it should be understood that metadata registries are intended to interact with other applications, rather than directly with human users. An application which wants a particular data or reference metadata set should query the registry to know where those data and metadata sets reside, and how they are structured. The application can then go and retrieve the data directly from another database.
The Euro-SDMX registry is a metadata registry which implements the SDMX registry specifications. The SDMX Registry is tasked with providing structure, organisation, and maintenance and query interfaces for most of the SDMX components required to support the data sharing vision. Provides the specification for the logical registry interfaces, including subscription/notification, registration of data and metadata, submission of structural metadata, and querying The SDMX Registry plays an important role in this architecture, in fact it can be seen as a central application which is accessible to other programs over the Internet (or an Intranet or Extranet) to provide information needed to facilitate the reporting, collection and dissemination of statistics. In its broad terms, the SDMX Registry – as understood in web services terminology – is an application which stores metadata for querying, and which can be used by any other application in the network with sufficient access privileges. 3 3 3

4. Role of the Euro SDMX Registry
To enable IT applications, organisations (NSIs, and individuals
To share data and metadata structures and other SDMX artefacts
To allow applications to sign up (or subscribe) for notifications
To act as central repository of SDMX objects used in (international) data and metadata exchange processes (DSDs, MSDs, Code Lists, etc.)
To interact with other IT applications such as:
ESS Metadata-Handler
Data transmission (eDAMIS)
Global Registry
To allow organisations to publish statistical data and reference metadata in known formats such that interested third parties can discover these data and interpret them accurately and correctly
To enable National Statistics Institutes and other external organisations to obtain Data Structure Definitions (DSD) and other structural metadata, such as Metadata Structure Definitions (MSD) for the Euro-SDMX Metadata Structures (ESMS) and European Standard Quality Reports (ESQR)
To provide information about the structure of datasets and reference metadata sets, answering questions like: what code lists do they use? What concepts are involved?
To provide information about what datasets and reference metadata sets are available, and where they are located
To allow applications to sign up (or subscribe) for notifications, so that when information in the registry is updated (for example, a dataset or reference metadata set of interest becomes available, or structural metadata are changed), the application will be automatically alerted.
To interact with other IT applications:
Central place in the ESS Metadata-Handler (e.g. interaction with the NRME)
Central place in data transmission (interaction with eDAMIS)
Global Registry 4 4 4

5. as central repository for structural data
The Euro SDMX Registry
as central repository for structural data
ESS Metadata Handler
Other tools
Euro SDMX
Registry
Storage of MSDs and report structures used to create Reference Metadata files (NRME)
Like Data Structure Wizard or SDMX Converter
The registry is the corner stone
Storage of DSDs used for validation and creation of WebForms
EDAMIS
Web Interface for the maintenance of the data

6. Euro SDMX Registry in figures
In 2008 the Euro SDMX Registry was fed with the content of two Access DB (ESTAT+ECB)
GESMES-TS objects used in data/metadata exchange at that time (SDMX compliant)
From 2008 and onwards: All newly created SDMX Artefacts used for data/metadata exchange are stored in the Euro SDMX Registry 6 6 6

7. Euro SDMX Registry in figures
Situation as of May 2012: 7 7 7

8. Evolution and Roadmap 2012 Extension of functionalities
Re-design of the Graphical User Interface
Revision of the Access Control Mechanism
Improvement of performance to meet service level objectives and limit expensive design rework
Support SDMX 2.1 Technical Specification 8 8 8

9. Evolution and Roadmap 2012 (1)
Extension of functionalities
Import & Export of artefact from other file formats
Locking & editing of artefacts
Be able to identify in which other artefacts a given artefact is referenced
Ensure that no objecct can be deleted if it is referenced by another object
Update warning mechanism
Enchancement of error messages/codes
Audit trail 9 9 9

10. Evolution and Roadmap 2012 (2)
New Graphical User Interface
Different philosphy with tabular approach, resizable screens/windows and new features such as:
Anonymous access
Login via ECAS/registered mail
Quick/Advanced Search
What’s new feature
Simple/detailed view of Artefacts
Wizard guided creation of artefacts
Storing of favourite artefacts/queries
Compare artefacts
Table sorting etc. 10 10 10

11. Collapsable and resizable elements to improve user experience
GUI re-design (1)
Tabular layout
Easy browsing on results, content of artefact selected displayed under results
Collapsable and resizable elements to improve user experience 11

12. GUI re-design (2) New Quick search functionality
What's new containing recently modified artefacts 12

13. GUI re-design (3) New filtering and sorting functionalities
New Referenced by functionality 13 13

14. GUI re-design (4) Advanced Search using more attributes
Customizable Search depending on the type of artefact 14

15. Evolution and Roadmap 2012 (3)
Revision of the Access Control Mechanism
ECAS authentication system
Revision of the login mechanism
Role-Based Access Control (RBAC)
Sub-agencies allowed
Assignment of artefacts to subgroup of users within an agency
Public/private artefacts
Administration Module 15 15 15

16. Evolution and Roadmap 2012 (4)
Resolve performance issues
Improvement of performance to meet service level objectives and limit expensive design rework
Improve SQL statements used to retrieve artefacts
Improve insertions of artefacts in the database
Resolve caching mechanism issue 16 16 16

17. Evolution and Roadmap 2012 (5)
Support SDMX 2.1 Specifications
Support for new web services interfaces
Explicit WSDL for SOAP services
New RESTful API
Different End Points for SDMX 2.1 and 2.0 messages
Unique dataset model for time series and non-time series
Data model adaption to deal with both versions of the SDMX standard 17 17 17

18. What is the SDMX Global Registry?
The Global Registry is a term which has been discussed on a community level since SDMX Global Conference in Washington The following are deemed as core functions:
Provision of mechanism for SDMX as a maintenance agency to maintain and make available SDMX global resources;
Offering a facility for registered Agencies to publish their structures for dissemination, their location of their Registry, acting as a “federated node”;
Support for hosting all types of SDMX structure.
core functions
The following are deemed as core functions of the Global Registry:
Provision of a mechanism for SDMX as a maintenance agency to maintain and make available SDMX global resources such as the top level Agency Scheme of Agency Ids, the cross domain concepts and code lists, and the concept scheme of dimension/attribute roles..
This is seen as a high priority requirement for SDMX as without these structures being maintained and globally accessible, at best there will be no guarantee of interoperability, and at worst SDMX simply will not work .
Offering a facility for registered Agencies to publish their structures for dissemination
This will be necessary for the wide dissemination of “global DSDs” or any metadata that is to be disseminated to a wide audience.
Offering a facility for registered Agencies to register the location of their Registry, acting as a ‘federated node’
If there is no registry federation then the Global Registry cannot know about structures maintained in an organisation-controlled registry and it cannot offer a service to retrieve such structures. This would force all DSDs that are to be shared or made available publicly to be uploaded in full to the Global Registry.
Support for hosting all types of SDMX structure (i.e. not restricted to a sub set of the structures contained in the SDMX standards)
Different organisation will require different structures to be maintained. If these are not supported there is a danger that (2) and (3) above cannot be offered.
As a minimum, an SDMX registry supports machine-to-machine interaction via a web service API. In addition, it should also offer a basic GUI to support user query for registry structures. 18 18 18

19. The SDMX Global Registry
The SDMX Sponsors will put it in place for storing SDMX global artefacts;
The SDMX Global Registry will be linked to other SDMX registries;
SDMX artefacts should only be stored once and be accessible through different registries;
The governance of the SDMX artefacts is discussed by the SDMX Secretariat (e.g. maintaining agency, versioning etc.); guidelines will be available in autumn 2012.
core functions
The following are deemed as core functions of the Global Registry:
Provision of a mechanism for SDMX as a maintenance agency to maintain and make available SDMX global resources such as the top level Agency Scheme of Agency Ids, the cross domain concepts and code lists, and the concept scheme of dimension/attribute roles..
This is seen as a high priority requirement for SDMX as without these structures being maintained and globally accessible, at best there will be no guarantee of interoperability, and at worst SDMX simply will not work .
Offering a facility for registered Agencies to publish their structures for dissemination
This will be necessary for the wide dissemination of “global DSDs” or any metadata that is to be disseminated to a wide audience.
Offering a facility for registered Agencies to register the location of their Registry, acting as a ‘federated node’
If there is no registry federation then the Global Registry cannot know about structures maintained in an organisation-controlled registry and it cannot offer a service to retrieve such structures. This would force all DSDs that are to be shared or made available publicly to be uploaded in full to the Global Registry.
Support for hosting all types of SDMX structure (i.e. not restricted to a sub set of the structures contained in the SDMX standards)
Different organisation will require different structures to be maintained. If these are not supported there is a danger that (2) and (3) above cannot be offered.
As a minimum, an SDMX registry supports machine-to-machine interaction via a web service API. In addition, it should also offer a basic GUI to support user query for registry structures. 19 19 19

20. The early version of the Global Registry
Will be based on (and limited to) the SDMX 2.1 Registry specification
The core use cases the early version must support are limited to the following ones:
Store and retrieve SDMX artefacts;
Store and retrieve internationally agreed DSDs;
Allow Agencies (organisations) to store and retrieve SDMX artefacts they maintain in the Global Registry;
Provide a GUI for maintenance of and viewing of SDMX artefacts.
Overall NSIs should access the EURO SDMX registry which will also provide access to the SDMX Global Registry.
We will communicate more on the use and access of the Euro SDMX Registry and the Global Registry in the ESS Metadata Handler Newsletter.
The early version of the Global Registry will be based on (and limited to) the SDMX 2.1 Registry specification. By “limited to”, we want to emphasize the fact that no modification of the SDMX information model or the SDMX Registry specification will be requested for the early version of the Global Registry.
The core use cases that the early version of the Global Registry must support are limited to the following ones:
Store (including updates) and retrieve the SDMX artefacts (core concepts and codelists from the content-oriented guidelines, list of top-level maintenance agencies, etc.).
Store (including updates) and retrieve the internationally agreed DSDs.
Allow agencies which do not want to deploy their own SDMX registry to store (including updates) and retrieve the SDMX artefacts they maintain in the Global Registry.
Provide a GUI for maintenance of and viewing of SDMX artefacts in the Global Registry. 20 20 20

21. Development of enhancements
Tentative Roadmap
Procurement
Development of enhancements
Initial setup
Security
audit
Deployment
3 months
3 months
2 weeks
2 weeks
1 month
June 2012
Sep 2012
Dec 2012
Jan 2013
Feb 2013 21 21 21

22. Contact Bengt-Åke Lindblad Bengt-Ake.Lindblad@ec.europa.eu
Emmanuel Clement 22 22 22