1. SDMX Standards Relationships to ISO/IEC 11179/CMR Arofan Gregory Chris Nelson Joint UNECE/Eurostat/OECD workshop on statistical metadata (METIS): Geneva 3-5 April 2006

2. Major Points Activity and Results Scope Areas of Commonality Applying ISO 11179 to SDMX CMR and SDMX Registries Use in Applications: A Case Study

3. Activity We were asked to describe the relationship between ISO 11179/CMR and SDMX There is not complete overlap –We focused on points of contact between them This was an interesting project, which we will continue working on

4. Results ISO 11179/CMR and SDMX: –ISO 11179 provides a useful pivotal model for mapping between CMR (or others) and SDMX constructs –ISO 11179 does not model the semantics of a metamodel – it is useful for the semantics of the models (specific metadata structure) or even instances of the model (specific data set) –The utility of this mapping depends on the application The SDMX and ISO 11179 registries are complementary in function (semantics + access)

5. Characterization ISO 11179 models the semantics of data elements CMR extends this to cover survey lifecycle metadata – adds structure SDMX models the structure of metadata and data for aggregate statistical data – you supply your own semantics (concepts) This is not a comparison of like things: it is a connecting of complementary ones

6. Scope – ISO 11179 ISO 11179 is a metadata content standard focused primarily on the semantics of data. Secondarily, it provides the rules and structures for registering descriptions of data. The standard exists in 6 Parts, and the 2nd edition is now available. –Part 1: Framework for the specification and standardization of data elements – Part 2: Classification for data elements –Part 3: Registry metamodel and basic attributes –Part 4: Rules and guidelines for the formulation of data definitions –Part 5: Naming and identification principles for data elements –Part 6: Registration of data elements This presentation is concerned mainly with Parts 3 and 5

7. SCOPE: ISO 11179 Part 3 High Level View - Data Elements and Concepts Country Identifier Countries of the World Country Name Country Code ISO 3166: Country Code List

8. SCOPE – CMR The CMR model is designed to support –metadata necessary to describe the survey life cycle –linkages between similar designs and processes used across surveys –use of metadata to drive systems in support of the survey life cycle The CMR model extends the ISO 11179 (Part 3) metamodel to support CMR specific artefacts

9. SCOPE - CMR ISO 11179 CMR Links to appropriate ISO 11179 artefact to supply semantics and management

10. SCOPE - SDMX SDMX standards comprise –Information Model –Syntax implementations based on the Information model (XML schemas and UN/EDIFACT) –Content-oriented guidelines Statistical subject matter domain scheme Cross Domain metadata concepts Metadata vocabulary (MCV – Metadata Common Vocabulary)

11. Information Meta Model Aggregated data, metadata, structural metadata Design Meta Schema for aggregated data and metadata Create data/metadata structure instance (data/metadata structure definition) Use cross domain metadata concepts and code lists Generate structure specific schema Report/publish data/metadata SDMX Registry Register

12. Areas of Commonality ISO 11179 and SDMX Both standards support the definition of –Concepts –Use of concepts in structures –Representations/Value domains Both standards support the concept of a registry –ISO 11179 specifies a registry metamodel –SDMX does not specify a registry metamodel –ISO 11179 does not specify registry interfaces based on the registry model –SDMX specifies registry interfaces based on the SDMX model Implementation of a registry and mapping to a registry model is left to the implementor Can use ISO 11179 registry implementation, ebXML registry implementation, or bespoke registry implementation

13. Concepts – ISO 11179

14. Concepts - SDMX Conceptual Domain (for SDMX this defines the core/default representation: implicit here is whether it is enumerated or non enumerated) [not given explicit name in SDMX] Data Element Concept ISO-11179 equivalent

16. Data Element: ISO 11179

17. Concept Usage - SDMX Concept Usage = Data Element [can be given explicit name in SDMX (e.g. ISO 11179 equivalent name), but this is not required and is not used for identification purposes] Value Domain

18. Concept Usage - SDMX There is no explicit “Data Element” name for usage of a Concept (e.g. FREQUENCY Concept used for a Dimension) –Concept names are used within the context of the role of the Concept (Dimension, Attribute, Measure) – these are structure components –The unique identifier for a structure component comprises: e.g. BOP.KEYDESCRIPTOR.FREQUENCY [actual identifier includes maintenance agencies] This is useful if one wishes to reference the component in, say, a registry scenario, but it is not normally retained in processing systems –For instance in the data structure definition the identifier of the Concept is used, but the full identifier can be constructed from its context in the structure –However the SDMX-ML specification does allow the full identifier [URN] to be specified as well ISO 11179 equivalent name could be BOP_KEYDESCRIPTOR.FREQUENCY.CODE

19. SDMX: Part of Key Family

20. CMR and ISO 11179 CMR is a model that supports the survey lifecycle Element and Concept semantics are provided by an association to ISO 11179 e. g Question is associated to Data Element Concept Response Domain is associated to Value Domain Data Set component (element) is associated to Data Element Data set contains the output from a survey All data elements are explicitly named ISO 11179 is the pivot between CMR and SDMX We need to apply ISO 11179 to SDMX

21. Applying ISO 11179 to SDMX ISO 11179 gives you: [Object_Class].[Property].[Representation] An ISO 11179 instance would have: [Object_ID].[Property_Term].[Representation_Term] We have these constructs in SDMX data and metadata sets

22. SDMX Objects For data, we have a limited set of object classes: –Data Sets –Groups –Series –Observations For reference metadata, we have a larger set of object classes: –Data Providers –Category Schemes –Data and Metadata Flows –Etc. Note that SDMX instance IDs are compound –For data, a “key” –For metadata, a “target identifier”

23. SDMX Properties and Representations In SDMX, properties are taken from Concepts The names and definitions are supplied by the user for most properties –Some are required by the model Representations are equivalent to those in ISO 11179 (code, text, etc.)

24. [Observation_ID]. Object (from the SDMX model) Property (from the Concept that is the Data Attribute or Measure in the Data Structure Definition) Representation Term (derived from the Representation of the Data Attribute or Measure in the Data Structure Definition) Confidentiality. code [Series_ID]. Availability. code [Group_ID]. Title.text [Observation_ID]. Value.number [Data Set_ID]. Title. text

25. SDMX Object IDs Compound object IDs in SDMX are either: –Data Keys (for Group, Series, or Observation), which depend on the data structure definition –Metadata Target Identifiers (for data provider, metadata flow, categorization scheme, etc.), which depend on the metadata structure definition Data Examples –Observation Key: “Annual – Total Population – Kenya – 1994” –Series Key: “Annual – Total Population – Kenya” –Group Key: “Annual – Total Population” (this is the group of all countries)

26. Examples, cont. For the concept of “Availability” for a series, in ISO 11179 we would have: Annual – Total Population – Kenya. Availability. Code Note that the SDMX distinction between “dimensions” and “attributes” is not important to ISO 11179 – the semantic models use the same basic approach for both –You only model the semantics of discrete, value- containing constructs: attributes and observation values

27. Reference Metadata Object IDs in SDMX All of the objects in the SDMX Information Model can be Object Classes for ISO 11179 They can be combined to identify new object classes The IDs for instances of objects are composite “target identifiers”

28. Category Scheme Category can have child categories comprises subject or reporting categories Data or Metadata Flow Data Provider Provision Agreement can get data/metadata from multiple data/metadata providers can provide data/metadata for many data/metadata flows using agreed data/metadata structure Structure Definition uses specific data/metadata structure can be linked to categories in multiple category schemes Data Set or Metadata Set publishes/ reports data sets or metadata sets conforms to business rules of the data/metadata flow Example of Some SDMX Object Classes

29. Reference Metadata Example A data provider is identified with an ID which references the organization scheme (and agency) the data provider ID code comes from: [OrgSchemeAgencyID]_[OrgSchemeID]_[Data_ProviderID] Example: sdmx_sdmxProviderScheme1_IMF shows the ID for a data provider from agency scheme 1 maintained by SDMX, identifying the IMF as a data provider

30. Reference Metadata Example, cont. To model a “Contact_Name” concept in an SDMX metadata report for a specific data provider, we have in ISO 11179 sdmx_sdmxProviderScheme1_IMF.Contact_Name.Text A more useful approach is possible: Data_Provider.Contact_Name.Text This is not possible for data objects in SDMX.

31. Registry Model: ISO 11179 But no API specification for implementing the model

32. Registry Metamodel: SDMX SDMX does not have an abstract registry metamodel SDMX specifies registry interfaces based on the SDMX Information Model Compliance with SDMX registry standards is based on implementation of the SDMX registry APIs The actual registry implementation can be whatever the registry service provider chooses –The JEDH pilot project is implemented using an ebXML registry (this has a model which is very similar to the ISO 11179 model) –An SDMX compliant registry could use an ISO 11179 registry (if such an implementation exists)

33. Functionality An ISO 11179 registry is used to provide a semantic registry of specific data elements –The focus is on understanding the meaning of data and metadata An SDMX registry provides mechanistic services to facilitate the exchange of data and metadata sets –The focus is on access to data and metadata These are complementary functions

34. Using the Mapping CMR is one example of a model which could be mapped against SDMX –SDMX is about aggregate data exchange –CMR is about survey lifecycle metadata –These are potentially related, but are not the same thing To be useful in an application, we care about the semantic equivalencies of data and metadata –These equivalencies allow us to re-use and map data between models and systems –ISO 11179 acts as a pivot

35. Semantics Data and Metadata Structures Semantics Data and Metadata Structures ISO 11179 SDMX Structures and rules prescribed by SDMX standards Structures and rules are proprietary to individual organisations Describe SDMX semantics based on SDMX structures Organisations map between their data and metadata structures and and SDMX data and metadata structures using ISO 11179 semantics CMR defines a conceptual model for data and metadata structures for the survey lifecycle, using the ISO 11179 model for structuring semantics SDMX defines a conceptual model for data and metadata structures for the aggregated data and reference metadata and provides a canonical syntax representation

36. Another Case Study One use case – CMR – is not enough to validate our work At the ISO TC-154 meeting in Vancouver we met with the designers of ISO-15000 part 5, the “Core Components Technical Specification” (CCTS) –CCTS is a modelling methodology for e-commerce, standardized in ISO TC 154 –It is based on ISO-11179 We were asked: “Could SDMX data be modelled according to CCTS?”

37. Why Bother? Statistics and e-commerce do have connection points: –Customs and international trade –Payments transactions and business reporting Some systems today map between business transactions and statistical reports Thus, mapping between CCTS (e-commerce transactions) and SDMX (statistics) is potentially a real-world use case.

38. The Answer is “Yes” Based on our SDMX-to-ISO 11179 mapping: –We could express our semantics –Compare them to the e-commerce semantics –Determine the relationship between specific data elements This was a quick, straightforward validation of our work

39. Conclusions ISO 11179 is very useful as a pivotal semantic model for working with other models (CMR, CCTS, etc.) SDMX is mapped at the model level, not the metamodel level –Semantics are introduced at this level The SDMX and ISO 11179 registries are complementary in function (semantics + access) The structural mapping of any model depends on the ISO 11179 implementation –ISO 11179 only does semantics, not structure –Applications need both structural and semantic mapping