METADATA HARMONISATION SDMX Training BANK INDONESIA SEPTEMBER 2015 YOGYAKARTA, INDONESIA

Harmonising Metadata Database Web Client External Clients And Organisations Internal Clients And Processes Programmers Data Providers Submit Data Fusion Registry

Harmonising Metadata Database Observations Observation Id Value Status Confidentiality … Observations Observation Id Value Status Confidentiality … Dimension Dimension Id Classification Id Dimension Title … Dimension Dimension Id Classification Id Dimension Title … Classification Classification Id Codelist Id Title Description Version Operative Date … Classification Classification Id Codelist Id Title Description Version Operative Date … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dimension Item Id Dimension Id Classification Item Id Display Order ….. Dimension Item Id Dimension Id Classification Item Id Display Order … More Tables

Harmonising Metadata Database Observations Observation Id Value Status Confidentiality … Observations Observation Id Value Status Confidentiality … Dimension Dimension Id Classification Id Dimension Title … Dimension Dimension Id Classification Id Dimension Title … Classification Classification Id Codelist Id Title Description Version Operative Date … Classification Classification Id Codelist Id Title Description Version Operative Date … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dimension Item Id Dimension Id Classification Item Id Display Order ….. Dimension Item Id Dimension Id Classification Item Id Display Order … More Tables Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value

Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations

Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository Obtain information about Datasets they must provide. Obtain rules on which series they must supply as a provider Data Providers Data Validation can be performed based on Registry metadata. It can be a central process. Not tied to the target database platform

Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository Web Client Build the entire data discovery part of the web client from Registry Information alone Data Repository is only required to get the numbers Registry information is used to decode the data, replacing ids for human readable labels in the specified language

Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository External Clients And Organisations Data Discovery Data Query Make use of existing libraries and connectors WEB SERVICE Oracle Database Internal Clients And Processes

Harmonising Metadata Fusion Registry Oracle Database Data Repository Metadata Repository Internal Clients And Processes Data Discovery Data Query Make use of existing libraries and connectors WEB SERVICE FAME SAS Oracle Fusion Matrix

Harmonising Metadata – Summary Fusion Registry Metadata Repository Internal Clients And Processes WEB SERVICE FAME SAS Oracle Fusion Matrix Web Client External Clients And Organisations Data Providers

SDMX Web Service on Non-SDMX Source ? What is the current API to get Data? What is the query syntax? How is the data expressed? Are the coding schemes the same? Can it be expressed as multidimensional cube? What is the response format?

SDMX Web Service on Non-SDMX Source ? Step 1 Decode Query Obtain Metadata from Fusion Registry Decoded URL Dataflow = Bank Indonesia, Gross Domestic Product, Version 1.0 FREQ=A INDICATOR=FF or EE

SDMX Web Service on Non-SDMX Source ? Step 2 Convert Query Convert and execute vendor specific query Converted Query SELECT * FROM xyz WHERE FREQ=A AND IND IN(FFF, EEE) AND ObsDate >= 2008

SDMX Web Service on Non-SDMX Source ? Step 3 Write Response Write Response in SDMX ?

SDMX Web Service on Non-SDMX Source ? DataWriterEngine StartSeries() WriteKey(FREQ,A) WriteKey(INDICATOR,FF) WriteObs(2008, 127.3) WriteAttribute(OBS_CONF, A) WriteObs(2009, 125) WriteAttribute(OBS_CONF, A)

? Obtain Metadata This is a common process, which can be achieved with re-useable module Convert Query This is a database specific action Write Response Call the appropriate methods on an API SDMX Web Service on Non-SDMX Source

SDMX Web Service - Demo Bank of England Bank of England Specific Query Bank of England Specific Response Bank of England Specific Response ? SDMX Query SDMX Response

SDMX Web Service - Demo In order to query this data source we need to describe the data with a Data Structure Definition In order to query this data source we need to describe the data with a Data Structure Definition We find that each series can be identified with a unique SCODE This SCODE is used in the URL API to request series. We could use this as the only Dimension in our Data Structure Definition We find that each series can be identified with a unique SCODE This SCODE is used in the URL API to request series. We could use this as the only Dimension in our Data Structure Definition We can see that there is a lot of metadata surrounding a series. Could these be defined as Dimensions and Attributes? We can see that there is a lot of metadata surrounding a series. Could these be defined as Dimensions and Attributes?

SDMX Web Service - Demo odeVer=new&xml.x=yes&SeriesCodes=RPQB2J4&Datefrom=31/Jan/201 1&Dateto=31/Jan/2012 Proxy Data is for Capital Expenditure Proxy Data is for Capital Expenditure Bank of England Obtain Structures Convert Query Transform Response

SDMX Web Service - Demo

SDMX Web Service - Demo odeVer=new&xml.x=yes&SeriesCodes=RPQB2J4&Datefrom=31/Jan/201 1&Dateto=31/Jan/2012 Proxy Bank of England Fusion Matrix Fusion Registry Register Web Service Notify

SDMX Web Service on Non-SDMX Source ? What is the current API to get Data? What is the query syntax? How is the data expressed? Are the coding schemes the same? Can it be expressed as multidimensional cube? What is the response format?

Structure Map

RPQB2J4 Internal View IADB_KEY RPQB2J5 RPQB2J6 RPQB2J7 External View SECTOR CURRENCY ISTR BOET P P TO BOET P P CH MFIS P P CH BOET C C SV

Structure Map Internal View Series Code External View SECTOR CURRENCY INSTR BOET CGOV TO CH P P SV RPQB2J4 RPQB2J5

Structure Map Internal View Series Code External View SECTOR CURRENCY INSTR BOET CGOV TO CH P P SV RPQB2J4 RPQB2J5

Map dataset to METATECH, CAPEX,1.0 Structure Map ? Flow=CAPEX,1.0 SECTOR= BOET Flow=CAPEX,1.0 SECTOR= BOET Fusion Registry Obtain Structure map Flow=CAPEX,1.0 SECTOR= BOET Flow=CAPEX,1.0 SECTOR= BOET RPQB2J4 + RPQB2J5 Response for DF_CAPEX

Structure Map Many to One Relationships One to Many Relationships Many to Many relationships Conditional Relationships Default Values Dropping Dimensions / Attributes

SDMX Web Service on Non-SDMX Source ? What is the current API to get Data? What is the query syntax? How is the data expressed? Are the coding schemes the same? Can it be expressed as multidimensional cube? What is the response format?

Harmonising Metadata – Summary Fusion Registry Metadata Repository Internal Clients And Processes WEB SERVICE FAME SAS Oracle Fusion Matrix Web Client External Clients And Organisations Data Providers

Web Client Categorised list of “Topics” Indication of series count per “Topic” Data start and End dates for topic, Last Updated date Data Query Builder List of all available series for topic Series picker Feedback series count based on current selection Prevent large queries on client & on server Store queries, share queries Promote queries Free Text Search Tabulation Graphing Data Discovery & Query from within Excel Simple Calculations Link in to additional “Reference Metadata” Notify Updates (RSS/ ) Advanced Release Calendar Feedback valid selections based on current selection

Web Client Categorised list of “Topics” Indication of series count per “Topic” Data start and End dates for topic, Last Updated date Data Query Builder List of all available series for topic Series picker Feedback series count based on current selection Prevent large queries on client & on server Store queries, share queries Promote queries Free Text Search Tabulation Graphing Data Discovery & Query from within Excel Simple Calculations Link in to additional “Reference Metadata” Notify Updates (RSS/ ) Advanced Release Calendar Feedback valid selections based on current selection

Web Client Series Count Concept Labels Dataflows Category Tree Multi-lingual

Web Client Fusion Registry Fusion Matrix External Data Source Custom Web Service SDMX Data Web Service SDMX Structure Web Service

Web Client Fusion Registry Fusion Matrix Custom Web Service Categories Topic Query Build SDMX Data

Web Client - Summary Fusion Registry Fusion Matrix Custom Web Service JavaScript Libraries BIS Styling BIS Styling ECB Styling ECB Styling Geography

Harmonising Metadata Database Web Client External Clients And Organisations Internal Clients And Processes Programmers Data Providers Submit Data Fusion Registry

Harmonising Metadata Data Store(s) Web Client External Clients Internal Clients And Processes Data Providers Fusion Registry Structural Metadata Store Fusion Matrix Middleware SDMX Structure Web Service SDMX Data Web Service Custom Web Service JavaScript Libraries

Summary Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Question Where is the Reference Metadata? Question Where is the Reference Metadata?

Summary Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Question Where is the Reference Metadata? Question Where is the Reference Metadata?

Reference Metadata

Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Question Where is the Reference Metadata? Question Where is the Reference Metadata? Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Describe Metadata Reports Metadata Structure Definition Describe Metadata Reports Metadata Structure Definition Structural Metadata Repository

Reference Metadata Database Fusion Registry Data Repository Metadata Repository Structural Metadata Repository Fusion Matrix Data or Structure Query Enrich Response with links to Reference Metadata Enrich Response with links to Reference Metadata

Harmonising Metadata Summary Data Store(s) Web Client External Clients Internal Clients And Processes Data Providers Fusion Registry Structural Metadata Store Fusion Matrix Middleware SDMX Structure Web Service SDMX Data Web Service Custom Web Service JavaScript Libraries Fusion Matrix Middleware SDMX Structure Web Service SDMX Data Web Service Custom Web Service Fusion Matrix Reference Metadata Store Reference Metadata Web Service