Presentation is loading. Please wait.

Presentation is loading. Please wait.

METADATA HARMONISATION SDMX Training BANK INDONESIA 16-18 SEPTEMBER 2015 YOGYAKARTA, INDONESIA.

Published byAron Hamilton Modified over 8 years ago

Similar presentations

Presentation on theme: "METADATA HARMONISATION SDMX Training BANK INDONESIA 16-18 SEPTEMBER 2015 YOGYAKARTA, INDONESIA."— Presentation transcript:

1 METADATA HARMONISATION SDMX Training BANK INDONESIA 16-18 SEPTEMBER 2015 YOGYAKARTA, INDONESIA

2 Harmonising Metadata Database Web Client External Clients And Organisations Internal Clients And Processes Programmers Data Providers Submit Data Fusion Registry

3 Harmonising Metadata Database Observations Observation Id Value Status Confidentiality … Observations Observation Id Value Status Confidentiality … Dimension Dimension Id Classification Id Dimension Title … Dimension Dimension Id Classification Id Dimension Title … Classification Classification Id Codelist Id Title Description Version Operative Date … Classification Classification Id Codelist Id Title Description Version Operative Date … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dimension Item Id Dimension Id Classification Item Id Display Order ….. Dimension Item Id Dimension Id Classification Item Id Display Order ….. 30+ More Tables

4 Harmonising Metadata Database Observations Observation Id Value Status Confidentiality … Observations Observation Id Value Status Confidentiality … Dimension Dimension Id Classification Id Dimension Title … Dimension Dimension Id Classification Id Dimension Title … Classification Classification Id Codelist Id Title Description Version Operative Date … Classification Classification Id Codelist Id Title Description Version Operative Date … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dataset Definition Id Collection Id Dataset Title Description Reference Period … Dimension Item Id Dimension Id Classification Item Id Display Order ….. Dimension Item Id Dimension Id Classification Item Id Display Order ….. 30+ More Tables Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value

5 Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations

6 Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository Obtain information about Datasets they must provide. Obtain rules on which series they must supply as a provider Data Providers Data Validation can be performed based on Registry metadata. It can be a central process. Not tied to the target database platform

7 Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository Web Client Build the entire data discovery part of the web client from Registry Information alone Data Repository is only required to get the numbers Registry information is used to decode the data, replacing ids for human readable labels in the specified language

8 Harmonising Metadata Database Fusion Registry Data Repository Metadata Repository External Clients And Organisations Data Discovery Data Query Make use of existing libraries and connectors WEB SERVICE Oracle Database Internal Clients And Processes

9 Harmonising Metadata Fusion Registry Oracle Database Data Repository Metadata Repository Internal Clients And Processes Data Discovery Data Query Make use of existing libraries and connectors WEB SERVICE FAME SAS Oracle Fusion Matrix

10 Harmonising Metadata – Summary Fusion Registry Metadata Repository Internal Clients And Processes WEB SERVICE FAME SAS Oracle Fusion Matrix Web Client External Clients And Organisations Data Providers

11 SDMX Web Service on Non-SDMX Source ? What is the current API to get Data? What is the query syntax? How is the data expressed? Are the coding schemes the same? Can it be expressed as multidimensional cube? What is the response format?

12 SDMX Web Service on Non-SDMX Source ? http://myserver/data/BI,GDP,1.0/A..FF+EE?startPeriod=2008 Step 1 Decode Query Obtain Metadata from Fusion Registry Decoded URL Dataflow = Bank Indonesia, Gross Domestic Product, Version 1.0 FREQ=A INDICATOR=FF or EE

13 SDMX Web Service on Non-SDMX Source ? http://myserver/data/BI,GDP,1.0/A..FF+EE?startPeriod=2008 Step 2 Convert Query Convert and execute vendor specific query Converted Query SELECT * FROM xyz WHERE FREQ=A AND IND IN(FFF, EEE) AND ObsDate >= 2008

14 SDMX Web Service on Non-SDMX Source ? http://myserver/data/BI,GDP,1.0/A..FF+EE?startPeriod=2008 Step 3 Write Response Write Response in SDMX ?

15 SDMX Web Service on Non-SDMX Source ? http://myserver/data/BI,GDP,1.0/A..FF+EE?startPeriod=2008 DataWriterEngine StartSeries() WriteKey(FREQ,A) WriteKey(INDICATOR,FF) WriteObs(2008, 127.3) WriteAttribute(OBS_CONF, A) WriteObs(2009, 125) WriteAttribute(OBS_CONF, A)

16 ? http://myserver/data/BI,GDP,1.0/A..FF+EE?startPeriod=2008 Obtain Metadata This is a common process, which can be achieved with re-useable module Convert Query This is a database specific action Write Response Call the appropriate methods on an API SDMX Web Service on Non-SDMX Source

17 SDMX Web Service - Demo Bank of England Bank of England Specific Query Bank of England Specific Response Bank of England Specific Response ? SDMX Query SDMX Response

18 SDMX Web Service - Demo In order to query this data source we need to describe the data with a Data Structure Definition In order to query this data source we need to describe the data with a Data Structure Definition We find that each series can be identified with a unique SCODE This SCODE is used in the URL API to request series. We could use this as the only Dimension in our Data Structure Definition We find that each series can be identified with a unique SCODE This SCODE is used in the URL API to request series. We could use this as the only Dimension in our Data Structure Definition We can see that there is a lot of metadata surrounding a series. Could these be defined as Dimensions and Attributes? We can see that there is a lot of metadata surrounding a series. Could these be defined as Dimensions and Attributes?

19 http://52.88.213.237/BOE/ws/rest/data/BOE,DF_CAPEX,1.0/RPQB2J4 SDMX Web Service - Demo http://www.bankofengland.co.uk/boeapps/iadb/fromshowcolumns.asp?C odeVer=new&xml.x=yes&SeriesCodes=RPQB2J4&Datefrom=31/Jan/201 1&Dateto=31/Jan/2012 Proxy Data is for Capital Expenditure Proxy Data is for Capital Expenditure Bank of England Obtain Structures Convert Query Transform Response

20 SDMX Web Service - Demo

21

22 http://52.88.213.237/BOE/ws/rest/data/BOE,DF_CAPEX,1.0/RPQB2J4 SDMX Web Service - Demo http://www.bankofengland.co.uk/boeapps/iadb/fromshowcolumns.asp?C odeVer=new&xml.x=yes&SeriesCodes=RPQB2J4&Datefrom=31/Jan/201 1&Dateto=31/Jan/2012 Proxy Bank of England Fusion Matrix Fusion Registry Register Web Service Notify

23 SDMX Web Service on Non-SDMX Source ? What is the current API to get Data? What is the query syntax? How is the data expressed? Are the coding schemes the same? Can it be expressed as multidimensional cube? What is the response format?

24 Structure Map

25 RPQB2J4 Internal View IADB_KEY RPQB2J5 RPQB2J6 RPQB2J7 External View SECTOR CURRENCY ISTR BOET P P TO BOET P P CH MFIS P P CH BOET C C SV

26 Structure Map Internal View Series Code External View SECTOR CURRENCY INSTR BOET CGOV TO CH P P SV RPQB2J4 RPQB2J5

27 Structure Map Internal View Series Code External View SECTOR CURRENCY INSTR BOET CGOV TO CH P P SV RPQB2J4 RPQB2J5

28 Map dataset to METATECH, CAPEX,1.0 Structure Map ? Flow=CAPEX,1.0 SECTOR= BOET Flow=CAPEX,1.0 SECTOR= BOET http://myserver/data/BI,GDP,1.0/BOET... Fusion Registry Obtain Structure map Flow=CAPEX,1.0 SECTOR= BOET Flow=CAPEX,1.0 SECTOR= BOET http://myinternal/data/BOE,DF_CAPEX,1.0/ RPQB2J4 + RPQB2J5 Response for DF_CAPEX

29 Structure Map Many to One Relationships One to Many Relationships Many to Many relationships Conditional Relationships Default Values Dropping Dimensions / Attributes

30 SDMX Web Service on Non-SDMX Source ? What is the current API to get Data? What is the query syntax? How is the data expressed? Are the coding schemes the same? Can it be expressed as multidimensional cube? What is the response format?

31 Harmonising Metadata – Summary Fusion Registry Metadata Repository Internal Clients And Processes WEB SERVICE FAME SAS Oracle Fusion Matrix Web Client External Clients And Organisations Data Providers

32 Web Client Categorised list of “Topics” Indication of series count per “Topic” Data start and End dates for topic, Last Updated date Data Query Builder List of all available series for topic Series picker Feedback series count based on current selection Prevent large queries on client & on server Store queries, share queries Promote queries Free Text Search Tabulation Graphing Data Discovery & Query from within Excel Simple Calculations Link in to additional “Reference Metadata” Notify Updates (RSS/Email) Advanced Release Calendar Feedback valid selections based on current selection

33 Web Client Categorised list of “Topics” Indication of series count per “Topic” Data start and End dates for topic, Last Updated date Data Query Builder List of all available series for topic Series picker Feedback series count based on current selection Prevent large queries on client & on server Store queries, share queries Promote queries Free Text Search Tabulation Graphing Data Discovery & Query from within Excel Simple Calculations Link in to additional “Reference Metadata” Notify Updates (RSS/Email) Advanced Release Calendar Feedback valid selections based on current selection

34 Web Client Series Count Concept Labels Dataflows Category Tree Multi-lingual

35 Web Client Fusion Registry Fusion Matrix External Data Source Custom Web Service SDMX Data Web Service SDMX Structure Web Service

36 Web Client Fusion Registry Fusion Matrix Custom Web Service Categories Topic Query Build SDMX Data

37 Web Client - Summary Fusion Registry Fusion Matrix Custom Web Service JavaScript Libraries BIS Styling BIS Styling ECB Styling ECB Styling Geography

38 Harmonising Metadata Database Web Client External Clients And Organisations Internal Clients And Processes Programmers Data Providers Submit Data Fusion Registry

39 Harmonising Metadata Data Store(s) Web Client External Clients Internal Clients And Processes Data Providers Fusion Registry Structural Metadata Store Fusion Matrix Middleware SDMX Structure Web Service SDMX Data Web Service Custom Web Service JavaScript Libraries

40 Summary Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Question Where is the Reference Metadata? Question Where is the Reference Metadata?

41 Summary Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Question Where is the Reference Metadata? Question Where is the Reference Metadata?

42 Reference Metadata

43

44

45 Database Fusion Registry Data Repository Metadata Repository Observations Dimension Ids Attribute Ids Time Value Observations Dimension Ids Attribute Ids Time Value Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Question Where is the Reference Metadata? Question Where is the Reference Metadata? Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Datasets Dataflow Data Structures Code Lists Concepts Describe Collection Data Providers Provision Agreements Constraints Describe Collection Data Providers Provision Agreements Constraints Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Dissemination Categories Hierarchical Codelists Structure Maps Data Registrations Describe Metadata Reports Metadata Structure Definition Describe Metadata Reports Metadata Structure Definition Structural Metadata Repository

46 Reference Metadata Database Fusion Registry Data Repository Metadata Repository Structural Metadata Repository Fusion Matrix Data or Structure Query Enrich Response with links to Reference Metadata Enrich Response with links to Reference Metadata

47 Harmonising Metadata Summary Data Store(s) Web Client External Clients Internal Clients And Processes Data Providers Fusion Registry Structural Metadata Store Fusion Matrix Middleware SDMX Structure Web Service SDMX Data Web Service Custom Web Service JavaScript Libraries Fusion Matrix Middleware SDMX Structure Web Service SDMX Data Web Service Custom Web Service Fusion Matrix Reference Metadata Store Reference Metadata Web Service

Download ppt "METADATA HARMONISATION SDMX Training BANK INDONESIA 16-18 SEPTEMBER 2015 YOGYAKARTA, INDONESIA."

Similar presentations

The use of SDMX at the ECB Xavier Sosnovsky European Central Bank Bonn, 2009-04-02.

The use of SDMX at the ECB Xavier Sosnovsky European Central Bank Bonn,
SDMX in the Vietnam Ministry of Planning and Investment - A Data Model to Manage Metadata and Data ETV2 Component 5 – Facilitating better decision-making.

SDMX in the Vietnam Ministry of Planning and Investment - A Data Model to Manage Metadata and Data ETV2 Component 5 – Facilitating better decision-making.
Federal Department of Home Affairs FDHA Federal Statistical Office FSO Meeting of the OECD Expert Group on SDMX 13 - 14 September, OECD, Paris Centralized.

Federal Department of Home Affairs FDHA Federal Statistical Office FSO Meeting of the OECD Expert Group on SDMX September, OECD, Paris Centralized.
SDMX training session on basic principles, data structure definitions and data file implementation 29 November 2007 2.

SDMX training session on basic principles, data structure definitions and data file implementation 29 November
SDMX Data Structure Definition for BPM6 and EBOPS Working Party on International Trade in Goods and Trade in Services Statistics Paris, France November.

SDMX Data Structure Definition for BPM6 and EBOPS Working Party on International Trade in Goods and Trade in Services Statistics Paris, France November.
© Metadata Technology ESCWA SDMX Workshop Session: SDMX Registry Registration of a Data Set.

© Metadata Technology ESCWA SDMX Workshop Session: SDMX Registry Registration of a Data Set.
SDMX data discovery, query, and visualisation within Excel

SDMX data discovery, query, and visualisation within Excel
Environment Change Information Request Change Definition has subtype of Business Case based upon ConceptPopulation Gives context for Statistical Program.

Environment Change Information Request Change Definition has subtype of Business Case based upon ConceptPopulation Gives context for Statistical Program.
Agenda Item 3.3 SDMX reference architecture for NSIs Francesco Rizzo 24 th Meeting of the STNE Working Group “Statistics, Telematic Network & EDI” 16-17.

Agenda Item 3.3 SDMX reference architecture for NSIs Francesco Rizzo 24 th Meeting of the STNE Working Group “Statistics, Telematic Network & EDI”
WP.5 - DDI-SDMX Integration E.S.S. cross-cutting project on Information Models and Standards Marco Pellegrino, Denis Grofils Eurostat METIS Work Session6-8.

WP.5 - DDI-SDMX Integration E.S.S. cross-cutting project on Information Models and Standards Marco Pellegrino, Denis Grofils Eurostat METIS Work Session6-8.
Survey Data Management and Combined use of DDI and SDMX DDI and SDMX use case Labor Force Statistics.

Survey Data Management and Combined use of DDI and SDMX DDI and SDMX use case Labor Force Statistics.
Using the SAS® Information Delivery Portal

Using the SAS® Information Delivery Portal
Using XML technologies to implement complex tables in short- term statistics Francesco Rizzo

Using XML technologies to implement complex tables in short- term statistics Francesco Rizzo
Sdmx web services Strutural data

Sdmx web services Strutural data
DATA PORTAL SDMX Training BANK INDONESIA 16-18 SEPTEMBER 2015 YOGYAKARTA, INDONESIA.

DATA PORTAL SDMX Training BANK INDONESIA SEPTEMBER 2015 YOGYAKARTA, INDONESIA.
Restricted Views expressed are those of the presenter and not necessarily those of the BIS 1 Building a Data Portal with SDMX The BIS SDMX Sandbox exercise.

Restricted Views expressed are those of the presenter and not necessarily those of the BIS 1 Building a Data Portal with SDMX The BIS SDMX Sandbox exercise.
CountryData Development Improving the collation, availability and dissemination of development indicators (including the MDGs) Nairobi, 27 November 2013.

CountryData Development Improving the collation, availability and dissemination of development indicators (including the MDGs) Nairobi, 27 November 2013.
SDMX AND DATA DISSEMINATION SDMX Training BANK INDONESIA 16-18 SEPTEMBER 2015 YOGYAKARTA, INDONESIA.

SDMX AND DATA DISSEMINATION SDMX Training BANK INDONESIA SEPTEMBER 2015 YOGYAKARTA, INDONESIA.
Restricted 13/14 September 2012 1 Building a Data Portal with SDMX The BIS SDMX Sandbox exercise 1 Gabriele Becker, Massimo Bruschi Bank for International.

Restricted 13/14 September Building a Data Portal with SDMX The BIS SDMX Sandbox exercise 1 Gabriele Becker, Massimo Bruschi Bank for International.
Restricted Daejeon, 26-29 April 2010 1 An SDMX based unified data catalogue (UDC) MSIS – Meeting on the Management of Statistical Information Systems 1.

Restricted Daejeon, April An SDMX based unified data catalogue (UDC) MSIS – Meeting on the Management of Statistical Information Systems 1.

Similar presentations

Ads by Google