ABS: Why and How we use SDMX Graeme Oakley Australian Bureau of Statistics "Towards Implementation of SDMX - Jan 9-11, 2007"

Content of Presentation  Introduction - objectives, overview of ABS  Principles relevant to ABS data and metadata dissemination  Case studies - one national, one international  Vision, Lessons, Issues "Towards Implementation of SDMX - Jan 9-11, 2007"

Objectives of Presentation  Why SDMX fits with ABS philosophy concerning principles and standards  Tell our stories about a national and international experience using SDMX  Discuss the future vision, concerns and lessons

Australian Bureau of Statistics  Australia's national statistical agency, statistical leadership role in National Statistical Service  ABS Mission  ".. assist and encourage informed decision- making, research and discussion.."  A broad range of statistics on Australian society  Economic, Demographic, Social, Health, Tourism, Culture and Leisure, Environment,...  Central and 8 Regional offices, about 2750 staff (FTE) plus 800 interviewers "Towards Implementation of SDMX - Jan 9-11, 2007"

ABS Principles  Relevance: good planning with a good understanding of current and future needs of users; timely and relatable statistics  Integrity: analysis and interpretation is objective; always publish our statistics; explain and inform; methodologies and approaches open to scrutiny  Access for all: strive for accurate and widespread media coverage; access through range of products and services "Towards Implementation of SDMX - Jan 9-11, 2007"

ABS Principles (ctd)  Professionalism: exercise highest professional standards; staff highly trained; development of statistical standards; open about the quality of our statistics  Trust of providers: encouraged to provide accurate information; data is strictly protected; explain why information is collected; every ABS officer required to give an undertaking of secrecy Source: ABS Corporate Plan Source: ABS Corporate Plan "Towards Implementation of SDMX - Jan 9-11, 2007"

ABS - Clients  Australia's national statistical agency serving:  Australian Government  State/Territory Governments  Local Governments  International statistical organisations  Business community  Education, religious, welfare, etc organisations  Media  Individual members of society "Towards Implementation of SDMX - Jan 9-11, 2007"

Principles and Standards Approach  Data Management - data is visible, accessible, understandable, reliable, relatable, presentable  Metadata - definitions (understanding), discovery, quality ('fitness for purpose')  ABS Enterprise Architecture - follow mainstream standards, collaborate & partner where effective, promote development of low complexity & maintainable solutions, remove dependence on vendor formats, extensible and scalable systems, use emerging technology as appropriate "Towards Implementation of SDMX - Jan 9-11, 2007"

Standards (ctd) "Towards Implementation of SDMX - Jan 9-11, 2007"  Dissemination -  all publishable data in ABS Information Warehouse  metadata associated with products  web site is prime dissemination channel  clients can 'pull' data for free  data stored once but published in many common formats  Statistical - use international standards eg ICD, SITC; and frameworks eg SNA

Case Studies  International - NAWWE with OECD (Note: ABS responds to many information requests from international organisations, in variety of formats - we are seeking an efficient solution that aligns with our principles and standards)  National - Reserve Bank of Australia (RBA) [A large, sophisticated and important user of ABS data. There are a number of other users of this type eg Treasury.] "Towards Implementation of SDMX - Jan 9-11, 2007"

International - OECD NAWWE  Involvement in pilot project since 2002/03  Tasks include:  Set up metadata and load National Accounts data to ABS Information Warehouse to suit agreed NAWWE Data Structure Definition  Extract from ABSIW and transform to agreed SDMX message structure (XML format)  Place on ABS website (recently decommissioned web page for original case study and yet to put up one for current study) "Towards Implementation of SDMX - Jan 9-11, 2007"

OECD FAME Time Series DB ABS Information Warehouse ABS Statistical Products EXCEL File Hand entered Current Process "Towards Implementation of SDMX - Jan 9-11, 2007"

OECD FAME Time Series DB ABS Information Warehouse ABS Statistical Products Time Series XML Transform to SDMX Publish to Future Process "Towards Implementation of SDMX - Jan 9-11, 2007"

ABS Experience  Changed business process required  Although supporting the initiative, ABS National Accounts group could not always provide resources to do necessary setup work  Long periods on inactivity whilst OECD worked on new Data Structure Definition  Experimental SDMX V1 files were delivered  Now have SDMX V2 DSD to understand and implement "Towards Implementation of SDMX - Jan 9-11, 2007"

National - RBA "Towards Implementation of SDMX - Jan 9-11, 2007"  Central Bank for Australia (RBA) - receives mainly time series data from many sources  Many formats and different frequencies  Previously maintained their time series data in an EXCEL based spreadsheet system  In past year, developed new system to centralise time series data storage and retrieval, based on SQL server

National - RBA (ctd)  ABS provided detailed Labour Force data in SDMX V1 format as a trial (RBA currently takes LFS data in a large number of EXCEL files and some multidimensional cubes)  The RBA's new system has been demonstrated to internal business areas, and accepted  RBA want to extend use of SDMX to other ABS series and improve the timeliness "Towards Implementation of SDMX - Jan 9-11, 2007"

ABS RBA Labour Force data in SDMX format SQL Server DB holding time series data XSLT SQL Server Integration Services (SSIS) with file attached What RBA do with SDMX files "Towards Implementation of SDMX - Jan 9-11, 2007" RBA economists

What RBA see as benefits?  Efficient data transfer - reduces need for clerical intervention when dealing with spreadsheets, can easily manipulate with programs  Metadata rich - SDMX allows more metadata to be conveyed compared to spreadsheets  Able to handle changes in time series metadata and data more easily - can programmatically check metadata for changes "Towards Implementation of SDMX - Jan 9-11, 2007"

What does RBA want ABS to change? "Towards Implementation of SDMX - Jan 9-11, 2007"  Nothing related to the transfer format - SDMX  Want the SDMX files on ABS web site so can download at 11.30am (embargo time) rather than wait for  Want ABS to extend use of SDMX to other statistical series  Additional information related to identification of discontinued series, code sets

Vision "Towards Implementation of SDMX - Jan 9-11, 2007"  Vision - framework and standard that ABS can use for all our international reporting; and also to use with large major national users of ABS data. SDMX is suitable as the primary dissemination vehicle for machine readable aggregate and time series datasets for both national and international clients. The SDMX format files will become ABS data products held on the web site for free access.

Lessons  SDMX has the potential to provide integrated data exchange across organisational boundaries. Challenge is not the SDMX 'technology', but dealing with 'meaning' and 'mapping' issues  Recognise costs involved in implementation, especially mapping local metadata and data structures to international requirement in each subject domain, and the importance of engaging local subject matter experts "Towards Implementation of SDMX - Jan 9-11, 2007"

Issues  Use SDMX for wide range of users, from technically competent to small data integrator. A short shallow learning curve with plenty of support, such as easy documentation, examples, tutorials, and tools  Simple time series identifier, in addition to the Data Structure Definition - eg local identifier as part of exchange - the key back to internal data holdings; existing international 'code' for each series used in current exchanges - doesn't vary from country to country.

Conclusion "Towards Implementation of SDMX - Jan 9-11, 2007"  Move to SDMX V2, recognising need to keep back-end systems independent of version changes.  Importance of establishing business benefit of SDMX - it is not just a technology.  Ideal of a common framework that would guide each statistical domain to determine their exchange requirements and definitions.  Need for good registration processes around all aspects of SDMX, and involvement of NSO's in governance.

QUESTIONS AND DISCUSSION "Towards Implementation of SDMX - Jan 9-11, 2007"