ABS Statistical Databases Session 6 Mark Viney Australian Bureau of Statistics 6 June 2007

INPUTTHRUPUTOUTPUT INPUTTHRUPUTOUTPUT INPUTTHRUPUTOUTPUT "Stove Pipe" approach

INPUT THRUPUT Standardised interface Standardised interface INPUT OUTPUT "Clearing-House" Approach Standardised interface OUTPUT IDW ABSIW

e-Census

e-Census 2006  Conducted 2006 Population Census with the option of electronic submission of responses ƒ drop-off/ pick up  drop-off/mail back in 2011  10.2% of returns were electronic ƒ no edits incorporated into electronic form ƒ less visits to pick up paper forms ƒ less paper forms  less scanning/repair

ABS Secure Deposit Box

Secure Deposit Box  An externally facing database to allow respondents to lodge their raw data electronically ƒ Excel spreadsheet (essentailly replacing a paper form) ƒ Administrative datasets

ABS Statistical Databases  ABS Input Data Warehouse (ABS IDW)  ABS Information Warehouse (ABSIW)

ABS Input Data Warehouse (ABS IDW)

Input Data Warehouse  Used as a repository for data as soon as it is entered into ABS computer systems ƒ Initially used for data received electronically ƒ Now used to load (and process) survey data

Input Data Warehouse  Structure ƒ Star schema  1 fact table and several dimension tables  each data cell is stored as 1 row in the fact table

Star Schema

ABS Input Data Warehouse - What it allows us to do  Keep a historical record of what each cell was at every point in the processing ƒ Reason for the change ƒ when it changed ƒ who changed it ƒ change in value  Ready access to both current and historical data

ABS Input Data Warehouse - What it allows us to do  A data store for use with :- ƒ editing ƒ imputation ƒ winsorisation ƒ estimation  Quick easy analysis and confrontation of data:- ƒ across time ƒ across dataitems ƒ across data sources

ABS Input Data Warehouse - Flow of Information

What we hope to achieve from IDW  Reduced costs  Improved data quality  Tools to assist with management of data providers  Better understanding of Editing processes ƒ Significance Editing  One single source of microdata ƒ for all statistical collections  Well managed and secure data storage

ABS Information Warehouse (ABSIW)

ABS Information Warehouse  Need to make both data and metadata:- ƒ Visible ƒ Relatable ƒ Accessible ƒ Understandable ƒ Reliable ƒ Media Independent

ABS Information Warehouse  Visible ƒ central known location  Relatable ƒ across collections  Accessible ƒ tools to allow extraction and manipulation

ABS Information Warehouse  Understandable ƒ data fully described by metadata  Reliable ƒ single source ƒ high availability  Media Independent ƒ single source for outputs  paper publications  electronic releases  ad - hoc requests

ABS Information Warehouse  Define and manage metadata  Load lightly aggregated data  Validate data as compliant with metadata  Manipulate data  Produce statistical outputs  Make data publicly available

ABS Information Warehouse - Flow of information Data from a collection Load info on how to categorize data Load info on what data items mean Load info about collection Load data to the ABSDB Closed DB Sign-off data to the ABSDB Open DB Disseminate output tables Derive ad- hoc client data requests Disseminate time series Processing System Information Warehouse PPW

ABS Information Warehouse - Define and Manage Metadata  Interfaces to manage metadata ƒ load, amend, validate, extract ƒ dataitems,classifications, collections,datasets,publications  Application Program Interfaces (API) to link with other systems/programs ƒ increasingly using XML

ABS Information Warehouse - Loading data  Load data from major sources ƒ Input Data Warehouse ƒ SAS ƒ FAME ƒ SuperCROSS

ABS Information Warehouse - Generating New data Cubes  Passing data through one or more steps to derive a new table ƒ aggregation ƒ drop dataitems ƒ calculate new items

ABS Information Warehouse - Other Manipulations  Seasonal Adjustment ƒ SeasABS (X-11)  Chain Volume Measures ƒ FAME (timeseries)  Supertables  Confidentialisation ƒ Disclosure Avoidance Analysis System

ABS Information Warehouse - Data Delivery  Data combined with metadata  Output formats created tailored to specific use ƒ spreadsheets ƒ timeseries ƒ supertables ƒ paper publications ƒ electronic release

ABS Information Warehouse - Public Release  Make data available on an internally accessible database at a predetermined time (usually 11:30 am Canberra time) ƒ This data is then available to ABS Statistical Consultants to satisfy customer requests  Feed data to website ƒ

ABS Website

National Data Network (NDN)

We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service Australian Bureau of Statistics

National Data Network  Website that raises visibility of statistical data ƒ regardless of publishing agency A national platform for acquiring, sharing and integrating data relevant to policy and research in Australia

National Data Network  One central website ƒ descriptions of data ƒ quality statement ƒ references to other data  Several websites (Nodes) owned and maintained by other agencies

National Data Network

 Current Focus ƒ Publish / Search / Acquire  Future Focus ƒ Design / Capture / Process ƒ Analyse / Report

We assist and encourage informed decision making, research and discussion within governments and the community, by providing leading a high quality, objective and responsive national statistical service Australian Bureau of Statistics

Questions?