Data and Metadata Session 5 Mark Viney Australian Bureau of Statistics 6 June 2007

What is Data?  Data is a defined, measured quantity  Types of statistical data ƒ Raw data ƒ Microdata ƒ Macrodata  Owners convert data from one type to another by cleaning, editing, imputing and aggregating during the data processing cycle

Raw Data  Data as collected from respondent ƒ It may be:-  incomplete  inconsistent ƒ It may still require:-  cleaning  imputation  follow up with respondent

Microdata  raw data with initial problems removed  data coded to standard classifications  may still contain identification of respondent

Macrodata  Data resulting from the aggregation of microdata  May include new data items:- ƒ totals ƒ averages ƒ percentages ƒ seasonally adjusted/trend data ƒ chain volume indices

 Typically publishable data ƒ does not contain any respondent identification ƒ confidentialised Macrodata

Some Macrodata but what does it mean?

What is Metadata? Metadata can be defined simply as data about data - Bo Sundgren 1973

What is Metadata?  Data that describes ƒ statistical data ƒ describes processes ƒ describes resources and tools used in statistics production  Helps people interpret data  Directs systems to process data

Some Macrodata with Metadata 0.6 GDP (Chain Volume Measure), %Change Sep qtr 06 to Dec qtr 06, Trend, Australia 1.0 GDP (Chain Volume Measure), %Change Sep qtr 06 to Dec qtr 06, Seasonally Adjusted, Australia 1.7 Terms of Trade %Change Sep qtr 06 to Dec qtr 06, Seasonally Adjusted, Australia

Some Macrodata with Metadata

How is metadata used?  tool for comprehension and understanding ƒ provides meaning for numbers  tool for interpretation, facilitate acquisition of new knowledge  help find data and determine its fitness for use  help develop new and improved processes

Types of Metadata  Passive ƒ documentation  Active ƒ used by systems to define the processing rules to produce outputs ƒ can be re-used by several systems

Metadata - applying context to data  Describes attributes of data  Can describe:- ƒ footnotes ƒ Units ƒ Scale/precision ƒ Publication, products ƒ Data users / suppliers ƒ Collection concepts, sources and methods ƒ Form definitions and question texts ƒ Data Item definitions ƒ Quality

Metadata - applying context to data (cont)  Can describe:- ƒ Classifications ƒ processing rules  systems  programs  databases  processes  flows  services  interfaces

When Collected Units Who provided? Concept / meaning Collections Allowable values Who owns the definition Dataitem Dataitem Metadata Time Period Dataitem Metadata

Jan 2004 Years Mark Viney Age (of person) Employment, Health Australian Bureau of Statistics Age Dataitem Metadata 2003/2004 Dataitem Metadata (example)

Question Modules Topics Collection Instruments Populations Data Item Definitions Collections Classific- ations Products Datasets Macrodata & Annotations Data Items Dataset Metadata

Dataset Metadata (example) Approved Building Jobs (from BAPS) etc. Dwellings Housing Area (SLA+) Type of building Type of work Excludes any existing floor area or any part of building not bounded by walls Form (e.g. BACS4) Floor area created by the job (Square metres) Building Activity Collection Floor area commenced during quarter 2344, 17, 5, 165, 360, 165, n.a., n.p. Building Activity: Number, Value by State by...

Metadata Standards  ISO  Dublin Core  SDMX

ISO  Standard structure of metadata repository  Makes metadata accessible, visible and searchable  Provides understanding and reuse of data elements and definitions  System interoperability

SDMX (Statistical Data and Metadata Exchange)  XML based  model to facilitate the exchange of statistical data and metadata ƒ data combined with metadata  Data Cubes / Timeseries

Dublin Core  Developing metadata standards for discovery across domains  Defining frameworks for the interoperation of metadata sets

XBRL - eXtensible Business Reporting Language  XML based  used for reporting of business based data  Standard Business Reporting ƒ possible to produce respondent information direct from business software  reduced provider burden  more standard and consistent reporting from providers

What Metadata helps us achieve  Enforcement of standards to strategic inputs and outputs  Encourage planning and management of statistical activities  Reuse ƒ single source of concept ƒ reduced need to reinvent and manage ƒ reduced costs

What Metadata helps us achieve (continued)  Quality ƒ consistent usage ƒ common dialogue ƒ improved understanding  Flexibility and Productivity  Knowledge Management ƒ consistency ƒ comparability

Combining Data and Metadata Select CODE,LABEL_SEX from CL_SEX; CODE LABEL_SEX ******* ************** 10 Males 20 Females 30 Persons BASE TOTAL *******

Combining Data and Metadata Select CODE,LABEL_STATE from CL_STATE; CODE LABEL_STATE ******* ************** 1 New South Wales 2 Victoria 3 Queensland 4 South Australia 5 Western Australia 6 Tasmania 7 Northern Territory 8 Australian Capital Territory 0 Australia BASE TOTAL *******

Combining Data and Metadata Select * from MD_LABOUR; CODE_SEX CODE_STATE EMPLOYMENT_RATE *************** ******************* ***************************

Combining Data and Metadata SELECT LABEL_SEX,LABEL_STATE,EMPLOYEMNT_RATE FROM CL_SEX,CL_STATE,MD_LABOUR WHERE MD_LABOUR.CODE_SEX = CL_SEX.CODE AND MD_LABOUR.CODE_STATE = CL_STATE.CODE; LABEL_SEX LABEL_STATE EMPLOYMENT_RATE *************** ******************* *************************** MalesTasmania77.3 FemalesTasmania72.1 PersonsTasmania74.0

Using Metadata 10000Total income 2000Other Income 260Income from hiring of equipment 270Cartage and setup 1000Hire Services 140Other construction equipment 10Compaction equipment 20Cranes 30Earthmoving equipment 180Other income from hire services 60Event/exhibition goods and equipment 70Transport equipment

Using Metadata CODE LABEL_STATE ******* ****************** 10Compaction equipment 20Cranes 30Earthmoving equipment 60Event/exhibition goods and equipment 70Transport equipment 140Other construction equipment 180Other income from hire services 260Income from hiring of equipment 270Cartage and setup 1000Hire services 2000Other income 10000Total income BASE,DETAILED,SUBTOTAL,TOTAL

Metadata Driven Systems  These systems use metadata to direct and assist their functions ƒ Active Metadata  In general, this creates a huge advantage and level of flexibility over systems that do not do this.  The metadata may also be external to the system and used for other purposes and systems.

Reuse across systems Metadata

Reuse across systems  Keep one copy of metadata ƒ reduces confusion and ambiguity ƒ reduces opportunities to get it wrong ƒ reduces maintenance ƒ reduces complexity to end user

 invest in metadata and integrated metadata driven systems rather than point solutions  costs will be repaid many times over  avoid duplication as much as possible ƒor automate duplication to retain consistency and integrity Key points

Questions?