On Tap: Developments in Statistical Data Editing at Statistics New Zealand Paper by Allyson Seyb, Felibel Zabala and Les Cochran Presented by Felibel Zabala Sept 2012

Aim of paper To describe the latest developments in Statistics New Zealand’s economic and household processing platforms 2

Strategic Developments Statistics 2020 Te Kāpehu Whetū - Recent actions Reduction in number of tools Use of Colectica to centralise storage of all information of Statistics NZ’s outputs Establishment of processes to research and introduce new standard methods and tools 3

Strategic Developments (cont’d) International collaboration Regular bilateral and trilateral meetings with various national statistical offices Involvement in the Statistical Networks on Confidentiality and the Industrialisation of Editing Continued investigation on the use of SELEKT Evaluation of SAS2Argus 4

Platforms Need for new infrastructure to produce statistics that are fit for purpose in a cost and effective way A platform is a logical cluster of functionality that enables components to be put together to provide a complete end-to-end system 5

System 1 System 2 System 3 System 4 System 5 System 6 System 7 System 8 System 9 System 10 System 1 System 2 System 3 System 4 System 5 etc System 6 System 7 System 8 System 9 System 10 System 11 System 12 System 13 System 14 System 80 etc System 60 System 1 System 2 System 3 System 4 System 5 System 6 System 7 System 8 System 9 System 10 etc System 50 System 1 System 2 System 3 System 4 System 5 System 6 System 7 etc System 25 Use standard tools Use for Surveys and censuses Administrative sourced data Mixed sources Use SAS for processing and analysis 6 Five main platforms Need Develop & Design Build Collect Process Analyse Disseminate Collection Dissemination Micro-economic Household National Accounts

Household platform Is a second generation platform with the design informed by an evaluation of the interim platform Processes and in the future analyse three social surveys and their supplements Uses standard tools to load, code, micro-edit and finalise a unit record dataset Processes a Blaise-based survey 7

Uses a mix of shared and specific systems The Household Platform Portal Process Configure Setup View Code Search Edit View Diary Workflow SAS Execute Save Load Load Diary File Save Diary Select Admin Statistical Toolbox Statistical Toolbox GREGWt X12 Core Edits Core DVs Derivations CANCEIS Edits Screens TasksRules Edit Diary Info Data Metadata Data Metadata Config. Classifications & Stds Metadata Survey Specific Shared across Social Shared across Stats Other Data Configure Paradata Treat Estimation Extract Setup Surv. Inst. Paradata Core Questions

Format of micro-level data 9 The Household platform Survey cycle code IDRepeatVariable name Value HLFS 107 Person111Age15 HLFS 107 Person111SexM HLFS 107 Person211SexF HLFS 107 Household111 Household composition Multi- person

Micro-economic Platform Previously referred to as BESt platform Processes and analyses economic surveys and administrative data collections Also has elements of ‘Develop and Design’, ‘Build’, ‘Collect’ (from general Business Process Model) Has a user configurable workflow Allows incremental statistical maintenance and uses standard tools for common processes 10

Uses a mix of shared and specific systems The Micro-economic platform Business Intelligence View Configure process Metadata Manual edit (generic) Manual edit SAS Load metadata Load data Workflow Statistical Toolbox Statistical Toolbox X12 Edits Derivations Configuration Banff Screens TasksRules Reporting Info Data Metadata Data Metadata Configuration Classifications & Stds Metadata Survey Specific Shared across Micro Economics Shared across Stats Other Data Process trace Run process Extract Trace Reporting Process trace System Admin View Respondent Build cubes Generic E&I Sample select Write-back Imputation Code Estimation Treat Allocate

Challenges and lessons learnt Balancing generic and specific needs Determining process elements suitable to be implemented as common services Moving from a process culture to a more constructive innovative culture Key enabler of the culture change - the adoption of an agile project management approach for IT development projects 12

Moving forward Review of Statistics NZ’s generic Business Process Model Implementation of a framework to measure and report on the benefits achieved with recent and on-going developments Transformation of the organisation’s data collection processes 13