1. Tomaž Špeh, Rudi Seljak Statistical Office of the Republic of Slovenia
Developing and using common tools for processing statistical micro data at SORS
Tomaž Špeh, Rudi Seljak
Statistical Office of the Republic of Slovenia

2. Content of the presentation
Problem description
Design and implementation guidelines
Example: common tools for statistical micro data editing
Other common tools for processing statistical data
Current state, future plans and lessons learned
Discussion

3. Problem description
Our statistical IT landscapes are becoming more and more complex, containing systems developed for decades containing multiple vendor components and technologies.
Current and future statistical business needs require flexibility and agility in support of IT.
Statistical micro data processing has always been one of the most important parts of the statistical process; however, due to the complexity of the related methodology, general acceptable solutions are rare.
The main aim is to create a modular platform that enables statisticians to design the process and process statistical micro data.
Current state and experiences gained during developing and using common tools for processing statistical micro data as well as how common tolls are used in the data editing process will be presented.

4. Data processing target workflow
Statistician
Methodologist
Statistician
Statistician
Request for
data processing
Define statistical
methodology
Choose and
connect
building blocks
Process
Repository of
building blocks

5. Design and implementation guidelines
Standardization: solution enforces standards
Flexibility: incremental development, agile maintenance
Reuse: build processes from reusable, well defined and well tested building blocks, layered architecture model
Resources: reduction of resources, less manual work
Reliability: manual work is error prone
Separation of design and execution
Process quality and product quality
Platform/ technology independent solutions
GSBPM aware

6. Streamlining statistical micro data editing processing
AGREGATIONS
DISSEMINATION
PREPARATION OF DISSEMINATION TABLES
KLASJE
Input
database
STATISTICAL SURVEY PLANNING
METADATA EDITING
Statistical
register
Data warehouse
Macro
Electronic
releases
DATA COLLECTION
DATA INTEGRATION
CUSTOMERS
(SLOVENIA,
EU ...)
DATA PREPARATION
STATISTICAL PROCESSING
SOURCES
Macrodata – standard tables customers
Latest data customers
Tailor–made tables
ARCHIVES
OFFICIAL
ARCHIVES 2,3
data and processes
ARCHIVE 1
P O Ž A R N A S T E N A
SCHEME OF TARGET DATA FLOW AT SORS
ARCHIVE 4
Micro
DOCUMENTATION EDITING
Dissemi-
nation
server
METIS
Documentation
According to
templates
SECONDARY
DATA
PRIMARY
Printed
publications
Organizations
International
reporting
Microdata
for
researchers
MICRO
EDITING
SEASONAL
ADJUSTEMENT
AND MACRO
DATA EDITING
USERS
Observation
units
FRAME AND
SAMPLE
CREATION
OF STAT.
REGISTER

8. Creation of the input database
Data integration, followed by the data editing is the crucial part of the process.
The quality of the final data to a large extent depends on this part of the process.
… Missing data
Field survey data X
… Inconsistent data X X
Other SORS data … X X
Admin data X X X

9. Data editing Main steps in the process:
Logical checks for a particular data set
Outlier detection for a particular data source
Corrections and missing data imputation for a particular data source
Data integration and derived variables calculation
Logical checks on integrated data
Additional corrections in the particular data source
Integration

10. Metadata driven, content independent and reusable building blocks
checks
Corrections,
imputations
checks
Corrections,
imputations
Integration
checks
Corrections,
imputations
Integrated
database
Integrated
database
checks
From the implementation point of view

11. Checks – metadata table
Processing

12. Corrections – metadata table
Individual data corrections
Heating costs
Inter-household cash transfer
Systematic data corrections
Apples produced for own consumption

13. Imputations – metadata table
24 different methods with different parameterizations can be used at the moment (hot-deck, regression, logical imputations, etc.)
Mortgage installment
Parameterization

14. Data editing – software application
The application is designed as a Metadata Driven System (MDD).
All the information referring to a specific survey execution are provided through the metadata tables.
Currently the application uses the following software:
SAS + Banff as stored procedures
SAS EG as standard interface
ORACLE (metadata repository)
ORACLE (data storage)
* Plan: Technology and platform independent environment for the automated execution of processes, other development environments

15. Existing MDD tools Logical checks Corrections Imputations Aggregation
Standard error estimation
Tabulation

16. Planed MDD tools Sampling Weighting Calculation of quality indicators
Disclosure control
Macro editing

17. Current state – main advantages
The subject-matter personnel can run the process independently from the IT sector.
All information about the data processing is transparently available through the metadata tables.
The process can be easily adjusted for the different executions of the survey.
Every change of the data in the process is systematically flagged  easier calculation of quality indicators and production of the quality report.

18. Management of the system
Adhoc procedure for preparing input data
Adhoc procedure for preparing output data
input data
output data
Micro data database
Edited data
Production metadata repository
Surveys and instances
Standard SAS application
All the processes are programmed as sas (+Banff) macros.
The user can run and control the processes through stored process interface.

19. Implementation approach
Design & development of editing system
Automated data editing system based on predefined checks (Fellegi, Holt)
Complex implementation and maintenance
Statisticians want more control over edited data
Semi automated system where statisticians control checks and edit rules through interactive metadata
Development of improved system for metadata management
Development of architecture for supporting the execution of statistical business processes
Solutions for visualisation of the workflow

20. Challenges for the future
To improve the procedure of the metadata management; at the moment there is a high risk of syntax errors
Improve the management of the system, especially running of different parts of the process
Additional MDD building blocks should be developed
Platform independent components and integration
Enable separation of process design from process execution

21. Conclusions
Current solution for processing statistical micro data is proven to work satisfactorily for the processing of several micro data collections (Population Census, Agriculture Census, EU SILC, SES).
Development of improved system for metadata management is the next phase.
During the development of the applications other (international) solutions will be considered:
sharing IT tools, building blocks
harmonizing IT infrastructure
Metadata definitions and their exchange