1. Transmission and preparation of data from secondary data sources at HCSO

2. Topics Background of the development 
KARÁT - data transmission system
ADAMES - data preparation system 
In my presentation, I’ll focus on 3 major issues:
Background of the development
The Data Transmisiion system, which is called KARAT system
And the data preparation system, which name is ADAMES
So let’s start by the background of the development.

3. Secondary data sources
Data sources at HCSO, 2016
Secondary data sources
Primary data sources

4. Standard process model in HCSO: GSBPM v.4 adaptation - ESTFM

5. Data collection system of the HCSO
Institutional data collections
GÉSA, ELEKTRA
Primary sources
Population surveys
LAKOS
Data
Collection
Before 2015:
no standard solution
Secondary sources
From 2015: KARAT

6. Data preparation at HCSO
Institutional data collections
ADÉL
Primary sources
Population surveys
SAS
Data
Preparation
Before 2015:
no standard solution
Secondary sources
Forthcoming: ADAMES

7. Topics Background of the development 
KARÁT - data transmission system
ADAMES - data preparation system  

8. Main objectives of the development of the KARÁT system
Documentation of secondary sources
Documentation of data transmissions
Secure channels for the transmission of the data sets
Formal control and check of the transmitted data
Loading data into data base
Archiving data
Supporting data providers with proactive functions
Monitoring the data transmissions

9. Metadata for the KARAT system
Data source register of the HCSO and the NSS
Identification of data sources
Description of the main characteristics of data sources
Metadata for the secondary sources - KARAT metadata
Reference periods and deadlines
Type and structure of the datasets
Other files to be transmitted
Special processing tasks related to the datasets
Special attachments to inform the data providers
Message types to be sent to data providers and statisticians

10. Operating environment of KARÁT
System-system type interfaces
System-system type interfaces
External appl.
FOR SENDING TOWARDS HCSO
External appl.
FOR DATA REQUESTS FROM HCSO
External data base
ADAMES
Internal data base
META
Internal KARÁT

11. The procedure of data transfer
Design
ORGANISATION
RECEPTION
DEADLINE CONTROL, REMINDERS
VALIDATION OF DATASETS, Loading INto DATA BASE
Download of data sets
Monitoring of data transfers
Passing to the next phase, Messages

12. Indicators computed by KARAT
Quality indicators on secondary data sources
Indicators on the documentation of secondary data sources
Indicators on the complexity of data transmissions
Indicators on the quality of data transmission processes
Indicators on the accuracy of the data transmission processes
Indicators on the timeliness of data transmissions
Indicators on the respondent burden
Number of datasets and documentation to be transmitted
Channel of data transmission

13. Involvement of data sources in KARÁT
Number of secondary data sources
266
Number of secondary sources involved in KARÁT
213
Number of secondary sources loaded directly into data base 20
Number of data transmission tasks
536
Number of realised data transmissions
402
Rate of realised data transmissions
75%
11 October, 2016

14. Topics Background of the development 
KARÁT - data transmission system
ADAMES - data preparation system   

15. Objectives of the development
Standardised solution for the preparation of data sets transmitted to HCSO via KARÁT (as data base tables)
General tool for statisticians (without the need for a regular IT support)
Standardised communication with other systems of the HCSO

16. Main functions of ADAMES
Data transfer from KARÁT to ADAMES
Validation
Other data preparation tasks
Editing – manual or automatic
Monitoring – quality indicators
Transfer of the output to other IT systems of HCSO
Other data preparation tasks: transformation, coding, definition of new variables…
A rendszerbe külső tábla is beolvasható lehetővé téve az ellenőrzött állomány adatainak összehasonlítását máshonnan származó értékekkel. Az ADAMES-en belül egyszerűbb adat-összekapcsolások is megvalósíthatók

17. Introduction of the new system
January 2015 – September 2016 : test period
September 2016 – September 2017: pilot period
From October 2017 : permanent operation

18. Conclusions KARÁT SYSTEM ADAMES SYSTEM
GENERAL:
for all secondary sources / all data release
METADRIVEN:
attributes described in meta database
general procedures are parameterized by metadata
process driven:
the queue of actions and procedures is regulated
Logged
history management, logging, version management
Proactive
support the partners in data supply
Standardised documentation system
Secure channel
Standardised processes
Monitoring
functions
Standardised data preparation

19. Thank you for your attention!
Ildikó Szűcs
Hungarian Central Statistical Office
Methodology Department