1. Model of transformation administrative data to statistical data Data used in Population and Housing Census 2011 – examples Janusz Dygaszewicz and Paweł Murawski Central Statistical Office POLAND

2. Outline 1. Purpose of the work on administrtive sources 2. Data quality 3. Extract data 4. Transform data 5. Summary

3. Data Owners: Ministry of Finance, Ministry of Interior and Administration, Ministry of Justice, Agricultural Social Insurance Fund, National Health Fund, Agency for Restructuring and Modernisation of Agriculture, Agricultural and Food Quality Inspection, Agency for Geodesy and Cartography, State Fund for Rehabilitation of Disabled Persons, County Offices, Commune Offices, Regional Offices, Telcoms, Energy Suppliers, Office For Foreigners, Social Insurance Institution, Housing Managers, Registers - data acquisition 3

4. Purpose of the work on administrative data Obtaining a sufficiently complete data set – subjective and objective completeness corresponding to classification standards, definitions and basic categories, and thus the effective use of administrative data

5. Data quality -measures- 1. Measuring the quality of administrative registers – timeliness of data – methodological compatibility – completeness – identification standards used in the registry – usefulness – compatibility of data in administrative sources to data obtained in the study/survey 2. Measuring the quality in processing of data registers – excessive coverage error rate – incomplete coverage error rate – subjective indicator of completeness – objective indicator of completeness – imputation rate – data correction index – integration data from various sources index

6. Extract data consolidation data from various source systems; different data format, extract data into the production environment based on the SAS software, converting data into one format that is suitable for processing – SAS tables, validate of imported data structure is an integral part of this process.

7. Extract data -examples-

8. Transform data Data processing in the production environment consisting of: profiling – create a raport on the data quality, unification/standardization of data, parsing (separation) or combining variables, standardization with schemes, conversion, validation, deduplication, data integration.

9. Transform data - profiling-

10. Transform data - standardization and parsing examples- Transform data - standardization and parsing examples-

11. Transform data - schemes examples-

12. Transform data - exemples: report data cleaning - DescriptionBefore cleaningAfter cleaning Group of variablesVariableTotal Inorrect Total Inorrect total incorrect In % total incorect In % Address of permanent residence COMMUNITY43207244284699,92%4316061727971,69% CITY43532092073994,77%4352983430860,99% STREET351415457389916,34%34409321253923,65% PREFIX00-10855100,00% Address of residence COMMUNITY73908810028213,57%738717116661,58% CITY742388306444,13%74233663440,86% STREET60793910272516,90%593370210123,55% PREFIX00-1841600,00% Corresponding address COMMUNITY20051326,59%2005301,50% CITY448791216784,84%44870447961,07% STREET3778496487117,17%374220205755,50% PREFIX00-1119200,00% Personal DataNAME435576472080,17%435575755340,13%

13. Transform data - conversion: gender variables famale male FM12

14. Transform data - conversion: marital status variable- 3 married (M) 503 – married (M) ZNY - married (M) 3 – married (M) 1 – bachelor KWR – bachelor 502 – bachelor 1 bachelor

15. Transform data -validation- checking the data, correcting abnormal values, according to the algorithms prepared by methodologists, eventual exclusion from further processing records which improvement is impossible.

16. Transform data - deduplication - removal of repeated units, requires detailed analisys, including alalysis of legal acts individual for each register, result of deduplication – one record with all the possible and unique information.

17. Transform data -expamle of deduplication process- Transform data -expamle of deduplication process-

18. Transform data -data integration- process of selection of the best, most current and correct value of several or a dozen of registers Used to create a statistical register, which will be available for use by analysts.

19. Transform data -intergation process – scheme- A Register B Register C Register ONE ID MULTIPLE IDENTIFIRES ALTERNATIVE LINKING KEYS LINKINGLINKING SELECTINGSELECTING ALGORYTHMS SELECTING THE BEST VALUES DATA COMPLETENESS STATISTICAL REGISTER

20. kraj_ur_kod_KEP # not null msce_ur_kod_POBYT # not null kraj_ur_kod_GZM # not null Transform data - data integration: example of algorythm Transform data - data integration: example of algorythm FALSE TRUE Kraj_ur_kod select kraj_ur_kod_GZM select kaj_ur_kod_POBYT select kraj_ur_kod_KEP

21. Data integration -example of process- Data integration -example of process-

22. Summary Common difficulties: - poor quality data, missing values, duplicates, - conflicting data, - technical: size of the registers, time-consuming process. Benefits: - obtain relevent, useful, accurate data - improve the quality of the output data. - selection of the best variables from multiple registers,

23. Thank you for your attention www.stat.gov.pl