New procedures for Editing and Imputation of demographic variables G. Bianchi, A. Manzari, A. Pezone, A. Reale, G. Saporito ISTAT.

Slides:



Advertisements
Similar presentations
1 Editing the Integrated Census in Israel. EDITING THE INTEGRATED CENSUS IN ISRAEL Prepared by Eva Rotenberg, Central Bureau of Statistics, Israel (1)
Advertisements

Data Imputation United Nations Statistics Division (UNSD) 16 March 2011 Santiago, Chile.
CHAPTER 1 WHAT IS RESEARCH?.
Ali Husseinzadeh Kashan Spring 2010
CBS-SSB STATISTICS NETHERLANDS – STATISTICS NORWAY Work Session on Statistical Data Editing Ljubljana, Slovenia, 9-11 May 2011 Jeroen Pannekoek and Li-Chun.
Topic 2. DECISION-MAKING TOOLS
Prediction and Imputation in ISEE - Tools for more efficient use of combined data sources Li-Chun Zhang, Statistics Norway Svein Nordbotton, University.
Quality assurance -Population and Housing Census Alma Kondi, INSTAT, Albania.
Maintaining high quality surveys with optimized interviewers replacements : the new French sample monitoring strategy Sébastien Faivre, INSEE, Head of.
1 5.6 No-Standard Formulations  What do you do if your problem formulation doeshave the Standard Form?  What do you do if your problem formulation does.
Dynamic lot sizing and tool management in automated manufacturing systems M. Selim Aktürk, Siraceddin Önen presented by Zümbül Bulut.
1 Chapter 2 Problem Solving Techniques INTRODUCTION 2.2 PROBLEM SOLVING 2.3 USING COMPUTERS IN PROBLEM SOLVING : THE SOFTWARE DEVELOPMENT METHOD.
Creating Research proposal. What is a Marketing or Business Research Proposal? “A plan that offers ideas for conducting research”. “A marketing research.
Edit and Imputation of the 2011 Abu Dhabi Census Glenn Hui and Hanan AlDarmaki Statistics Centre - Abu Dhabi UNECE CES Work Session on Statistical Data.
Joint UNECE/Eurostat Meeting on Population and Housing Censuses (13-15 May 2008) Sample results expected accuracy in the Italian Population and Housing.
Identifying Solutions
FARMS MULTIFUNCTIONALITY AND HOUSEHOLDS INCOMES IN SUSTAINABLE RURAL DEVELOPMENT Session 4: Income and Employment of the Rural Household By Marco Ballin.
READING A PAPER. Basic Parts of a Research Paper 1. Abstract 2. Introduction to Technology (background) 3. Tools & techniques/Methods used in current.
THE MAIN INNOVATIONS OF DATA EDITING AND IMPUTATION FOR THE 2010 ITALIAN AGRICULTURAL CENSUS G. Bianchi, R. M. Lipsi, P. Francescangeli, G. Ruocco, A.
Copyright © 2007 Pearson Education, Inc. Publishing as Pearson Addison-Wesley. Ver Chapter 9: Algorithm Efficiency and Sorting Data Abstraction &
Prime Esperienze di Utilizzo di R all’Interno dell’Istat The Web-Based Information System of Italian Population Census Maura Giacummo Leonardo Tininini.
New and Emerging Methods Maria Garcia and Ton de Waal UN/ECE Work Session on Statistical Data Editing, May 2005, Ottawa.
Topic (ii): New and Emerging Methods Maria Garcia (USA) Jeroen Pannekoek (Netherlands) UNECE Work Session on Statistical Data Editing Paris, France,
Solving Linear Programming Problems: The Simplex Method
1 Dealing with Item Non-response in a Catering Survey Pauli Ollila Statistics Finland Kaija Saarni Finnish Game and Fisheries Research Institute Asmo Honkanen.
Recommended Practices for Editing and Imputation in the European Statistical System: the EDIMBUS Project* Orietta Luzi (Istat, Italy) Ton De Waal (Statistics.
Major objective of this course is: Design and analysis of modern algorithms Different variants Accuracy Efficiency Comparing efficiencies Motivation thinking.
Lesson 3-5: Solving Equations with the Variable on Each Side.
CP Summer School Modelling for Constraint Programming Barbara Smith 2. Implied Constraints, Optimization, Dominance Rules.
Topic (vi): New and Emerging Methods Topic organizer: Maria Garcia (USA) UNECE Work Session on Statistical Data Editing Oslo, Norway, September 2012.
MGT-491 QUANTITATIVE ANALYSIS AND RESEARCH FOR MANAGEMENT OSMAN BIN SAIF Session 16.
Using Hierarchical Reinforcement Learning to Balance Conflicting Sub- problems By: Stephen Robertson Supervisor: Phil Sterne.
Supporting Researchers and Institutions in Exploiting Administrative Databases for Statistical Purposes: Istat’s Strategy G. D’Angiolini, P. De Salvo,
CBS-SSB STATISTICS NETHERLANDS – STATISTICS NORWAY Work Session on Statistical Data Editing Oslo, Norway, September 2012 Jeroen Pannekoek and Li-Chun.
Statistical Expertise for Sound Decision Making Quality Assurance for Census Data Processing Jean-Michel Durr 28/1/20111Fourth meeting of the TCG - Lubjana.
Lyne Guertin Census Data Processing and Estimation Section Social Survey Methods Division Methodology Branch, Statistics Canada UNECE April 28-30, 2014.
SemiBoost : Boosting for Semi-supervised Learning Pavan Kumar Mallapragada, Student Member, IEEE, Rong Jin, Member, IEEE, Anil K. Jain, Fellow, IEEE, and.
Paolo Valente - UNECE Statistical Division Slide 1 Technology for census data coding, editing and imputation Paolo Valente (UNECE) UNECE Workshop on Census.
Topic (i): Selective editing / macro editing Discussants Orietta Luzi - Italian National Statistical Institute Rudi Seljak - Statistical Office of Slovenia.
Establishing E&I capability and best practices at Statistics NZ Vera Costa & Tracey Savage 2008 UNECE Work Session on Statistical Data Editing.
Evaluating the Quality of Editing and Imputation: the Simulation Approach M. Di Zio, U. Guarnera, O. Luzi, A. Manzari ISTAT – Italian Statistical Institute.
United Nations Workshop on Evaluation and Analysis of Census Data, 1-12 December 2014, Nay Pyi Taw, Myanmar DATA VALIDATION-I Evaluation of editing and.
Problem Reduction So far we have considered search strategies for OR graph. In OR graph, several arcs indicate a variety of ways in which the original.
Collection of Data Jim Bohan
Census Processing Baku Training Module.  Discuss:  Processing Strategies  Processing operations  Quality Assurance for processing  Technology Issues.
Methods and software for editing and imputation: recent advancements at Istat M. Di Zio, U. Guarnera, O. Luzi, A. Manzari ISTAT – Italian Statistical Institute.
OPTIMAL CONNECTIONS: STRENGTH AND DISTANCE IN VALUED GRAPHS Yang, Song and David Knoke RESEARCH QUESTION: How to identify optimal connections, that is,
An Overview of Editing and Imputation Methods for the next Italian Censuses Gianpiero Bianchi, Antonia Manzari, Alessandra Reale UNECE-Eurostat Meeting.
SemiBoost : Boosting for Semi-supervised Learning Pavan Kumar Mallapragada, Student Member, IEEE, Rong Jin, Member, IEEE, Anil K. Jain, Fellow, IEEE, and.
Session topic (i) – Editing Administrative and Census data Discussants Orietta Luzi and Heather Wagstaff UNECE Worksession on Statistical Data Editing.
Advanced Sorting 7 2  9 4   2   4   7
Make or Buy transport.
Optimization Of Robot Motion Planning Using Genetic Algorithm
Task: It is necessary to choose the most suitable variant from some set of objects by those or other criteria.
UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Work Session on Statistical Data Editing April 2017 The Hague,
Classical Waterfall Model
Life Cycle Models PPT By :Dr. R. Mall.
Topics for Research Paper on Human Resource Management
3-3 Optimization with Linear Programming
READING A PAPER.
Warm Up Solve for x:
Albania 2021 Population and Housing Census - Plans
Preliminaries Training Course «Statistical Matching» Rome, 6-8 November 2013 Mauro Scanu Dept. Integration, Quality, Research and Production Networks.
Jeroen Pannekoek, Sander Scholtus and Mark van der Loo
X y y = x2 - 3x Solutions of y = x2 - 3x y x –1 5 –2 –3 6 y = x2-3x.
Automatic Editing with Soft Edits
Multi-Mode Data Collection
Modernization of Social statistics: integrated use of survey and
Chrysostomos Koutsimanis and G´abor Fodor
Presentation transcript:

New procedures for Editing and Imputation of demographic variables G. Bianchi, A. Manzari, A. Pezone, A. Reale, G. Saporito ISTAT

The ISTAT purposes in handling the 2001 Italian Population Census data was providing a complete and consistent set of data by performing plausible imputations and preserving the maximum amount of collected information Adopted strategy: Dividing the E&I problem into simpler sub-problems and finding an appropriate solution for each of them The overall E&I process consists of several procedures addressing specific E&I problems and implementing different E&I methods. The aim of this strategy is to improve the quality of final results because each problem is solved by a suitable tool In the paper three new procedures are presented New procedures for Editing and Imputation of demographic variables

The first procedure has been developed to deal with problems occurring when connected subsets of variables are handled in sequential E&I steps An approach suggested by the graph theory has been used, consisting in performing E&I of variables handled in the first step taking into account the information provided by variables treated in the second step The procedure consists of three main phases: A.Location of the variable (pivot) involved in the highest number of connections among the subsets. The pivot variable is edited in the first step B.Definition of a new auxiliary variable, the Subset of Admissible Values (SAV) of the pivot variable, identifying the values of the pivot variable that are as much consistent as possible with the information provided by the subset of variables that will be edited in the second step C.Performing the E&I of the pivot variable using its SAV

The second procedure aims at locating the household reference person (Person 1) when: one person has declared to be the Person 1 but his Year of birth is not consistent with such a role (17 years old or younger) or it is missing either more than one person or no person has declared to be the Person 1 New procedures for Editing and Imputation of demographic variables The approach used is based on optimization techniques and has been carried out by adapting the first fields then donors algorithm implemented in the DIESIS system to the specific problem The procedure assigns the Person 1 role to the person which allows the minimum change of the demographic variables values to restore the household consistency

The third procedure is concerned with the treatment of invalid or inconsistent responses for the demographic variables The demographic variables have been processed by the DIESIS system using the first donors then fields (data driven approach) and the first fields then donors (minimum change approach) algorithms The data driven approach has been selected as default with the option to turn to the minimum change approach when, for a given failed edit household, the number of changes proposed by the data driven approach was exceedingly high, compared to the number of changes proposed by the minimum change approach The two algorithms have been jointly used in order to balance the plausibility of the imputation actions with the preservation of the collected information. New procedures for Editing and Imputation of demographic variables