Data Processing of the 2010 Population and Housing Census September 2008, Bangkok, Thailand National Statistical Office, Thailand

 Hardware & Software of ICR System  TELEform / ABBYY Functions  Step of ICR System in NSO  Specific questionnaires for ICR System CONTENT DATA CAPTURING

NSO was firstly used ICR System to process the Population Census questionnaires in 2000 by scanning the 16 million households (16 million Forms) which spent only 8 months to process the raw data instead of 18 months by using Key in Data System. ICR for The Population Census 2000 DATA CAPTURING

TELEform Hardware & Software System in 2000 TELEform Hardware System TELEform Software System  NetServer for TELEform Server (1)  NetServer for Database Server (1)  Reader Modules Workstations (21)  Verifier Modules Workstations (55)  Scanner Control Workstations (6)  Scanner Fujitsu M4099D (6) TELEform 6.2 Elite Enterprise Edition Components :  TELEform Designer  TELEform Reader  TELEform Verifier DATA CAPTURING

ICR System in NSO (Thailand) can be divided into 2 parts : ICR System in 2003  TELEform Software System  ABBYY Software System DATA CAPTURING

NSO hired ABBYY Software to process about 25% of The Agricultural Census 2003 questionnaires that were totally 5.8 million households (24 million forms). ICR for The Agricultural Census 2003 DATA CAPTURING

TELEform Hardware & Software System in 2003 TELEform Hardware System TELEform Software System TELEform 6.2 Elite Enterprise Edition Components :  TELEform Designer  TELEform Reader  TELEform Verifier  NetServer for TELEform Server (1)  NetServer for Database Server (1)  Reader Modules Workstations (21)  Verifier Modules Workstations (30)  Scanner Control Workstations (6)  Scanner Fujitsu M4099D (6) DATA CAPTURING

ABBYY Hardware & Software System in 2003 ABBYY Hardware System ABBYY Software System ABBYY FormReader 6.0 Enterprise Edition Components:  Form Design  Administration Station  Recognition Station  Correction Station  IBM Server X Series 225 (1)  Correction Station (1)  Verifier Modules Workstations (25)  Scanner Control Workstations (4)  Scanner Fujitsu M4099D (4)  Storageflex LT707 (1) DATA CAPTURING

TELEform & ABBYY Functions

TELEform / ABBYY Designer Function To create template form by fix field boxes on questionnaire. Questionnaire Template DATA CAPTURING

 To evaluate the questionnaires  Export the corrected questionnaires to a data file  Send the unclear questionnaires to TELEform/ABBYY Verifier Function for correcting and transferring the corrected questionnaires to a data file  Store scanned images TELEform Reader / ABBYY Administration Function DATA CAPTURING

 To correct questionnaires that contain mismarked or illegible fields  The corrected questionnaires are automatically exported to a data file TELEform / ABBYY Verifier Function DATA CAPTURING

Scanning speed support A7 to A3 paper sizes  Simplex is provided 90 papers / minute. (A4 portrait)  Duplex is provided 180 images / minute.(A4) NSO questionnaires projects are mostly printed with A3 (297 x 420 mm.) paper sizes. Functions Speed FunctionsEstimated Speed (sheets/minute) Scanner 45 Reader17 Verifier 5 DATA CAPTURING

Step of ICR System in NSO

Scan and Forms Distribution : The questionnaires are scanned in each Block / Village and created Multi Page Image Files. Step of ICR System in NSO Forms Evaluation : The questionnaire images are evaluated. The corrected questionnaires which skipped Verifier Workstations and directly exported to Database server. DATA CAPTURING

Forms Verification : The unclear questionnaires are needed to review and corrected it in Verifier Workstations before transferring to Database server. Step of ICR System in NSO (cont’) Data Export :  Link a data file from Database server to IBM Mainframe System  Store Scanned image files to CD. DATA CAPTURING

Scan & Forms Distribution Questionnaire Scan Image File DATA CAPTURING

Forms Verification and Data Export Verify Storage (Images files) Export data for processing CD DATA CAPTURING

Input – Output of ICR System Questionnaire ICR DATA CAPTURING Ascii files Image files

ICR Linkage System ABBYY Software Transfers Storage (HD 880 GB) Mainframe HP Serve r COMPAQ Server Processing (Editing & Reporting) Scanners 4 unit IBM Serve r controller PC 6 unit PC 4 unit controller 30 unit Verifications 21 unit Readers Correction 1 unit station 25 unit Verifications - Administration - Export - Recognition CD Questionnaires - Backup Data - Software - Database S Scanners 6 unit Questionnaires TELEform Software

DATA CAPTURING Specific Questionnaires for ICR System

Specific questionnaires for ICR System  The questionnaires must be designed and printed in quality of paper, specific colour answer field boxes (blue, green, red)  To record the questionnaires should be used at least 2HB pencil  To distribute and collect as well as return questionnaires should be done with caution. DATA CAPTURING

ICR Benefits  Reduce Cost  Reduce Time  Efficient Data Capture  Increase Data Accuracy DATA CAPTURING

 Strictly designed questionnaires : Paper, size, color, figure and answer field boxes  Record questionnaires should be fixed pencil and handwriting  Distribution and return questionnaires should be carful Major Problems Encountered in 2000 Census DATA CAPTURING

DATA EDITING EDITING & TABULATION

DATA PROCESSING STEP Machine Edit Listing Error? Checking Error & List Data Comparing with questionnaire images Table Checking Yes No Questionnaires ICR Accept? NoYes Validate Data (Manual, Cold deck, Hot deck) Tabulation & Report Tab/Report

 Possible code: Check in each field which the out-of-range fields values is shown in asterisk (*) code.  Validity: Check characteristics of the message structure. DATA PROCESSING STEP Editing Process

 Consistency:  Imputation: Automatic editing programs. DATA PROCESSING STEP Editing Process (cont’) Check inconsistent values within record and across record. Messages are shown the related conditional codes. All error is printed in continuous paper forms to be considerated and validated by subject matter until no messages error found.

 Tabulation: DATA PROCESSING STEP Tabulation Process Report summary data which can be processed after data completely cleaned for subject matter to analyze the results of output.

DATA PROCESSING STEP Mainframe : IBM Multiprise 2000 Model Operating System - OS/390 v.2 release Compiler - PL/I 1.3 Statistic Program - Base SAS 1.4 Application Development Tools - Performance Reporter for OS/390

DATA PROCESSING STEP Personal Computer (PC) 2.1 Operating System - Windows XP 2.2 Package - MS Office MS Studio v.6 (Visual FoxPro) - SPSS - CSPro 3.3

THANK YOU FOR YOUR ATTENTION