1 DATA CAPTURE – PROCESSING 2006 POPULATION & HOUSING CENSUS OF NIGERIA Presented at UN Regional Workshop on Census Data Processing By Adesola Fatilewa NATIONAL POPULATION COMMISSION At Dar-es-Salaam, Tanzania 9 th -13 th June 2008

2 MAP OF NIGERIA 36 STATES AND FCT ABUJA

3 ABOUT NIGERIA NIGERIA IS THE MOST POPULATED COUNTRY ON THE AFRICAN CONTINENT AND THE 10 th BIGGEST IN THE WORLD. AN AREA OF ABOUT 9.28 MILLION SQ. KMS. POPULATION OF 140.2million BY 2006 CENSUS COMPRISES OF 36 STATES AND FEDERAL CAPITAL TERRITORY 774 LOCAL GOVERNMENT AREAS - LGA (DISTRICTS) DELINEATED INTO OVER 662,000 ENUMERATION AREAS

4 Background Since the late nineties NPopC was being inundated with proposals on various document scanning systems. As at 2005, statements were being made, suggesting that the idea of using scanning technology was utopia.

5 P rocessing Pre-test and Trial Census A scanning system was used to process the second pre-test of April Number of documents processed was about of 100,000 forms as survey covered one local government area (Lga) in each of the 36 States of the country and the Federal Capital Territory. The forms were only optical mark readable and editing was mainly to correct alignment errors.

6 Processing Pre-test and Trial Census Continued Another solution provider supplied five scanners along with two servers for the processing of the Trial Census. Trial Census which took place in April 2005 covered about 5% of the country, which translated to about 10million population. Processing was distributed between two data processing centres (DPCs); Lagos and Kano

7 Lessons learnt staff were identified for suitable roles in data processing of the main census staff gained experience on the new technology alignment and recognition problems detected and rectified decision taken on appropriate archiving system for storage and retrieval of documents need to have various reports to enable management follow progress of processing decision to completely eliminate manual coding and editing

8 Data capture 2006 census Scanning technology was fully deployed in processing Nigeria 2006 Population and Housing Census. This was achieved with 21 scanners distributed in 7DPCs located strategically across the country. Immediately after the census, OMR/ICR forms (questionnaires) used to collect data started arriving at the DPCs. Inventory control was done using an EA tracking system

9 Data capture 2006 census Documents were enveloped by EA, tied in convenient batches and stacked on labelled shelves At the end of the receiving/archiving exercise, batches were retrieved for data capture

10 Paper Preparation before Scanning STORE IN Program Print and add Batch Header cut the paper with cutting machine Otherwise: paper damaged, introduce dirt on the scanned image, reject increased NPC0x Batch Heade r Remove the envelopes Bring the envelopes with the questionnaires from the Archive room Envelope ARCHIVE Jog the paper with the supplied jogger NPC0x Batch Heade r

11 Data Processing Steps at DPCs Schematic diagram Jog Docs Scanner Server Edit Stations

12 Scanner Views Scanner Feeder Questionnaire processing

13 Scanning Sheets loaded on the feeder in batches separated by batch header went through transport system of scanners HR80 SC Scanner speed was 8000 sheets/hr barring jams and other loading difficulties. Scanning was effected by ProScan software and scanned documents were collected at the output tray. The sheets were returned into their envelopes and sent back to archive

14 SC80HC + ProSort + kEOPs NPC0x Batch Header Work Data Storage 8. DVD 5. kEOPs recognition 1. Preparation for Scanning: cut & jogg 3. PaperArchive 3. PaperArchive 4. Data + Images 6. Correction Balancing Archive Data Storage 9. TAPE 7. Export CS Pro 8. Local reports 2.Scanner MANUAL WORK HQ Carto

15 Editing Two levels of Editing: First level at DPC Second level at DVU at NPopC hq. in Abuja

16 First Level Editing XML format stored in SAN on servers networked to scanners Forms in XML loaded onto edit stations The editing system used was called KEOPs and it was designed to check geographic ids against the batch headers, check ‘mandatory fields’ Transactions or whole batches could be passed for ‘balancing correction level’ which was handled by more experienced staff designated ‘Supervisor’,

17 Typical KEOPs Edit Screen

18 EXPORT

19 Second Level Editing Data in ASCII,was encrypted, backup on cds at the DPC and sent to NPopC Hq., Abuja Data is decrypted, validated, collated and further edited at Abuja Data is then checked for completeness to ensure that each delineated EA for any local government had data associated with it CsPro package was then utilized to edit data and aggregate appropriately

20 Second Level Editing Continued Structure checks Range checks Skip pattern checks Inter-record and intra-record consistency checks Imputation methods applied for missing or invalid values: Hot deck’ and ‘Cold deck’ or a combination of both

21 Occupational Coding The only data that was not coded on the field was occupation The occupational coding was effected automatically using a computer-assisted coding system ‘Exceptional Coding’ was applied where coding clerk could not find an appropriate occupation code for an occupation

22 Ensuring that documents for particular geographic locations were archived in sections of the archive and shelves designated for them That all forms were separated before taking them for scanning Breakdown of jogger Rate of getting documents ready for scanning was slower than rate of scanning Difficulty in maintaining belts and fixing them over pulley That correct batch headers were properly placed on EA batches and that after scanning, EAs were correctly returned to their marked envelopes Challenges

23 Challenges Continued Instances of poor field work which resulted in ‘missing values’ of ‘mandatory fields’, outright wrong values for fields Difficulty in linking forms for households of greater than 8 persons Integration of the two solution providers: form design and equipment and software solutions were provided by two different companies Cleaning of blank records of data associated with them at data capture Dealing with sensitivity of Nigerians to census figures lack of reliable and uninterrupted power supply

24 Conclusion The Commission was proud that the decision to deploy a new technology for part of the processing of Nigeria 2006 Population and Housing Census was a success About 35million forms were scanned and edited using 21 scanners, over 220 edit stations and data in XML format and ASCII stored in about 76TB of SANs. All scanning and first level editing was completed within nine months of enumeration period. About 1000 Nigerians were trained and gained expertise in various aspects of the scanning technology There is a need for intensive trainings in these areas of OMR/OCR forms design and development of appropriate scanning softwares.

25 End Thank you for your attention