1. New Methods in Household Surveys
TALIP KILIC
Living Standards Measurement Study Team
Poverty & Inequality Group
Development Research Group
The World Bank
PREM Learning Days DEC Course
“The Living Standards Measurement Study:
Innovation in Survey Data for Better Policy Making”

2. Outline Computer-Assisted Personal Interviewing Geo-referencing
Methodological Survey Experiments on Agriculture

3. COMPUTER-ASSISTED PERSONAL INTERVIEWING (CAPI)

4. Computer-Assisted Personal Interviewing (CAPI)
PAPI: Paper-based personal interviewing, coupled with computer-assisted field-based data entry (CAFE) pioneered by the LSMS
CAPI: Integration of interviewing & data entry through the use of a handheld device, preloaded with an electronic questionnaire
Household (HH) surveys implemented on CAPI platform since the late’ 80’s in high- & middle-income countries, inc. the Netherlands, the UK, the US, Norway & Turkey
Increasing number of applications in low-income setting in recent years
Mobile Phones, PDAs vs. Netbooks & Tablet PCs

5. LSMS Experience LSMS operations marked by a gradual transition to CAPI
CAPI survey experiment (~200 households) (Albania)
Application developed in CSProX
2007- CAPI survey (~500 households) (Ngara District, Tanzania)
Application developed in CWEST
Kagera Health and Development Survey (KHDS) (Tanzania)
Uganda National Panel Survey (UNPS) ( )
Supported by LSMS-ISA, implemented by Bureau of Statistics
Partial transition to CAPI in 2010/11 (in CWEST); CAPI transition completed in 2011/12 (on-going; in CWEST & Surveybe); Next round 2013/14
Ethiopia Rural Socioeconomic Survey (ERSS) ( )
Supported by LSMS-ISA, implemented by Central Statistical Agency
CAPI application developed in Surveybe for the Ag Questionnaire, implemented in a subset of EAs in 2011/12; Next round in 2013/14

6. Hardware Options General Features: 7-10’’ stylus-friendly screens
Rapid navigation across questionnaire
Several questions displayed at one time
Camera, microphone, virtual keyboard & hand-writing recognition software 
5-7 hours of (initial) battery life
Extended battery pack, external battery pack & daily charge of batteries recommended
Generators in low-electrification settings
Multiple ports: Internet dongles, GPS units, external keyboards
Samsung Q1b Ultra (KHDS; UNPS) $
Asus Eee PC T101MT (UNPS; ERSS) $

7. Software Options
Traditional DE software designed for transfer from paper questionnaire to computer
Benefits of relying on CAPI better realized working with software packages designed for interactive interviewing
CAPI software packages make up a small market, with varying degrees of cost effectiveness & types of strengths
Key players: Blaise, CASES, CSProX, MMIC & Surveybe
LSMS-commissioned comparative assessment of software programs for the development of CAPI applications (available on

8. Why Contemplate Transition to CAPI?
Enhanced tools for in-field & remote management of mobile teams
Headquarters & Team Leaders: Assigning work, tracking progress, immediate & comprehensive feedback
Expected gains in timeliness of data availability
Data entry, checking & exportation in one application
Expected gains in data quality
Accommodation of non-linear/integrated questionnaires
Automated routing reduces the incidence of missing data
Data checking, reporting & revision facilities during the interview
Range & consistency checks, flags for missing fields
Improvements in quantification of nonstandard units
Instructions on questions, note taking facilities

9. Non-linear Navigation

10. Automated Routing

11. Consistency Checks

12. Consistency Checks (Cont’d)

13. Use of Media for Better Quantification

14. Use of Media for Better Quantification

15. Managing Expectations
Data quality control principles in CAPI set-up no different than surveys based on PAPI with CAFE
CAPI tools useful as much as enumerators & field supervisors take advantage of available facilities & act on inconsistencies
Relative impact of CAPI on data quality: Open question
Limited evidence on improved data quality with respect to a well-supervised survey based on PAPI with CAFE
Fafchamps, M., McKenzie, D., Quinn, S., and Woodruff, C. (2010). Using PDA consistency checks to increase the precision of profits and sales measurement in panels. CSAE Working Paper Series No
Caeyers, B., Chalmers, N., and De Weerdt, J. (2012). “Improving consumption measurement and other survey data through CAPI: Evidence from a randomized experiment.” Journal of Development Economics, 98, pp. 19–33.

16. Cost Implications
CAPI generates (minimal) savings in printing costs & data entry
Savings increase with the complexity & frequency of survey
Significant up-front costs in hardware procurement
More cost-effective if machines are used in other survey operations
Transition into CAPI also driven by field work structure
Size of the enumerator corps may be prohibitively large
Gradual transition to CAPI as part of the LSMS operations primarily underlined by demand for increased data quality & availability

17. Uganda National Panel Survey (UNPS) CAPI Experience
Teams quick to adapt, instrumental in training & knowledge sharing
Required change in institutional thinking on surveys: Greater up-front work (& costs) with respect to PAPI with CAFE
Prep of Wave I (PAPI) data uploaded onto Wave II (CAPI) application
Hundreds of intra/inter-module consistency checks, in addition to range & default checks for missing values
Programming of rules on generation of household & individual identifiers for new additions to the sample
Training of UBoS Headquarters staff on case management suite

18. UNPS CAPI Experience (Cont’d)
In-country procurement problems
Lags assoc. with operating within Government systems/unreliable suppliers
US procurement by the LSMS-ISA project: Not straightforward either
Anti-virus software critical to maintaining the hardware integrity
Application glitches even after piloting three times: Need for more intensive testing in comparison to PAPI with CAFÉ
CAPI application platform based on multiple software packages: CWEST & CSPro (in 2010/11); CWEST & Surveybe (in 2011/12)
Dependence on the CWEST application developer for adjustments
Continued reliance on multiple software packages necessitated by lack of case management features on Surveybe
Timely communication of bugs that might compromise the integrity of incoming data critical: No paper questionnaires to re-enter

19. UNPS CAPI Experience (Cont’d)
Continuing improvements to the CAPI application on a rolling basis throughout the field work
Even with internet dongles, slow internet speeds & lack of service in certain areas
Affects timely headquarters review of data sent from the field
Receipt of application updates by the survey teams not always timely
Regular backup of interview files in the field & at the HQ crucial
Lags associated with Surveybe data export
Still need a paper questionnaire for dissemination purposes: CAPI application dictionary is not more than a linear questionnaire report

20. Comparative Assessment of Software Programs for the Development of CAPI Applications
Initially twofold objective: 
Inform internal decision making on the choice of surveys for upcoming surveys planned under the LSMS-ISA project, in Uganda, Ethiopia, and Nigeria
Fill the gap in public knowledge on the relative performance of available software packages for the development of CAPI applications for multi-topic household surveys
Peer-reviewed report, managed by the LSMS team, compiled by the IRIS Center at the University of Maryland, reviewed by software developers prior to release
Available on

21. Comparative Assessment… (Cont’d)
Software packages screened as suitable for the development of CAPI applications
for multi-topic household surveys & evaluated by the report include:
Software
Developer
Blaise
Westat & Statistics Netherlands
CASES
Computer-Assisted Survey Methods Program
at the University of California, Berkeley
CSProX
Serpro, S. A.
Entyware
Techneos
MMIC
RAND Labor and Population
Open Data Kit
The University of Washington’s Department
of Computer Science and Engineering
Pendragon Forms
Pendragon Software Corporation
Surveybe
Economic Development Initiatives

22. Comparative Assessment… (Cont’d)
Structure of the report
Brief overview of each software package
Comparative assessment of each software package in 12 areas:
Detailed evaluation of each software package, accompanied with full functionality check lists for each evaluation area
Evaluation Areas
Programming
Data Transfer
Questionnaire Development
Data Exporting
Questionnaire Implementation
Support & Documentation
Interface for Field Users
Hardware & Software Needs
Questionnaire Navigation
Pricing & Upgrades
Case Management
Extensibility

23. Comparative Assessment… (Cont’d)
No single software package is an unequivocal frontrunner in all evaluation areas
Ideal approach to questionnaire design: Marrying menu-driven development environment for novice users with a command line for more experienced users, replicating functionality in the menu environment, accommodating customization needs
Missing across all evaluated programs
Non-trivial task in this set-up: Allowing for simultaneous questionnaire development by several survey designers & being able to integrate each piece into an application
Positive relationship between quality/scope of documentation & proprietary nature of the software (MMIC, ODK vs. Blaise)
Top contenders:
Surveybe: Ease to use (menu-based development environment) but lacks case management suite & only allows for sequential workflow for qx development
MMIC & Blaise: Powerful & expansive in scope but steep learning curve (command line driven development environment) & high need for technical assistance
Differences in quality of documentation & user community, in favor of Blaise
Differential cost structures
Open source (TA needs?) vs. per software installation (corporate licensing) vs. data points

24. Where Next?
Sustainability of adoption relies on availability of a user-friendly, yet highly customizable, public solution around which in-country capacity could be built
LSMS and Development Economics Computational Tools (DECCT) Unit of the World Bank supporting the development of a publicly available, closed-source CAPI software platform
Informed by LSMS field experience & comparative CAPI software assessment
Core interface components: Builder (for Survey Designers), Manager (for Survey Managers & Team Leaders), Client (for Interviewers)
Approach to Builder: Coupling a menu-driven development environment for a core set of functionality (common across LSMS-type household surveys) with a “command-line” for programming more complex features & supporting customization
Key decisions: Target hardware/software platforms
Hardware is not independent of OS!
Mouse/keyboard vs. Stylus vs. Finger Touch: Implications for questionnaire design
OS platform independence: Implications for software development,
robustness of interface components

25. GEO-REFERENCING

26. Geo-referencing
Recording longitude and latitude of households & other POI (plots, markets, schools, health centers)
GPS-based data collection not new
Technology is fairly cheap, wider appreciation for usefulness of spatial data (GoogleEarth, GoogleMaps, remote sensing data)
Innovation in uses of GPS data:
Survey Management
Evaluation of Survey Responses
Data Integration
Dataset Characterization
Research Questions

27. Survey Management
Standard mapping-grade GPS units (Garmin eTrex, Trimble Juno) should produce fairly accurate readings
Visit verification (timing & location)
Navigation to & positive identification of sample household in successive panel survey rounds
Standard mapping grade units (Garmin eTrex, Trimble Juno) with 5-10 m accuracy. Conditions that affect accuracy include dense canopy and/or buildings, configuration of satellites in sky at time of capture, disturbance in upper atmosphere). Also found that age of equipment (battery) might affect accuracy.

28. Data Evaluation Internal consistency checks & validation
Distance from household to agricultural plots
HHID
Plot ID
EST_KM
GPS_KM
1234 M2
3.0
1.9 M3
2.5 M4
2.0
1.3
GPS data also provides new ways to check for consistency and validate survey data.
Very large differences might be indication of error, also can look for bias in reporting

29. Data Evaluation (Cont’d)
Distribution of household responses on the occurrence of drought / irregular rains shows large local variation
Simply looking at the geographic distribution of survey responses can also provide some insights into the data.
This map shows that in some cases there is a significant amount of local variability in drought responses.
You might expect that rainfall conditions would be more consistent within an ea or village,
but what we see here is really a mix of responses within communities.
(explain map: colors show % of hh aggregated at cell level, zoom into 4 grid cells and see household level
responses blue/red). the mixed responses within each block may be partly due to the subjective nature of the drought identification in hh shock module.

30. Data Integration
Having GPS locations enables integration with other spatial datasets, making available large range of additional variables.
3 types of geovariables we are preparing for dissemination with LSMS-ISA survey datasets are:
Distance – mapping showing distance to major market – darker areas more remote
some geophysical characteristics, particularly with relevance to ag productivity, this just a subset
3. Time series – as we mentioned yesterday in shock module, information on timing of rains and rainfall anomalies can be useful in interpreting production statistics and other survey data. Where time series are available we can use use rainfall or vegetation response measures to enhance augment what is capture in the survey instruments
Distance
HH to Plot
HH to Market
HH to Major Road
Environmental
Climatology
Landscape typology
Soil
Elevation
Terrain
Time Series
Rainfall
Vegetation Indices

31. Data Characterization
Representativeness of the Tanzania National Panel Survey (TZNPS) 2008/09 sample across agro-ecological zones
Agro-Ecological Zone
# of Households
Coastal, Islands &
Alluvial Plains
1531
Arid &
Semi-Arid Lands
535
Northern, Southern &
Western Highlands
688
Plateau
512
TOTAL
3266
Survey sample may be chosen from population reference frame but you can also look at how well different strata are represented in the data
Example here is looking at the distribution of households in Tanzania NPS across major agroecological zones

32. METHODOLOGICAL SURVEY EXPERIMENTS ON AGRICULTURE

33. Methodological Survey Experiments on Agriculture
Identification process
Alignment with Global Strategy
Field experience
Iterative
Peer-review
Methodological survey experiments in the pipeline on the measurement of
Agricultural land areas
Soil fertility
Water resources
Agricultural labor input
Continuous/extended harvest crop production

34. Methodological Survey Experiments on Agriculture (Cont’d)
As part of a DFID-funded 3-year ( ) program led by the LSMS team, conducted in collaboration with the
Statistics Department of the Food & Agriculture Organization of the United Nations (Component 1 & 5)
World Agroforestry Centre (Component 2)
World Bank Environment Department (Component 3)

35. Component 1: Agricultural Land Areas
Why is it important?
Fundamental component of agricultural statistics (forecasting production and yield measurement)
Priority #1 of Global Strategy
Recent research (Carletto et al ) documents bias in farmer-reported land areas with respect to GPS-Based counterparts
Small (large) farms shown to over-(under-)report: Implications for the inverse farm size-productivity relationship
Available measurement methods
Farmer Reporting
P2/A method
GPS (already utilized in LSMS-ISA surveys)
Traversing (Compass & Rope)

36. Component 2: Soil Fertility
Why is it important?
Physical soil characteristics remain key unobserved variables for analysis of agricultural productivity
Available measurement methods
Farmer Evaluation
Spectral soil analysis (SSA)
Conventional soil analysis (CSA)

37. Component 3: Water Resources
Why is it important?
Water essential input into production & agriculture in sub-Saharan Africa is predominantly rainfed
Large discrepancies across data sources on water availability
Publicly available data sources defined at low resolutions
Higher resolution data not publicly available
Available measurement methods
Farmer Reporting
Remote Sensing
Weather Stations
Communal Rain Gauges

38. Component 4: Agricultural Labor Input
Why is it important?
Essential for accurate labor productivity measurement
Existing data very poorly measured
Available methods
Recall
Computer-Assisted Telephone Interviews
Labor input diaries

39. Component 5: Continuous/ Extended Harvest Crops
Why is it important?
Continuous/extended harvest crops are major staples in many African countries
Inaccuracy of recall: May extend across seasons, harvest on an on-going, at times need, basis
Available methods
Recall
Crop card (with local monitors)
On-going work with the Uganda Bureau of Statistics
CATI (data transmission/supervision)

40. New Methods in Household Surveys
TALIP KILIC
Living Standards Measurement Study Team
Poverty & Inequality Group
Development Research Group
The World Bank
PREM Learning Days DEC Course
“The Living Standards Measurement Study:
Innovation in Survey Data for Better Policy Making”