1. 1 Innovations in LSMS data & implementation Kathleen Beegle Workshop 17, Session 3 Designing and Implementing Household Surveys March 31, 2009

2. 2 Overview Testing: measuring ability through tests Computer Assisted Personal Interviewing (CAPI) New health measures Impact evaluation Other new areas….

3. 3 Testing (D. Filmer and N. Schady work in Cambodia) Students  Mathematics test  Peabody Picture Vocabulary Test Early Child Development  Test de Vocabulario en Imagenes Peabody (TVIP)  Woodcock-Johnson III Memory for Names  Ages & Stages test for child development Control measures of caregiver cognitive ability  Progressive Matrices Test

4. 4 Testing, cont. Appropriateness vs. Benchmarking  For example: Developing math test (need to ensure compatibility with expectations/curriculum) Adapting vocabulary test (at what point is it no longer comparable?) Translation Even more important to be precise Training Familiarity with / mastery of materials Rapport with interviewees (esp. children for child development tests) Implementation Quiet space Cost Internationally comparable copyrighted tests expensive (TVIP/WJ)

5. 5 Implementing tests

6. 6 What is CAPI? Interviewing and data entry are brought together. No paper questionnaires PDAs, mini-PCs (ultra mobile PCs) Software Off-the-shelf products designed for linear questionnaires On-going development of software for complicated questionnaire structures (like LSMS) Hardware can come with built in cameras, sound recording, CPS, external keyboard, finger scanner……etc….. Models to allow internet access through cell phone network

7. 7 Why CAPI? Interviewing and data entry are brought together  Faster data: data are available as soon as transferred (same day)  More accurate data: quality improved through validation checks during/immediately after interview Other advantages Flexible questionnaire design Can embed listing, sampling, past data (panels), multi- languages, instructions from manual

8. An example: UMPCs in Tanzania Ultra Mobile Personal Computer Large touch Screen Runs standard desktop operating systems Multiple ports just like a desktop

9. Large screen allows you to see the whole picture Whole sections are displayed and flow maintained

10. 10 Rapid click navigation to other sections You are about to finish an interview when a child walks in that the respondent had not mentioned before. With a UMPC layout you can get to the census/ roster section in one click to add the details and continue your interview

11. Display of picture options for greater accuracy Respondent says “a bunch of bananas”. With a large UMPC screen and storage you can display different sizes to choose from, calculate the correct energy value and therefore obtain accurate data (food, medication, amount of production, size of asset...)

12. 12 CAPI Experience Adapting DE software is complicated. Through 3 surveys, gaining experience on practicalities of this technology: security, remote support, battery life, etc… Challenges of developing questionnaire UMPCs have some advantages over PDA:  Maintaining the best features of a well-laid out questionnaire: screen size and segueing to the next screen.

13. 13 Collecting New Health Measures in LSMS/IS Surveys  Anthropometrics  BP/pulse  Observed physical ability  Arm circumference  Hemoglobin (anemia)  HIV/AIDS  Malaria  Avian flu  STDs Health measures collected multi-topic surveys to measure outcomes that are not reliably self-reported. In the latter group, testing in a number of ways: laboratory testing of blood samples in-home testing (small blood sample from finger pin-prick in case of Hemoglobin, saliva for HIV) physical examination.

14. 14 Collecting New Health Measures: Challenges Staffing. May require a trained nurse (VCT) to be added to the field team. Cost/Management implications Human Subjects Protocols. Internal Review Board (IRB) process: both within country and with other institutions  Consent processes: verbal consent replaced with signed forms.  Data security: both within field teams and once electronic.  Informing respondents of the test result. Obligations for follow-up (ability to provide, implications for research).

15. 15 Impact evaluation LSMS is (usually) not designed for specific evaluation work, but can potentially be used for this Usually requires coordination of the evaluation team with the national statistics office either before or after, since additional may be needed

16. 16 Impact evaluation, cont. Nicaragua FISE evaluation (Social Fund)  Booster sample of FISE communities. 1998 LSMS provided adequate non-FISE communites (piggy-backing)  Oversample FISE households and use the LSMS survey data (i) as a backdrop for comparison for assessing poverty targeting and (ii) to construct a control group to assess impact Malawi Fertilizer Voucher Evaluation (2006/07)  Constructed a panel from the Malawi LSMS in 2004/05. Re-interviewed households after the implementation of a new fertilizer voucher scheme.

17. 17 Pros matching method can allow for use of data collected during one time period to conduct an evaluation -- more rapid than method with baseline good measures of targeting and impact (consumption- based poverty estimates; variety of outcomes including anthropometric measures, etc.) more cost-effective than separate, in-depth survey Cons somewhat lengthy evaluation process use of LSMS questionnaire limits ability to tailor questions to FISE Evaluation cross-sectional surveys may not be intended to become panels (availability of re-contact information, names, etc..) Pros and Cons of “Piggy Backing” on an LSMS

18. 18 Extending an LSMS to other new areas New methods to extend the range of topics in an LSMS and improve the quality of data. Expectations of the future.  Using stones/beans to illicit expected probabilities of respondents  E.g. Pastoralists with little formal education asked about expectations of rainfall in the coming season, allocating stones across three different piles for “above normal”, “normal” and “below normal”  Validity discussed in “Measuring Subjective Expectations in Developing Countries.” 2009. Delavande et al. World Bank Policy Research Working Paper 4824.

19. 19 Extending an LSMS to other new areas Subjective reporting of health, welfare, satisfaction of life, etc…  Problem of different scales across people. A person’s standard of good health may be different from another person’s standard.  Hypothetical vignettes: Respondents score a set of vignettes describing different scenarios related to the topic (such as health status) for a hypothetical person. These vignettes can help reveal the respondents own scale.  King at al (2004) and King and Wand (2007).

20. 20 Extending an LSMS to other new areas These new areas imply new costs:  Piloting to ensure they work in different context  Time to administer questionnaire  Training interviewers appropriately