1. 1 Health Results-Based Financing Impact Evaluation Surveys Quality Assurance and Data Management Álvaro Canales, Beatriz Godoy, Juan Muñoz Sistemas Integrales July 16, 2008

2. 2

3. 3 The keys to survey data quality Instruments & methods –Questionnaires –Other Anthropometrics Biological samples Geopositioning Etc –Piloting and field-testing Sampling –Design –Implementation Organization –Core staff team –Fieldworkers Selection Training Supervision –Time frame Data Management –Before fieldwork –During fieldwork –After fieldwork

4. 4 Instruments and methods Questionnaire design can be time-consuming –Requires testing, probably many times –Requires translation into local language/s –Requires drafting, formatting, etc Most households need to be visited many times, to –Talk to specific respondents –Reduce respondent fatigue Instrumental measures have serious implications: ethical clearance, procurement, training, logistics Testing is a 2-stage process –Pilot-test the questionnaire –Field-test procedures and methods

5. 5 Sampling In scientific sampling, each household in the population of interest is given a probability of being selected in the sample that is positive and known The HRBF samples will not be simple random samples. They will be –Stratified by Region, by Urban/Rural, by Control/Treatment, … –Selected in two stages or more Area Units in the first stage/s Households in the last stage The last sampling stage will normally require a household listing operation, which –may be used to identify eligible households –should be implemented carefully, to avoid selection biases

6. 6 Organization The survey requires a dedicated core staff team, composed of –a project manger –a field manager –a data manager Fieldwork should be organized on the basis of field teams, each composed of –a supervisor –3-4 interviewers –Possibly, specialized technicians, nurses, etc. –Ideally, a data entry operator It is better to field the survey with few teams during many months than the other way around. This will –Permit effective monitoring –Capitalize the investment on training –Allow keeping the best fieldworkers

7. 7 Training Consists of –Plenary sessions –Small group sessions –Field practice Needs to be preceded by –Preparation of training materials –Training of master trainers A typical HRBF survey will require –2-3 weeks for the household questionnaires –Extra time for anthropometrics, biological tests, GPSs, etc. Should incorporate the final selection of fieldworkers Good training is extremely cost-effective, but often underestimated

8. 8 Computer-Assisted Field Editing The CAFE approach can –Deliver quick, consistent databases –Produce more reliable data: inconsistencies are solved in the households, not through office guesswork –Eliminate the long and frustrating “data cleaning” phase –Permit effective monitoring of fieldwork: shortcomings can be detected early and corrected before it’s too late To implement CAFE, data management needs to considered as a part of survey design and implementation, not as part of survey analysis

9. 9 This presentation only flagged Some of the most important issues Some of the most dangerous pitfalls For a broader coverage, check –A manual for planning and implementing the LSMS survey (can be downloaded from the LSMS website, in English, Spanish and Russian) –Survey Quality Assessment Framework (handbook and checklist, soon to be released by DECDG) –DECRG annual training courses on Survey design and implementation Sampling for household surveys

10. 10 Last but not least International comparability is an additional, specific challenge faced by the HRBF The conceptual framework is essential, but survey methods and the details of implementation are important too. It is better to address them earlier rather than later

11. 11