1. Evaluating Safety Net Programs Family Allowance Program-Honduras Oportunidades - Mexico Carola Alvarez, Inter-American Development Bank

2. Background  Initiated in 1990 as a cash transfer program to compensate the poor for lost purchasing power as a result of adjustment.  The program was restructured in 1998, to become a Targeted Human Development Program (THDP) known as PRAF/IDB - Phase II.  The objective of the evaluation was to test whether the new model had greater impact than the previous one.  Evaluations are usually designed after programs are running.  PRAF II was designed at the same time of its evaluation.

3.  Cash transfer to households (demand side only)  Untargeted  Amount of transfers not referenced to poverty level  Unstable beneficiaries Unsustainable benefits  Impact not evaluated PRAF I PRAF II Testing the models  Subsidizes demand and supply  Targeted to the poorest  Amount of transfers referenced to opportunity cost of children’s labor and cost of women’s time.  Beneficiaries covered for the duration of the program  Evaluation of impact

4. Targeting

5. The design choice: Proxy Means Test  Objective and verifiable  Transparent (political maneuvering)  National coverage  Large benefits Rationale  Consumption survey  Synthetic score construction  Qualify household by household (census or demand based) Required Activities

6. The implementation choice: Geographic  High social costs (Progresa experience)  High administrative costs  Technical inefficiency (predictive algorithm was weak)  Poverty level was high 70.5% extreme poor 78.0% poor 87.1% under the international poverty line of USD$2 per capita Rationale

7. Geographical Targeting: Malnutrition Index to identify municipalities at highest risk Extremely High Risk High risk Moderate Risk Low Risk

8. Evaluation Design

9. Main hypothesis  PRAF combines supply-side and demand-side interventions in an experiment to measure the relative effectiveness and synergy of supply and demand actions. Group I Demand only Group III Supply only Group II Supply & Demand Group IV Control group

10. Treatments Demand: –BS - Nutrition and Health cash transfer (Pregnant women and kids 0-3 years old) –BE - School cash transfer (Kids 6 - 12 years old, <5th Grade) Supply improvement transfers: –OS - Monetary transfers to parent & teacher school associations + teacher training –OE - Monetary transfers to health units + AIN-C

11. Amounts referenced Demand: –BS - Mother’s opportunity cost of attending the health center = USD$46 per year. –BE - Direct costs of sending a kid to school + household forgone income or opportunity cost = USD$58 per kid per year. Supply improvement transfers: –OS - Based on school size, number of grades, number of teachers + fixed amount per unit = USD$4,000 on average –OE - A fixed amount + beneficiary population covered + number of health professionals = USD$6,000

12. Duration of program coverage Beneficiary households remain in the program for at least 3 years. Only households that migrate out stop benefits. Ensuring duration of treatment and impact.

13. Municipality is the unit of assignments for each treatment No children in the same school assigned to incompatible groups Well defined limits Lack of conflict among school & health interventions Compatible with the decentralization policy Total Coverage

14. Randomized experimental design “Controls” for multiple interventions being implemented in the same area. No selection bias. (all units have the same probability of being selected) Statistically equivalent groups Best counter-factual situation. Easily interpreted results

15. Randomized assignment of municipalities Random selection of municipalities into 3 treatment groups and one control group using a lottery Transparency Selection supervised by community leaders, central and local government institutions.

16. Results of randomized selection

17. Measuring Impact

18. Selected Interventions ControlOS – OE BS-BE-OS-OEBS – BE BE – OS – OEBS – OEOE BS – OS – OEBS – OSOS BS – BE – OEBE – OEBE BS – BE – OSBE - OSBS

19. Clearly identified key performance indicators during program preparation % of children attending school % of children graduated standardized tests 100% of parent’s associations created and participating % of teachers trained 100% of school supplies received % of maternal mortality reduced 80% women receiving post and prenatal care 80% of children receiving growth control 100% pregnant women receiving health care 100% of children vaccinated Education Health & Nutrition:

20. Impact indicators: education Increase of 5 points in the average score of standard achievement tests in mathematics and language, kids in 2nd and 4th grade. 50% reduction in overall repetition rate 18%  9%

21. Impact indicators: health and nutrition Infant malnutrition reduced in 6 percentage points for children under 3 years of age. 52%  46% (0,20 increase in the ZTE score) 10% reduction in the incidence of anemia Children: 35%  31.5% Mothers: 31%  28%

22. Estimate of the minimum difference identifiable According to: –Variability of the population variable (variance – s2) –Number of selected units (municipality - m) within each selected condition –Number of households (h) surveyed per municipality –Average of the variance of observations per each interviewed household (π) –Geographical concentration (coefficient of inter-municipality correlation) Source: Murray, 1998

23. Sample size required

24. Sources of Information Socio-economic survey of municipalities Census for roster construction Household panel surveys (2000, 2002, y 2003) –G1 -1600 households –G2 -1600 households –G3 - 800 households –G4 -1600 households Total =5600 household (80 / Municipalities) Standardized tests Qualitative studies Cost effectiveness study

25. Impacts and Lessons learned (if we could do it over again….)

26. % of children >3 that had a medical check–up in the last 30 days Randomly assigned groups Demand Demand and Supply SupplyControl 2000 2002 N1,011 948 1,005 883 471 404 959 816 2000 2002 %44.0% 65.9% 46.3% 62.4% 45.2% 43.8% 44.3% 45.3% Change Diff-in-diff 21.9% 20.9% 16.1% 15.1% -1.4% -2.4% 1.0% -----

27. Implementation difficulties Supply interventions encountered line ministries resistance Weak capacity to maintain periodicity of payments Misunderstandings of original design (allow entrance of pregnant women in roster)

28. Lessons Learned The success of the evaluation depends on the success of the program’s implementation Underestimated implementation capacity of the country Underestimated needed supervision and involvement from the Bank Too much emphasis on impact evaluation We needed a strong monitoring framework

29. Monitoring to ensure treatment

30. Evaluation Measures IMPACT Monitoring Model Monitors perfomance and detects problems in the operation Management of daily processes Monitoring processes as they take place Measurement Strategy

31. Monitoring Model Measurement Tools Records the implementation of activities Identifies critical aspects in the operation Measures improvements in implementation Proposes alternative solutions

32. Measurement Strategy Institution Results / Other Clients Accountability Impact Results Coverage Roster Program services results Payments Health and Education Services Financial services Performance Eficiency indicators for local offices Quality of conditionality records Active engagement of municipal links Follow rules of operation Drop-outs from roster Capturing of Forms E2/S2 End of process mesures Operation / Daily management Quantitative benchmarks # of families need recertification Training of community respresentatives Measures processes in all its phases Measure impact and results Operational Monitoring and identification of problem areas Short Cycle Internal / Continous improvem ents INDICATORS strategic Performance Management SIIOP

33. 64.4% Breatfeeding women that receive micronutrient suplementation July-August 2003

34. 60 % Malnurished 2 to 4 year olds that recieved micronutrient supplementation July - August 2003

35. Pregant women that received micronutrient supplementation July-August 2003 67%