Evaluating Safety Net Programs Family Allowance Program-Honduras Oportunidades - Mexico Carola Alvarez, Inter-American Development Bank

Background  Initiated in 1990 as a cash transfer program to compensate the poor for lost purchasing power as a result of adjustment.  The program was restructured in 1998, to become a Targeted Human Development Program (THDP) known as PRAF/IDB - Phase II.  The objective of the evaluation was to test whether the new model had greater impact than the previous one.  Evaluations are usually designed after programs are running.  PRAF II was designed at the same time of its evaluation.

 Cash transfer to households (demand side only)  Untargeted  Amount of transfers not referenced to poverty level  Unstable beneficiaries Unsustainable benefits  Impact not evaluated PRAF I PRAF II Testing the models  Subsidizes demand and supply  Targeted to the poorest  Amount of transfers referenced to opportunity cost of children’s labor and cost of women’s time.  Beneficiaries covered for the duration of the program  Evaluation of impact

Targeting

The design choice: Proxy Means Test  Objective and verifiable  Transparent (political maneuvering)  National coverage  Large benefits Rationale  Consumption survey  Synthetic score construction  Qualify household by household (census or demand based) Required Activities

The implementation choice: Geographic  High social costs (Progresa experience)  High administrative costs  Technical inefficiency (predictive algorithm was weak)  Poverty level was high 70.5% extreme poor 78.0% poor 87.1% under the international poverty line of USD$2 per capita Rationale

Geographical Targeting: Malnutrition Index to identify municipalities at highest risk Extremely High Risk High risk Moderate Risk Low Risk

Evaluation Design

Main hypothesis  PRAF combines supply-side and demand-side interventions in an experiment to measure the relative effectiveness and synergy of supply and demand actions. Group I Demand only Group III Supply only Group II Supply & Demand Group IV Control group

Treatments Demand: –BS - Nutrition and Health cash transfer (Pregnant women and kids 0-3 years old) –BE - School cash transfer (Kids years old, <5th Grade) Supply improvement transfers: –OS - Monetary transfers to parent & teacher school associations + teacher training –OE - Monetary transfers to health units + AIN-C

Amounts referenced Demand: –BS - Mother’s opportunity cost of attending the health center = USD$46 per year. –BE - Direct costs of sending a kid to school + household forgone income or opportunity cost = USD$58 per kid per year. Supply improvement transfers: –OS - Based on school size, number of grades, number of teachers + fixed amount per unit = USD$4,000 on average –OE - A fixed amount + beneficiary population covered + number of health professionals = USD$6,000

Duration of program coverage Beneficiary households remain in the program for at least 3 years. Only households that migrate out stop benefits. Ensuring duration of treatment and impact.

Municipality is the unit of assignments for each treatment No children in the same school assigned to incompatible groups Well defined limits Lack of conflict among school & health interventions Compatible with the decentralization policy Total Coverage

Randomized experimental design “Controls” for multiple interventions being implemented in the same area. No selection bias. (all units have the same probability of being selected) Statistically equivalent groups Best counter-factual situation. Easily interpreted results

Randomized assignment of municipalities Random selection of municipalities into 3 treatment groups and one control group using a lottery Transparency Selection supervised by community leaders, central and local government institutions.

Results of randomized selection

Measuring Impact

Selected Interventions ControlOS – OE BS-BE-OS-OEBS – BE BE – OS – OEBS – OEOE BS – OS – OEBS – OSOS BS – BE – OEBE – OEBE BS – BE – OSBE - OSBS

Clearly identified key performance indicators during program preparation % of children attending school % of children graduated standardized tests 100% of parent’s associations created and participating % of teachers trained 100% of school supplies received % of maternal mortality reduced 80% women receiving post and prenatal care 80% of children receiving growth control 100% pregnant women receiving health care 100% of children vaccinated Education Health & Nutrition:

Impact indicators: education Increase of 5 points in the average score of standard achievement tests in mathematics and language, kids in 2nd and 4th grade. 50% reduction in overall repetition rate 18%  9%

Impact indicators: health and nutrition Infant malnutrition reduced in 6 percentage points for children under 3 years of age. 52%  46% (0,20 increase in the ZTE score) 10% reduction in the incidence of anemia Children: 35%  31.5% Mothers: 31%  28%

Estimate of the minimum difference identifiable According to: –Variability of the population variable (variance – s2) –Number of selected units (municipality - m) within each selected condition –Number of households (h) surveyed per municipality –Average of the variance of observations per each interviewed household (π) –Geographical concentration (coefficient of inter-municipality correlation) Source: Murray, 1998

Sample size required

Sources of Information Socio-economic survey of municipalities Census for roster construction Household panel surveys (2000, 2002, y 2003) –G households –G households –G households –G households Total =5600 household (80 / Municipalities) Standardized tests Qualitative studies Cost effectiveness study

Impacts and Lessons learned (if we could do it over again….)

% of children >3 that had a medical check–up in the last 30 days Randomly assigned groups Demand Demand and Supply SupplyControl N1, , %44.0% 65.9% 46.3% 62.4% 45.2% 43.8% 44.3% 45.3% Change Diff-in-diff 21.9% 20.9% 16.1% 15.1% -1.4% -2.4% 1.0% -----

Implementation difficulties Supply interventions encountered line ministries resistance Weak capacity to maintain periodicity of payments Misunderstandings of original design (allow entrance of pregnant women in roster)

Lessons Learned The success of the evaluation depends on the success of the program’s implementation Underestimated implementation capacity of the country Underestimated needed supervision and involvement from the Bank Too much emphasis on impact evaluation We needed a strong monitoring framework

Monitoring to ensure treatment

Evaluation Measures IMPACT Monitoring Model Monitors perfomance and detects problems in the operation Management of daily processes Monitoring processes as they take place Measurement Strategy

Monitoring Model Measurement Tools Records the implementation of activities Identifies critical aspects in the operation Measures improvements in implementation Proposes alternative solutions

Measurement Strategy Institution Results / Other Clients Accountability Impact Results Coverage Roster Program services results Payments Health and Education Services Financial services Performance Eficiency indicators for local offices Quality of conditionality records Active engagement of municipal links Follow rules of operation Drop-outs from roster Capturing of Forms E2/S2 End of process mesures Operation / Daily management Quantitative benchmarks # of families need recertification Training of community respresentatives Measures processes in all its phases Measure impact and results Operational Monitoring and identification of problem areas Short Cycle Internal / Continous improvem ents INDICATORS strategic Performance Management SIIOP

64.4% Breatfeeding women that receive micronutrient suplementation July-August 2003

60 % Malnurished 2 to 4 year olds that recieved micronutrient supplementation July - August 2003

Pregant women that received micronutrient supplementation July-August %