CROSS-COUNTRY WORKSHOP FOR IMPACT EVALUATIONS IN AGRICULTURE AND COMMUNITY DRIVEN DEVELOPMENT Addis Ababa, April 13-16, 2009 A CONCEPT NOTE FOR IMPACT EVALUATION OF THE ETHIOPIA NEWLY RE-DESIGNED FOOD SECURITY PROGRAM

Team Members Ato Kebede Tafesse, Food Security Coordination Directorate Ato Ahmed Adem, Pastoralist Community Development Ato Behailu Shewangzaw, Food Security Coordination Directorate Ato Hailu Ankisso, Food Security Coordination Directorate Ato Welelaw Sendeku, WB-CIDA Food Security Project Ato Zeleke Aged, Food Security Coordination Directorate Ato Zena Habewold, Food Security Coordination Directorate Dr. Hussien Hamda, Addis Ababa, University Ato Isayas Abate, The World Bank Ato Getahun Tafesse, CIDA Ms. Laketch Mickael, The WB Ms. Maddalena Honorati, The WB Ms. Marie Gaarder, 3ie Mr. Ludovic Subran, The WB

Background Long period of Emergency Assistance Decision to differentiate Chronic vs Transitory Food Insecure people Food Security Program that has three pillars was implemented during the past five years Only one of the pillar program is under regular evaluation The Current Redesign Exercise is aimed at Integrating the three pillar programs into one.

Framework for Welfare Status Categorization

Title FOOD SECURITY PROGRAM COMPONENTS 1.Productive Safety Net 1.1 Direct Support - Cash or Food transfer 1.2 Public Works - Cash or Food transfer 2.Household Investment & Financial Service 3.Complementary Community Infrastructure Investment (newly incorporated) 4.Land Access

Title FOOD SECURITY PROGRAM COMPONENTS 1.The Safety Net Program is planned to address: 8.3 million chronically food insecure people in 290 woredas 2.The Credit Support, Ag. Extension, etc (household investment package) will be primarily targeted to Safety Net beneficiaries

Title 1.Does provision of Credit makes a difference? 2.Do different types of households benefit differently from Credit? 3.Is Credit more effective in certain type of intervention setting than other? PRIORITIES FOR LEARNING: INTERVENTIONS

Title Does Provision of Credit on top of Safety Net Makes a Difference in making progress towards Graduation? 1.Magnitude 2.Speed 3.Sustainability EVALUATION QUESTIONS

Title EVALUATION DESIGN Sample Population: Safety Net Kebelles (households receiving the transfer cash, food or mix) within the Kebelle Community/ kebelle is unit of Intervention for the credit component and therefore will be unit for randomization The Plan is to Use a Randomized design that will be based on: o The Roll-out / Phasing of the Intervention by the GoE over the next year o Excluding current RUSACA kebelles that have a pseudo-household investment component HH level is Unit of observation/ analysis for Outcome Measurement namely: Asset value index and livelihoods measurement, both currently used as benchmark indicators for measuring ‘graduation’

Title EVALUATION DESIGN INDICATORS o Asset value index currently used as benchmark indicators for measuring ‘graduation’ o Livelihood measurement Additional indicators that can potentially be considered o Human capital o Anthropometric o Link with the national poverty line (per adult expenditure that takes into account minimum calorie intake and non-food need)

Title EVALUATION DESIGN Sampling Frame A total of 290 woreda, composed of 15 Kebelles on average, ie 700 Households on average per Kebelle Approx 3000 Kebelle to be considered (excluding RUSACA ones), out of which (tbd) 300 will be included in the evaluation design, with a hundred household in the each Kebelle (tbd) Control group: Kebelle with safety net support, without access to rural finance

Title EVALUATION DESIGN Effects that have to be Captured/ Controlled: 1.Households in the Control group should be households that are eligible, offered the credit and have accepted to take credit but did not get the credit yet 2.Purposeful manipulation of the selection criteria by potential beneficiaries 3.In the survey design, households will be asked on whether they would have received Credit if they had been offered

Title EVALUATION DESIGN Stratification/ Results disaggregation 1.Different Levels of Support provided under household investment (three credit categories: subsidized I, subsidized II and commercial rate) will be offered according to household food insecurity levels – see ‘further options’ slide) 2.Different timeline for support (3-, 6- and 9-months) 3.Communities/ kebelles that have acquired complementary Infrastructure investment –vs- those without (see ‘further options’ slide) 4.Cash beneficiaries, Food and mix Beneficiaries 5.Last outcomes for male-headed and female-headed households will be particularly compared

Title EVALUATION DESIGN Assumptions: 1.Households who are offered Credit will take it (so far acceptance rate close to 100%) 2.The financial institutions are new ones so that a treated kebelle is not at the expenses of a control kebelle (available ‘supply’)

Title 1. Sampling will be done among Safety Net kebelle/beneficiaries 2.The Main Data Source will be the on-going IFPRI Survey for both baseline and follow-up (annual panel data) Need for Over-sampling and adding of additional Module (set of questions) SAMPLING AND DATA

Title Explore potential linkage with other surveys such as 1.CSA HHICE Survey 2.CSA Welfare Monitoring Survey 3.Agricultural Census 4.Planned LSMS SAMPLING AND DATA

Title 1. Evaluation Concept Note to be discussed at the next Program Design Workshop in May, Baseline will be conducted end 2009 and/ or early Surveys will be repeated annually TIMELINE FOR IMPACT EVALUATION

Title 1. The GoE and Donor Joint Food Security Monitoring and Evaluation Technical Task Force will steer the Process – METT 2.METT will also get involved in technical issues of survey design, methodology and sampling with close support from local Academic/ research institutes 3.The GoE and Donor Joint Food Security Coordination Committee (JCC) will deal with policy and resource issues IMPACT EVALUATION TEAM: STAFFING

Title 4. IFPRI will be responsible for Survey Design, follow-up on quality of data collection and undertaking in-depth analyses 5.The CSA will be responsible for Data Collection and production of Basic Report 6.The GoE and Donor Joint Food Security Coordination Committee (JCC) will deal with policy and resource issues 7. Technical Assistance is expected from 3ie, AADAPT and DIME IMPACT EVALUATION TEAM: STAFFING

Title 1.The Study is expected to be financed through the Multi- Donor Trust Fund (MDTF) 2.To Pickyback on IFPRI, about 300,000 USD is needed 3.3ie, AADAP or DIME to cover TA on a) Impact Evaluation Design b)Subject matter (Rural Finance) expert BUDGET

Amasagenallehu

Title Compare impact of different programs in addition to the safety net: household investment, community infrastructure and household investment+community infrastructure Check for opportunity for proper impact evaluation (randomized phasing or non experimental toolbox). If take-up much less than expected among households, a randomized encouragement design (two-step approach) could be explored to capture both the impact of ‘offering the package’ and the impact of ‘actually borrowing’ Unfortunately, rural finance packages are targeted to different strata among the food insecure and randomization would need to be possible in practice (offering the credit to households irrespective of food insecurity characteristics) FURTHER OPTIONS