Armenias Millennium Challenge Account: Assessing Impacts Ken Fortson, MPR Ester Hakobyan, MCA Anahit Petrosyan, MCA Anu Rangarajan, MPR Rebecca Tunstall, MCC Ken Fortson, MPR Ester Hakobyan, MCA Anahit Petrosyan, MCA Anu Rangarajan, MPR Rebecca Tunstall, MCC

MCA-Armenia Program Compact was signed on March 27, 2006 Total amount is $235.6 million Implementation started on September 29, 2006 Duration is five years Compact was signed on March 27, 2006 Total amount is $235.6 million Implementation started on September 29, 2006 Duration is five years

MCA-Armenia Program (continued) Overall Goal: Create conditions for sustainable economic growth to reduce rural poverty Overall Objective: Rural development through investment in (1) Rural Roads to improve access to economic and social infrastructure, and (2) Irrigation infrastructure to increase agricultural productivity Overall Impact: It is expected that by 2011 rural poverty will decline by additional 6 percentage points. Overall Goal: Create conditions for sustainable economic growth to reduce rural poverty Overall Objective: Rural development through investment in (1) Rural Roads to improve access to economic and social infrastructure, and (2) Irrigation infrastructure to increase agricultural productivity Overall Impact: It is expected that by 2011 rural poverty will decline by additional 6 percentage points.

Projects Rural Roads Rehabilitation: –Improve up to 943 km of rural roads Irrigation Infrastructure Rehabilitation: –Improve access to and efficiency of irrigation systems Rural Roads Rehabilitation: –Improve up to 943 km of rural roads Irrigation Infrastructure Rehabilitation: –Improve access to and efficiency of irrigation systems

Projects (continued) Water-to-Market Activities –Improve the profitability of WUA members –Institutional strengthening of Water User Associations Water-to-Market Activities –Improve the profitability of WUA members –Institutional strengthening of Water User Associations

How is the MCA-Armenia Program Different? Country ownership of the program –Developed by the country –Implemented by the country Economic analysis focuses on integrated approach to solve rural/agricultural issues New rigorous approach to evaluate the program –Project level impact evaluations Country ownership of the program –Developed by the country –Implemented by the country Economic analysis focuses on integrated approach to solve rural/agricultural issues New rigorous approach to evaluate the program –Project level impact evaluations

Overview of Presentation I.Water-to-Market Project (Anahit) II.Associated Impact Evaluation (Ken) I.Water-to-Market Project (Anahit) II.Associated Impact Evaluation (Ken)

Water-to-Market Activity

Water-to-Market Activity Objectives Increase agricultural efficiency and profitability –Training –Technical assistance –Credit Maintain long-run sustainability of the irrigation infrastructure through increased water usage fee collection The direct impacts of the Water-to-Market activities are employment generation in rural areas and increased incomes for farms and rural businesses Increase agricultural efficiency and profitability –Training –Technical assistance –Credit Maintain long-run sustainability of the irrigation infrastructure through increased water usage fee collection The direct impacts of the Water-to-Market activities are employment generation in rural areas and increased incomes for farms and rural businesses

Introduction of New On-Farm Water Management Technologies Objective: Improve farmers skills in water management and access to farm-level equipment to enhance the efficiency of water usage –Train 60,000 farmers in water management Expected Result: Improved productivity and increased income from agriculture by adoption of water saving technologies Objective: Improve farmers skills in water management and access to farm-level equipment to enhance the efficiency of water usage –Train 60,000 farmers in water management Expected Result: Improved productivity and increased income from agriculture by adoption of water saving technologies

Transition to Higher Value Agriculture Objective: Support transition to more profitable agricultural production through –Train 30,000 farmers in high-value agriculture –Crop substitution –Increased cropping intensity –More productive livestock activities –Introduction of new technologies, techniques, and higher yield generating inputs Expected Results: Improved productivity and increased income from agriculture by increasing the commercial value of farm outputs Objective: Support transition to more profitable agricultural production through –Train 30,000 farmers in high-value agriculture –Crop substitution –Increased cropping intensity –More productive livestock activities –Introduction of new technologies, techniques, and higher yield generating inputs Expected Results: Improved productivity and increased income from agriculture by increasing the commercial value of farm outputs

WtM Impact Evaluation

Key Research Questions Did the program affect agricultural practices of Armenian farmers? –Irrigation –Higher-value crops Did the program affect agricultural productivity? Did the program improve household well-being? –Increase income –Reduce poverty Did the program affect agricultural practices of Armenian farmers? –Irrigation –Higher-value crops Did the program affect agricultural productivity? Did the program improve household well-being? –Increase income –Reduce poverty

Random Assignment: The Gold Standard Randomly assign eligible participants into two groups –Program group: Receives the program –Control group: Does not receive the program for some period Program and control groups are the same on average, except one group has access to program –Any observed differences over time can be attributed to the program Randomly assign eligible participants into two groups –Program group: Receives the program –Control group: Does not receive the program for some period Program and control groups are the same on average, except one group has access to program –Any observed differences over time can be attributed to the program

Treatment and Control Groups Randomly assign when training will start in each village, with three groups: –Compact Year 2 (Treatment) –Compact Years 3 and 4 –Compact Year 5 (Control) Random selection is a fair way to determine timing of training Compare Compact Year 2 villages to Compact Year 5 villages Randomly assign when training will start in each village, with three groups: –Compact Year 2 (Treatment) –Compact Years 3 and 4 –Compact Year 5 (Control) Random selection is a fair way to determine timing of training Compare Compact Year 2 villages to Compact Year 5 villages

Selection of Villages Total of 277 village clusters will be assigned: –Year 2: 120 clusters –Years 3 and 4: 77 clusters –Year 5: 80 clusters Some villages not included in the selection –Pilot phase villages –Villages that do not have adequate water and would not benefit from training yet (eligible later) Total of 277 village clusters will be assigned: –Year 2: 120 clusters –Years 3 and 4: 77 clusters –Year 5: 80 clusters Some villages not included in the selection –Pilot phase villages –Villages that do not have adequate water and would not benefit from training yet (eligible later)

Selection of Villages (continued) Random assignment within Water User Associations –Equitable –Ensures balance Selection conducted in public in August 2007 –Transparency Random assignment within Water User Associations –Equitable –Ensures balance Selection conducted in public in August 2007 –Transparency

Data Sources Farming Practices Survey conducted for this evaluation –Interview sample of WUA members –Fielded each year, Fall 2007 through Fall 2010 WUA databases, compiled by Irrigation PIU –Information on all WUA members –Source of outcome data at end of follow-up period Farming Practices Survey conducted for this evaluation –Interview sample of WUA members –Fielded each year, Fall 2007 through Fall 2010 WUA databases, compiled by Irrigation PIU –Information on all WUA members –Source of outcome data at end of follow-up period

Evaluation Timeline Fall 2007: Baseline Survey –Prior to training in targeted villages Fall 2008 and Fall 2009: Follow-up Surveys Summer 2009: Preliminary Evaluation –Focus on short-term outcomes Fall 2010: Final Follow-up Survey Summer 2011: Final Impact Evaluation –Include all outcome measures Fall 2007: Baseline Survey –Prior to training in targeted villages Fall 2008 and Fall 2009: Follow-up Surveys Summer 2009: Preliminary Evaluation –Focus on short-term outcomes Fall 2010: Final Follow-up Survey Summer 2011: Final Impact Evaluation –Include all outcome measures

Outcome Measures Adoption of agricultural techniques Agricultural productivity –Quantity and quality –Revenue and profits Household well-being –Consumption –Non-farm employment –Income –Poverty Changes in these outcomes for Year 2 villages compared to Year 5 villages –Difference in means is the impact Adoption of agricultural techniques Agricultural productivity –Quantity and quality –Revenue and profits Household well-being –Consumption –Non-farm employment –Income –Poverty Changes in these outcomes for Year 2 villages compared to Year 5 villages –Difference in means is the impact

Findings will Inform Future Investments Provides essential information when deciding whether to expand project activities Identifies what conditions or characteristics affect program success Provides better data for economic analysis on similar programs Contributes lessons for all future development programs Provides essential information when deciding whether to expand project activities Identifies what conditions or characteristics affect program success Provides better data for economic analysis on similar programs Contributes lessons for all future development programs

Additional Materials

Post-Harvest, Processing and Marketing Objectives: –Introduce and expand post-harvest operations that best preserve the quality of agriculture products and add value to production –Improve access to reliable information on market conditions and opportunities –Ensure compliance with food safety and quality standards Expected Results: By the end of the program 300 agribusinesses directly and 15,000 farmers indirectly will benefit from these activities. Objectives: –Introduce and expand post-harvest operations that best preserve the quality of agriculture products and add value to production –Improve access to reliable information on market conditions and opportunities –Ensure compliance with food safety and quality standards Expected Results: By the end of the program 300 agribusinesses directly and 15,000 farmers indirectly will benefit from these activities.

Improved Access to Credit Objective: Support the increase of affordable, longer-term credit to Water-to-Market beneficiaries by: –Developing the capacity of credit providers to lend efficiently in the agriculture sector –Developing the capacity of Water-to-Market beneficiaries to access and use credit effectively Objective: Support the increase of affordable, longer-term credit to Water-to-Market beneficiaries by: –Developing the capacity of credit providers to lend efficiently in the agriculture sector –Developing the capacity of Water-to-Market beneficiaries to access and use credit effectively

Training and Adoption Targets On-Farm Water Management Training60,000 farmers Higher Value Agriculture Training30,000 farmers Post-harvest Enterprises Technical Assistance 300 enterprises/ 15,000 farmers Bank Loans provided to project beneficiaries and related businesses $8.5 million in loans Adoption of improved farm water management 38,350 farmers Hectares Converted to High Value7,845 hectares

Impact Evaluation Estimating impacts involves comparing: –Outcomes with the program –Outcomes if there were no program (counterfactual) Counterfactual: What participants would have experienced if there were no program –True counterfactual is not directly observed Goal of impact study is to identify a comparison group to approximate the counterfactual Estimating impacts involves comparing: –Outcomes with the program –Outcomes if there were no program (counterfactual) Counterfactual: What participants would have experienced if there were no program –True counterfactual is not directly observed Goal of impact study is to identify a comparison group to approximate the counterfactual

Importance of the Counterfactual: An Illustrative Example Rural poverty has decreased in recent years, prior to MCA-Armenia programs In this case, we want to see how much more poverty declined because of the program Rural poverty has decreased in recent years, prior to MCA-Armenia programs In this case, we want to see how much more poverty declined because of the program

True Impacts May Be Smaller Than Observed Changes Impact A B C