Evaluation of Nutrition-Sensitive Programs* Presented by: Deanna Olney, PhD October 19, 2016 * Olney, Leroy and Ruel. Evaluation of Nutrition Sensitive Interventions. Chapter 44. In: De Pee, Taren and Bloem (eds.) Nutrition and Health in a Developing World. 3rd Edition (forthcoming). * Leroy, Olney and Ruel. Evaluation of Nutrition Sensitive Interventions. AOTR

Background The Lancet Series on Maternal and Child Nutrition calls for greater investments in large-scale nutrition- sensitive programs. Rigorous evidence of their effectiveness, how they work and cost-effectiveness is scant attributed to: Weaknesses in program design and implementation Poor evaluation designs. There is a need for rigorous, theory based-evaluations that can provide evidence of: What works to improve nutrition How programs work to achieve nutrition impacts What other impacts programs have Cost and cost-effectiveness of achieving impacts

Key challenges: Program complexity Complex in design and implementation: Multiple goals, inputs, pathways of impacts Span across different sectors (e.g. health, agriculture, education) Within each program intervention potential variability in: Delivery (quantity and quality) Utilization Adherence to the program protocol

Key challenges: Long time frames and impact pathways Long time frames for implementation and evaluation: Long pathways from program inputs to impact Implement program at the level of quality desired Achieve meaningful impacts on some biological outcomes such as child growth Design and carry out a rigorous and comprehensive evaluation … vs often short time frames imposed by donors

Priorities of implementers: Priorities of evaluators: Key challenges: Differing priorities, expectations, incentives, and perceptions Priorities of implementers: Deliver a high quality program Meet targets within the specified budget and time frame Priorities of evaluators: Rigorously evaluate the program Answer key questions related to what impact was achieved why or why not it was achieved and at what cost Misperceptions that can undermine trust: Process evaluation viewed as evaluating the performance of the program implementers vs generating evidence to improve programming and understand impacts Cost studies can be perceived as auditing the program’s finances

Key challenges: Implementation constraints vs. evaluation rigor TNFP Control Regular program  

Key challenges: Assessing benefits beyond targeted beneficiaries Potential “spillover” of benefits beyond the targeted beneficiaries: E.g. siblings, neighbors, etc. Adds complexity: Capturing all impacts of nutrition-sensitive programs Measuring the cost-effectiveness of programs

Two essential ways to address these challenges Build a strong partnership and collaboration between implementers and evaluators Design and use a rigorous and comprehensive evaluation framework: In-depth understanding of the program, program theory, program impact pathways, rigorous comprehensive evaluation design

Comprehensive evaluation: Embrace the complexity of the program HKI partners with local NGOS Village Model Farms (VMF) established and VMF owners trained Small animal production established Increased production of micronutrient-rich fruits and vegetables   Women’s groups established Agriculture inputs (e.g. seeds, saplings and poultry distributed) Provision of agriculture training Provision of nutrition, health and hygiene education Improved and developed gardens established Increased poultry production Increased household consumption Increased Income Improved maternal and child health and nutritional status Health, hygiene and nutrition education received and understood by beneficiaries Improved child care and feeding practices Outcomes Agriculture training received and understood by beneficiaries Work with local authorities (e.g. village chief) Village Health Support Group identified and trained Improved women’s empowerment Beneficiaries’ agriculture knowledge improved Outputs Inputs Women’s increased control over resources resource Impacts Processes Beneficiaries’ health, hygiene and nutrition knowledge improved Cost study Impact evaluation Process + impact evaluation What is the impact of the program? How and why does the program (not) have an impact? Cost of the program? Complex in design and implementation: Multiple goals, multiple inputs, multiple pathways of impacts, multiple outcomes and impacts. Address both the underlying and direct causes of undernutrition Span across different sectors (e.g. health, agriculture, education), requiring coordination and integration. Within each program intervention potential variability in: Delivery (quantity and quality) Utilization Adherence to the program protocol

Impact evaluation: Experimental designs Designed to assess program impacts at different levels Uses quantitative methods Ideally conducted before the program starts and at the end of the program but can include intermediary time points Requires a valid counterfactual and appropriate target groups, duration and indicators (with adequate sample sizes for each) Experimental (or randomized) designs considered gold standard for impact evaluations. Randomization: individual or group (cluster) level. *If* done well, one can assume: that both groups are comparable; that the only difference between the groups is the program; that the control group provides a valid counterfactual for the intervention group exposed to the program. differences found in the outcomes of interest attributable to the program. Consider time and timing in the evaluation design. Essential to establish a valid counterfactual.

Impact evaluation: Invalid counterfactuals Stunting (%) Impact (pp) 0.0 -5.0 Randomized design -5.5 pp Before-and-after + 0.2 pp With-and-without - 10.5 pp

How and why does the program (not) have an impact? Two ways to assess how impact is achieved : Measurement of intermediary measures (outcomes) in impact study Process evaluation Can also be used to improve on-going or future programs Solid understanding of program theory and program impact pathways is key

Process evaluation Examines the primary inputs, processes, outputs and outcomes along the primary program impact pathways Conducted about one year after program implementation begins and can be repeated Use a mix of quantitative and qualitative methods to examine the different steps along the impact pathways Include different key stakeholders (e.g. program implementers and beneficiaries) and different program delivery points Share the results: Workshop with implementers Identify what is feasible to improve and how improvements can be made

Cost study Designed to estimate the overall cost of the program, the cost of the main program components and its cost-effectiveness Can also be used to estimate savings or costs associated with adding, changing or dropping program activities or beneficiaries One method that can be used is the Activity Based Costing Ingredients (ABC-I) approach. Using the program impact pathways: provide detailed description of all program activities. Identify the program’s main activities and “ingredients” needed for each. Define the unit cost algorithms, i.e. the different types, quantities and costs of the “ingredients” necessary for each activity. Calculate cost of each program activity and of full program.

In summary Evidence of what works in nutrition-sensitive programs, how and at what cost is extremely limited. Evaluation of nutrition-sensitive programs is complex but necessary. Strong partnerships between program evaluators and implementers are needed to carry out rigorous program evaluations and to ensure that results are used to improve programming. Comprehensive evaluations should include rigorous impact and process evaluations and when possible cost studies. Guidance for future investments requires strong evidence from rigorous, theory-based comprehensive evaluations of different nutrition-sensitive program models that bring together interventions from a variety of sectors.