1. Beyond logical frameworks to program impact pathways CIIFAD M&E Workshop 5 November 2011  135 Emerson Hall Sunny S. Kim, MPH PhD candidate Division of Nutritional Sciences

2. Role of research in evaluation Evaluation usually focuses on an internal situation, such as collecting data about specific programs, with no intent to generalize the results to other settings and situations. Evaluation determines the merit, worth, or value of things. Evaluation asks what should be, which makes reference to the standards and criteria of evaluations. Program-related research usually conducted with to the intent to generalize the findings from a sample to a larger population. Research asks why, which indicates a theoretical basis.

3. Basic steps of evaluation Identifying program goals and objectives Mapping the program Selecting an evaluation design – Sample size, control group, sampling frame Selecting indicators Determining data collection strategy Developing data collection instruments Analyzing the data Communicating the results

4. Program theory vs. logic model Program theory explains why a program is expected to work, and a logic model illustrates a program theory. Program logic model is a picture of how your organization does its work… it links the theoretical assumptions of the program and program activities/ processes and outcomes.

5. Translating program theory… Program theory has also been referred to as: program logic or intervention logic theory-based or theory-driven evaluation theory of change theory-of-action impact pathway analysis all ways of developing a causal modal, linking program inputs and activities to intended or observed outcomes

6. Translating program theory… Program theories are often captured in a series of “if-then” statements. And between the “if” and “then,” there should be “how” and “why”, i.e. evidence or well-established connections (established theory, previous results, published research, feedback from participants).

7. IF…THEN…

8. IF…THEN… + CONTEXT

9. … into logic models a commonly-used tool for illustrating program theory. a picture of describing the sequence of activities to bring about change and how these activities are linked to the results the program is expected to achieve. often presented in a flow chart that illustrates the linkages between program components and outcomes: InputsResources, supplies, staff ActivitiesProcesses, techniques, tools, events, technology, actions OutputsDirect results of activities (size and/or scope) OutcomesChanges in attitudes, behaviors, knowledge, skills, status, or level of functioning ImpactOrganizational, population, community, and/or system level changes (improved conditions, increased capacity, and/or policy changes)

10. Program Logic Model Template

11. … into logic models “Logic models come in as many sizes and shapes as the programs they represent.” [W.F. Kellogg Foundation, 2001, p.7] No best logic model, only the model that fits your program best and provides the information you need in the format that is most helpful.

12. Why use a logic model? Design and planning  helps to explain and illustrate program concepts and approach for key stakeholders (advocate and train); finds “gaps” in the theory or logic of a program and work to resolve them. Implementation and management  focuses attention on the most important connections between actions and results for monitoring, improved programming, and reporting. Evaluation, communication, and marketing  presents progress toward goals; provides a way to involve and engage stakeholders in the design, processes, and use of evaluation.

13. Why use a logic model? Donor requirements: Indicators Means of verification Assumptions Goals Objectives Outputs Activities Resources Goal: Health status improved Strategic Objective: Use of key services and behaviors improved Intermediate Result 1: Access and availability of services and supplies increased Intermediate Result 2: Quality of services improved Intermediate Result 3: Demand for key services and behaviors increased Intermediate Result 4: Social and policy environment enabled Log frame (OECD/DAC, USAID, CIDA…) Results framework (USAID…)

14. Logic model example

15. Intervention model of Programa de Nutrición Infantil (PNI) Maternal and child health and nutrition Small-scale economic activities Strengthening of civil society Basic sanitation and sanitary education Improved health and nutrition knowledge and practices Improved development plans of CBOs in health and sanitation Improved basic sanitation Increased access to health services Improved health and nutritional status of children <3 years Improved health and nutritional status of pregnant women Strategies: Information Education Social communication FINAL OUTCOMES INTERMEDIATE OUTCOMES COMPONENTS

16. Intermediate Result 1: Improved nutrition, health knowledge and practices of those attended in the target communities Strategic Objective: Improved health and nutritional status of children <3 years and pregnant women Intermediate Result 3: Improved basic sanitation in the target communities Intermediate Result 4: Community-based organizations applied their improved knowledge on health and sanitation in the formulation of their development plans GOAL: Reduced rates of child malnutrition Intermediate Result 2: Increased access to health services of the target population Results framework of Programa de Nutrición Infantil (PNI)

17. From boxes to arrows… While “filling in” the logic models with the program components, we must not neglect the mechanisms and linkages… don’t forget about your program theory or impact pathways.

18. “The theme/issue is the training that you give through the participatory methdology to the mothers. If you just give the theory, it is the theme. The workshop is the practice. So if that day, for example, the mothers didn’t bring food to prepare their workshop, then the [health] promoter only gives the theory. They discuss, socialize, and receive only the theories in the small groups… Then the workshop returns another day to learn about doing. For example, the health themes, they make their workshop do dramatizations. The mothers do their dramatizations, it tells the priority, what happens and everything, and so they are going to learn.” [Regional level, ex-ADRA adviser] Description of mothers workshops

19. Impact pathway of mothers workshops ?

23. Main references Kellogg Foundation. (2004). Logic model development guide: Using logic models to bring together planning, evaluation, and action. Battle Creek, MI: W.K. Kellogg Foundation. Rogers, P.J. (2000). Program theory: Not whether programs work but how they work. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (Eds), Evaluation models, 209-232. Boston, MA: Kluwer Academic Publishers. Rogers, P.J. (2008). Using program theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1): 29-48. Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: teaching and training guide. Madison, WI: University of Wisconsin-Extension.