1. Theory of Change
J-pal

2. What is Theory of Change?
How to make a Theory of Change?
Why is Theory of Change important?

3. Components of program evaluation
What is the problem?
Needs Assessment
How, in theory, does the program fix the problem?
Program Theory Assessment
Does the program work as planned?
Process Evaluation
Were its goals achieved? The magnitude?
Impact Evaluation
Given magnitude and cost, how does it compare to alternatives?
Cost Effectiveness Analysis
In the presentation “What is Evaluation”, these 5 different components of program evaluation were already explained.
Each component answers different questions.
Rossi, Lipsey, Freeman, Evaluation, a Systematic Approach

4. Components of program evaluation
¿What is the problem?
Needs Assessment
How, in theory, does the program fix the problem?
Program Theory Assessment
Does the program work as planned?
Process Evaluation
Were its goals achieved? The magnitude?
Impact Evaluation
Given magnitude and cost, how does it compare to alternatives?
Cost Effectiveness Analysis
This presentation will focus on Program Theory Assessment.
This type of evaluation assesses how, in theory, the program fixes the problem.
Theory of Change is an approach that can be used to answer this question.
Rossi, Lipsey, Freeman, Evaluation, a Systematic Approach

5. What is Theory of Change?
A comprehensive description and illustration of how and why a desired change is expected to happen in a particular context.
Maps out the “missing middle” (between what a program does and its long-term goals)
Source:
Different organizations think quite differently about the term ‘theory of change’.
In this lecture, ToC refers to a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context.
It is focused in particular on mapping out or “filling in” what has been described as the “missing middle” between what a program or change initiative does (its activities or interventions) and how these lead to desired goals being achieved.
It does this by first identifying the desired long-term goals and then works back from these to identify all the conditions (outcomes) that must be in place (and how these related to one another causally) for the goals to occur. 
J-pal

6. Cartoon illustrating the missing middle (Image credit: Sidney Harris).
J-pal

7. ToC can have different looks…
Vulnerability of households, in particular for women
Needs Assessment
Intervention
Saving for Change
Participation in the program
Replication
Assumptions
Access to credit
More savings
Group Solidarity
Intermediary outcomes
A main component of a Theory of Change is the causal chain, a graphical representation of how a program leads to its desired results. There are many different possible ways to draw such a chain, depending on the characteristics of the program and personal preferences. In this and the following slides are some examples.
Means to cope with shocks
More self-confidence
More investment
More assets
More income
Final
Outcomes
Female empowerment
Better education
Better health
Food security
J-pal

8. ToC can have different looks…
Drawn by: Oxfam America,
Freedom From Hunger

9. ToC can have different looks…

10. ToC can have different looks…

11. ToC can have different looks…

12. A close cousin: the Logical Framework
Another approach to describe how programs lead to results is the Logical Framework, also called LogFrame.
Over the last few decades there has been an ongoing debate in the international development community about which of the two is the best approach.
Although academics are still debating the relationship between the two formats, in practice there are some differences in how they are used. At the simplest level, a Theory of Change shows the big, messy “real world” picture, with all the possible pathways leading to change, and why you think they lead to change (do you have evidence, or is it an assumption?).
A Logical Framework is like zooming in on the specific pathway that your program deals with and creating a neat, orderly structure for it. This makes it easier for you and the donor to monitor program implementation.
But besides these differences, ToC and LogFrame have a lot in common. Both are approaches that can be used for Program Theory Assessment, to answer the question how the program intends to solve a particular problem.
In this course we will use ToC for this, but LogFrame is also an option.
J-pal

13. Our example of Theory of Change
Need
Input / Activity
Output
Output
In particular, we will use this format for the causal chain.
Intermediate outcome
Intermediate outcome
Intermediate outcome
Final Outcome
Final Outcome
J-pal

14. What is Theory of Change?
How to make a Theory of Change?
Why is Theory of Change important?

15. 3 steps Determining final outcomes Completing the causal chain
Identifying assumptions, risks and indicators
As explained earlier, different organizations think quite differently about the term ‘theory of change’.
There are therefore also different approaches for creating a ToC.
In this course we will use this 3-step approach.
J-pal

16. Theory of Change Example:
Incentives for Immunizations

17. Community mobilization campaign Improve healthcare worker attendance
What do you think is the most cost-effective way to increase immunization rates?
Community mobilization campaign
Improve healthcare worker attendance
Develop new vaccines, such as pneumococcal
Hold special ‘immunization camps’
Incentivize parents to immunize their children

18. Background: The Need
Every year, between 2 and 3 million people die from vaccine-preventable diseases
Only 54% of 1-2 year olds in India receive the basic package of immunizations
Only two percent of children are receiving the full immunization package
So what is the need in India :

19. Underlying reasons
Immunizations: free for all, life-saving, in theory widely available at health centers
Needs Assessment: Immunization coverage is really low - less than 5% in rural Udaipur
One possibility is that the supply channel is the problem:
Low attendance by health staff meant that heath centers were frequently closed (45% absenteeism)
Second possibility, there is a problem of demand:
High opportunity cost of going for 5 rounds of vaccination
Cultural resistance to vaccination?
Immunizations are free in India.. What is the problem?
Could supply be really poor? We have very high nurse absenteeism rates in Udaipur, so it’s possible that people are trying to get immunizations but are unable to do so
Or could demand be very low? High opportunity cost because it’s very difficult to get there, as well as potential cultural resistance to vaccinations

20. Step 1: Determining final outcomes
Final outcomes are the long term goals of a program, the ultimate change it aims to achieve.
Very important step, the rest of the ToC depends on it.
To determine the right final outcomes, a good context analysis is important
Define needs, opportunities and barriers
Identify target population
Map relevant stakeholders
Analyze broader political and economic context
J-pal

21. 2. Completing the Causal Chain
Need
Input / Activity
Output
Final Outcome
Intermed. outcome
Design & implementation
ToC’s main component, the causal chain, consists of different steps. The first three constitute the design and implementation of the program, the last two its impact.
Impact
J-pal

22. 2. Completing the causal chain
Need / problem The deficiency or problem that a program aims to solve or ameliorate.
Need / Problem
Input / Activity
Output
Output
The first step of our 3-step approach for making a ToC already gives us 2 steps or components of the causal chain, the first and the last one: the need / problem (from the context analysis) and the final outcomes.
Final outcomes
Long term goals of a program, the ultimate change it aims to achieve
Interm. outcome
Interm. outcome
Interm. outcome
Final Outcome
Final Outcome
J-pal

23. In our Immunization Example
High health worker absenteeism, low value of immunization, limited income and time
Need/ Problem
Let’s take the example of a program testing kids in math and reading and informing the parents about the results with the hope to foster their involvement.
In our example the needs would be poor school performance, and the final outcome would be improved school performance.
Increased Immunisation
Final Outcome
J-pal

24. 2. Completing the causal chain
Need / Problem
Input / Activity
Output
Output
Backwards mapping
Now we need to connect this first and last step of the causal chain. We can do that in two ways. The first way, backwards mapping, is for the ideal scenario that the ToC is made before the intervention has been defined.
Based on the final outcomes we want to achieve, we first define the necessary intermediate outcomes, then the outputs and ultimately the inputs & activities.
Interm. outcome
Interm. outcome
Interm. outcome
Final Outcome
Final Outcome
J-pal

25. 2. Completing the causal chain
Need / Problem
Input / Activity
Forward mapping
Output
Output
Although it is recommendable to make a ToC before designing a program, even for programs that are alredy being implemented it can be of great value to make a ToC, in order to verify the quality of the logic behind it, and making changes if necessary.
In that case (when the program - the inputs and the activities - are already known) it is more practical to start there and do forward mapping: inputs/activities -> ouputs -> intermediate outcomes.
Interm. outcome
Interm. outcome
Interm. outcome
Final Outcome
Final Outcome
J-pal

26. The Programme Solutions to Address Supply and Demand Side Constraints
Incentives: 1 kilo of lentils per visit + steel plates
Regular Immunization Camps
So what did Seva Mandir together with the J-PAL affiliated researchers decide to do to address both the supply and demand barriers that were limiting immunisation
To address the supply side they developed regular immunization camps. These camps happened monthly on a fixed date and a social worker would notify the surrounding community
What about the demand side issue?
(2) 1 kilo of lentils per visit + plate at the end to encourage parents to finish the whole package of vaccines.

27. Inputs and activities
Need
Input / Activity
Output
Final Outcome
Interm. outcome
Inputs and activities Program activities and the resources used for it.
… and we end with the inputs / activities.
J-pal

28. An example: Incentives for Immunization
High health worker absenteeism, low value of immunization, limited income and time
Need/ Problem
Input/ Activity
Incentives for immunization
Immunisation camps
Increased immunication
Final Outcome
J-pal

29. Outputs Outputs The direct results of program activities Need
Input / Activity
Output
Final Outcome
Interm. outcome
Outputs The direct results of program activities
… after that the outputs…
J-pal

30. An example: Incentives for Immunization
High health worker absenteeism, low value of immunization, limited income and time
Need/ Problem
Input/ Activity
Incentives for immunization
Immunisation camps
Output
Incentives are delivered
Camps are reliably open
Increased immunisation
Final Outcome
J-pal

31. Intermediate outcomes
Need
Input / Activity
Output
Final Outcome
Interm. outcome
Intermediate outcomes
Changes necessary to achieve the final outcomes.
Usually changes in:
Knowledge & beliefs
Attitudes & aspirations
Capacity & ability
Decisions, behaviors & actions
Let’s complete the causal chain of our example using backwards mapping.
So we first define the intermediate outcomes.
We refer to many of these as “Behavior responses”. But that can span from changes in knowledge and beliefs, to tangible, observable changes in actions
J-pal

32. An example: Incentives for Immunization
High health worker absenteeism, low value of immunization, limited income and time
Need/ Problem
Input/ Activity
Incentives for immunization
Immunisation camps
Output
Incentives are delivered
Camps are reliably open
Intermediate Outcome
Parents bring children to camps
Parents bring children to camps repeatedly
Increased immunisation
Final Outcome
J-pal

33. 3. Identifying assumptions, risks and indicators
Assumptions External conditions that are believed to be true (and that need to be true to achieve the desired results) Risks Unintentional negative effects caused by the program Indicators Observable signals of change
ToC is more than only the causal chain. Among its main components are also assumptions, risks and indicators.
These can be defined after but also during the construction of the causal chain, for example right before determining the inputs/activities in case of backwards mapping.
J-pal

34. An example: Incentives for Immunization
High health worker absenteeism, low value of immunization, limited income and time
Need/ Problem
Input/ Activity
Incentives for immunization
Immunisation camps
Output
Incentives are delivered
Camps are reliably open
Assumption: Parents Value incentives
Parents bring children to camps
Intermediate Outcome
Parents bring children to camps repeatedly
Assumption: Camp provides immunizations
Final Outcome
Increased immunisation
J-pal

35. Sources of Verification
Log Frame
Objectives Hierarchy
Indicators
Sources of Verification
Assumptions / Threats
Impact
(Goal/ Overall objective)
Increased immunization
Immunization rates
Household survey
Adequate vaccine supply, parents do not have second thoughts
Outcome
(Project Objective)
Parents attend the immunization camps repeatedly
Follow-up attendance
Household survey; Immunization card
Parents have the time to come
Outputs
Immunization camps are reliably open; Incentives are delivered
Number of kg bags delivered; Camp schedules
Random audits; Camp administrative data
Nurses/assistants will show up to camp and give out incentives properly
Inputs
(Activities)
Camps + incentives are established
Camps are built, functional
Random audits of camps
Sufficient materials, funding, manpower
Needs assessment
Impact evaluation
The theory of change can also be written out in a log frame.
Process evaluation

36. What is Theory of Change?
How to make a Theory of Change?
Why is Theory of Change important?
Theory of Change can have multiple purposes. It is used in different ways by different actors.
Here we will discuss the importance it can have for implementers and for evaluatorrs.
J-pal

37. Helps design the intervention
ToC thinking forces us to be results-oriented and to think about how credible our assumptions are
ToC can help implements keep a focus on results—eye on the prize.
Using the “backward mapping” approach to a Theory of Change is particularly helpful. It helps us be more open minded when determining potential solutions. We don’t want to start the exercise already psychologically committed to specific inputs.
Programme implementation folks are usually focused on activities and outputs: How many trainings have we completed? How many beneficiaries have we reached? We should move away from interventions for interventions’ sake toward development that explicitly serves a larger purpose and goal. E.g. that means I shouldn’t train partner NGOs just because I think it’s fun, but I should also be oriented toward the reason I am doing it.
Identify the best strategy to achieve a certain result.
J-pal

38. Helps design the evaluation
Generate research questions
Decide what to measure
Select the right intermediary variables to understand the “why”
ToC can help design an impact (or other type of) evaluation, by:
Generating research questions (often related to the assumptions identified in the ToC
Deciding which data to collect, to understand the “why”
By measuring the right intermediate variables, we can get “into the black box”
Allows to understand the “why”, thus giving richer policy lessons
Gives more generalizable knowledge
J-pal

39. Why is Theory of Change Important
For evaluators, reminds us to consider process
For implementers, it helps us be results oriented
What is it?
Example
Components
Assumptions
Conclusion
Increased Immuni-
zation
Incentives for Immunization
Camps are reliably Open
Parents bring children to the camps
Parents bring children to the camps repeatedly
GOAL
OUTPUT
OUTCOME
Immunization
Camps
Incentives are delivered
INPUT
You all have this theory of change diagram in handout form. On this slide it’s not important that you be able to read what is in the boxes.
The idea here is that we’ve talked quite a bit about what theory of change is. Now why is it important?
For Programme people, it helps us keep a focus on results—eye on the prize.
Programme implementation folks are usually focused on activities and outputs: How many trainings have we completed? How many beneficiaries have we reached?
We should move away from interventions for interventions’ sake toward development that explicitly serves a larger purpose and goal.
E.g., In the example above, that means I shouldn’t train partner NGOs just because I think it’s fun, but I should also be oriented toward the reason I am doing it.
For evaluators, it reminds us to consider process
Evaluators are very focused on assessing outcome change. But sometimes this is done with a blind eye to how change was achieved.
We call this the black box problem (next slide).
ASK: How many of you have heard of the black box problem before?
Can anyone take a stab at explaining the black box problem?

40. Solving the Black Box Problem
Low immunization rates
Needs Assessment
Intervention
Intervention design/Inputs
Black Box
**Adapted from Marc Shotland, JPAL
So we’ve done our needs assessment. We have designed and implemented our intervention.
But when we come back to evaluate the Programme we find that there is no increase in immunization rates.
If we didn’t do any process monitoring or process evaluation—in other words, if we didn’t track every step of our theory of change—we might not be able to figure out why the intervention didn’t work, or where it broke down.
If, as evaluators, we look only at our final outcomes or impact, we may not understand why it worked, or if it didn’t, where it broke down.
Let’s peer into the black box. Were our activities implemented as planned? Were outputs produced? Were assumptions met? Did the necessary outcomes change?
Final outcome
No increase in full immunization

41. Identifying Theory Failure vs. Implementation Failure
Inputs
Activities
Outputs
Outcomes
Goal
Identifying Theory Failure vs. Implementation Failure
Successful intervention
Inputs
Activities
Outputs
Outcomes
Goal
Implementation failure
Inputs
Activities
Outputs
Outcomes
Goal
Theory failure
**Adapted from Weiss, Evaluation, 1998, p 129
The black box problem can be described in other terms.
In a successful intervention that sees the expected outcome and goal change, our theory of change is realized. That is, inputs are sufficient to conduct the activities, the appropriate products and services are provided, our outcomes change in the way we hope and this contributes to our targeted goal.
It is not unusual, however, that we do not see the outcome or goal change that we had hoped for. Without an articulated and measured theory of change, however, we might not know why that is. Is it because our implementation process was problematic? That is, our inputs did were not sufficient/appropriate to implement the planned activities, or somehow the activities we implemented did not lead to the right outputs. This is called implementation failure.
On the other hand, maybe our implementation process was tight. Our resources were sufficient. Our activities were implemented as planned and the services provided were good. Instead, the idea behind our Programme was faulty. That is, the products and services we provided didn’t actually lead to the expect knowledge or behavior change, and so our ultimate goal was not achieved. If we have breakdown at this end of the chain (outputsoutcomesimpact) this is called theory failure.
The implications are really different because with implementation failure we can only know that our mechanism didn’t work. With theory failure we know that our idea doesn’t work. If we couldn’t distinguish between implementation failure and theory failure it would be very difficult to use evaluation results to shape policy and Programming because we wouldn’t know whether it was a bad idea or whether we had just implemented the intervention so poorly that the causal chain broke down.

42. Which of the following is a possible theory failure?
Though the programme implementers believed the nurses would hold the camps at the same time every month in each village, this did not happen.
The programme led to an increase in parents taking their children for the first immunization, but did not improve full immunization rates.
All of the above.

43. What is the Best Time to Create a Theory of Change?
At the end of Programme implementation and before commencing an evaluation
While identifying a solution to specific development challenges
Before data collection
During times of failures of Programme implementation

44. Results of the evaluation
Banerjee, Abhijit Vinayak, Esther Duflo, Rachel Glennerster, and Dhruva Kothari "Improving Immunisation Coverage in Rural India: Clustered Randomised Controlled Evaluation of Immunisation Campaigns with and without Incentives." BMJ

45. Conclusions
J-pal

46. In summary
A Theory of Change makes explicit how a program is expected to achieve impact.
Its main components are the causal chain, assumptions, risks and indicators.
Theory of Change can be used for both program as well as evaluation design.
J-pal

47. Theory of Change
J-pal