1. 7 common challenges in using theory of change - and how to address them Professor Patricia Rogers BetterEvaluation Royal Melbourne Institute of Technology, Melbourne, Australia UNEG Brown Bag seminar 25 July 2014 New York www.betterevaluation.org

2. The essence of theories of change I am cutting rocks http://img359.imageshack.us/img359/7104/picture420bt2.jpg I am building a temple 2

3. What is a theory of change? A theory (or logic or model) 3 contributes to the intended or observed outcomes. of how an intervention (project, program, policy, strategy)

4. Other labels are also used Causal model Cause map Impact pathways Intervention theory Intervention framework Intervention logic Investment logic Logic model Outcomes hierarchy Outcome line Program logic Program theory Programme theory Theory-based evaluation Theory-driven evaluation Theory-of-action Theory-of-change

5. Different types of logic models RESULTS CHAIN OUTCOMES HIERARCHY RMIT University 5

6. Different types of logic models TRIPLE ROW (OR COLUMN) RMIT University 6

7. CHALLENGE #1: NO ACTUAL THEORY PROBLEM “We do stuff and stuff happens” Magic arrows STRATEGIES Create an explanatory narrative version Identify both a change theory (how change occurs) and an action theory (what we do to activate this change theory) 7

8. Create an explanatory narrative version The computer project works with motivated students from disadvantaged backgrounds who do not have access to computers. It works through a combination of increasing capacity to a threshold level, envisioning a new future, and strengthening social capital. Students develop foundational IT skills, opening up opportunities to undertake internships. Through presentations and mentoring by computer professionals and visits to university campuses, they develop a new image of their possible future. At the same time, they are creating a supportive community of students who make a safe space to learn and to make, and implement, ambitious plans about their future education and career. RMIT University 8

9. Change theories and action theories Situation analysis Theory of actionTheory of changeIntended result A tidy roomDeterrence – avoiding a negative interaction Shouting Withhold pocket money Offer a reward (money, movie tickets) Positive incentive Praise efforts

10. Situation analysisTheory of change Theory of actionIntended results Deterrence (disincentive) Shouting Tidy room Withhold pocket money Positive incentive Movie tickets Praise Change theories and action theories

11. CHALLENGE #2: IMPLAUSIBLE THEORY PROBLEM “That trick never works” STRATEGIES Seek stakeholder review Draw on relevant research literature 11

12. Draw on research-based theories 12 The COM-B system - a framework for understanding behaviour. Michie et al. Implementation Science 2011 6:42 doi:10.1186/1748-5908-6-42

13. 13 The Behaviour Change Wheel. Michie et al. Implementation Science 2011 6:42 doi:10.1186/1748-5908-6-42 Draw on research-based theories

14. CHALLENGE #3: TOO SIMPLE PROBLEM Assuming “it works” for everyone Ignoring contributions of other organizations STRATEGIES Attention to equity issues in patterns Realist analysis to identify contexts in which causal mechanisms operate Include contributions of other organizations 14

15. Include contribution of other organizations

16. CHALLENGE #4: NOT CLEAR PROBLEM “Spaghetti diagram” Everything linked to everything STRATEGIES Chunk information into 7 +/- 2 pieces Focus on key causal linkages Layer the logic model 16

17. CHALLENGE #5: UNINTENDED RESULTS IGNORED PROBLEM Negative impacts not anticipated and risk managed – or included in evaluation Additional benefits ignored Cost-shifting STRATEGIES Do a negative program theory Listen for news of unintended impacts 17

18. CHALLENGE #6: NOT USED FOR EVALUATION PROBLEM Theory of change not reflected in evaluation STRATEGIES Use logic model to identify existing information and gaps – and guide analysis Use logic model to inform strategies for causal attribution Use program logic matrix to explore criteria and standards of success 18

19. CHALLENGE #7: FORGET IT’S ONLY A MODEL PROBLEM Treat it as perfectly representing reality and fixed STRATEGIES Treat as a heuristic, not a formula or wiring diagram Expect gaps and ongoing adaptation and refinement 19

20. Sources for learning more about theories of change RMIT University 20

21. Sources for learning more about theories of change RMIT University 21

22. Are these common problems? Are there others? What are effective strategies? 1. No theory 2. Implausible theory 3. Too simple 4. Not clear 5. Unintended results ignored 6. Not used for evaluation 7. Forget it’s just a model RMIT University 22