Really Using (Useful) Theories of Change IPDET 2013 John Mayne, Advisor on Public Sector Performance

Using Theories of Change 1.Understanding and agreeing on interventions 2.Addressing equity issues 3.Designing interventions 4.Ex ante evaluation 5.Designing monitoring systems 6.Managing adaptively 7.Reporting performance 8.Designing evaluation questions, methods and tools 9.Making causal claims 10.Generalizing to other locations

Behaviour changes Activities and Outputs with respect to beneficiaries External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction Direct Benefits Impacts A Generic Results Chain Timeline

Behaviour changes Activities and Outputs with respect to beneficiaries External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction Direct Benefits Impacts A Generic Results Chain Timeline Note No labeling of outcomes types Focus on program theory Timeline Built on capacity and practices changes Explicit recognition of reach Feedback loops

Behaviour changes Activities and Outputs with respect to beneficiaries External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction Direct Benefits Impacts Assumptions and Risks Timeline Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes? Risks: Risks to the link not occurring. Other Explanatory Factors: Peer or trend pressure; other interventions An assumption is NOT a description of the causal link

Behaviour changes Activities and Outputs with respect to beneficiaries External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction Direct Benefits Direct Benefits Assumptions & Risks Other explanatory factors Behaviour Change Assumptions & Risks Other explanatory factors Capacity Change Assumptions & Risks Other explanatory factors Reach Assumptions & Risks Impacts Impact Assumptions & Risks Other explanatory factors A Generic Beneficiaries Theory of Change Timeline

timeline Awareness and skills training provided to teachers on girls’ education needs External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Girls learning improves Results Chain for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Improved education outcomes for girls Teachers provide girls with more empathetic and supportive teaching in schools Intervention to improve educational outcomes for girls by providing gender sensitivity training to teachers, in a region where girls education has not been a priority.

Girls learning improves Other Factors for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Teachers provide girls with more empathetic and supportive teaching in schools Parents and community support Time for studying Access to schools Accommodation in schools

timeline Awareness and skills training provided to teachers on girls’ education needs External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Girls learning improves Unintended effect: Boys’ education suffers Girls’ Behaviour Assumptions & Risks  Girls have access to schools  Schools can adequately accommodate girls Other explanatory factors  Internet access becomes widely available Theory of Change for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Improved education outcomes for girls Teachers provide girls with more empathetic and supportive teaching in schools Girls’ Direct Benefit Assumptions & Risks  Girls have time to study  Girls education supported by parents and community  Focus on girls is not disruptive to boy’s education Other explanatory factors  Internet access becomes widely available

Other factors Seems clear that more than training teachers may be required, such as: Getting girls to school Parental and community support Schools accommodate girls Teachers support girls education Empathetic teaching is rewarded Focus on and resources for girls’ education are ongoing (outcomes take time)

timeline Awareness and skills training provided to teachers on girls’ education needs External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Girls learning improves Unintended effect: Boys’ education suffers Teacher Behaviour Change Assumptions & Risks  Training relates to the local conditions  Teachers are assessed in part on their application of the training Capacity Change Assumptions and Risks Most teachers want to better education for girls Reach Assumptions & Risks  Only teachers with a predisposition to the importance of girls’ education attend Impact Assumptions & Risks  Resources and will continue for multi-year involvement Theory of Change for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Improved education outcomes for girls Teachers provide girls with more empathetic and supportive teaching in schools Girls’ Behaviour Assumptions & Risks  Girls have access to schools  Schools can adequately accommodate girls Other explanatory factors  Internet access becomes widely available Girls’ Direct Benefit Assumptions & Risks  Girls have time to study  Girls education supported by parents and community  Focus on girls is not disruptive to boy’s education Other explanatory factors  Internet access becomes widely available

Equity: Distribution of impacts As set out so far, ‘girls’ is an general group But among ‘girls’, there are worst-off girls, who may need special attention (e.g., access, income, conservative parents) And as already noted, there may be unintended impacts on boys To get at these distributional affects, we can build nested theories of reach: – A ToC for a particular reach group within an overall ToC And can then tell reach impact stories about the effects and impacts of different reach groups

timeline Awareness and skills training provided to teachers on girls’ education needs External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Girls learning improves Unintended effect: Boys’ education suffers Teacher Behaviour Change Assumptions & Risks  Training relates to the local conditions  Teachers are assessed in part on their application of the training Capacity Change Assumptions and Risks Most teachers want to better education for girls Reach Assumptions & Risks  Only teachers with a predisposition to the importance of girls’ education attend Impact Assumptions & Risks  Resources and will continue for multi-year involvement Theory of Change for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Improved education outcomes for girls Teachers provide girls with more empathetic and supportive teaching in schools Girls’ Behaviour Assumptions & Risks  Girls have access to schools  Schools can adequately accommodate girls Other explanatory factors  Internet access becomes widely available Girls’ Direct Benefit Assumptions & Risks  Girls have time to study  Girls education supported by parents and community  Focus on girls is not disruptive to boy’s education Other explanatory factors  Internet access becomes widely available Teachers Theory of Reach Girls Theory of Reach Worst-Off Girls Theory of Reach

timeline Awareness and skills training provided to teachers on girls’ education needs External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Girls learning improves Unintended effect: Boys’ education suffers Teacher Behaviour Change Assumptions & Risks  Training relates to the local conditions  Teachers are assessed in part on their application of the training Capacity Change Assumptions and Risks Most teachers want to better education for girls Reach Assumptions & Risks  Only teachers with a predisposition to the importance of girls’ education attend Impact Assumptions & Risks  Resources and will continue for multi-year involvement Theory of Change for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Improved education outcomes for girls Teachers provide girls with more empathetic and supportive teaching in schools Girls’ Behaviour Assumptions & Risks  Girls have access to schools  Schools can adequately accommodate girls Other explanatory factors  Internet access becomes widely available Girls’ Direct Benefit Assumptions & Risks  Girls have time to study  Girls education supported by parents and community  Focus on girls is not disruptive to boy’s education Other explanatory factors  Internet access becomes widely available

timeline Awareness and skills training provided to teachers on girls’ education needs External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Girls learning improves Unintended effect: Boys’ education suffers Teacher Behaviour Change Assumptions & Risks  Training relates to the local conditions  Teachers are assessed in part on their application of the training Capacity Change Assumptions and Risks Most teachers want to better education for girls Reach Assumptions & Risks  Only teachers with a predisposition to the importance of girls’ education attend Impact Assumptions & Risks  Resources and will continue for multi-year involvement Theory of Change for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Improved education outcomes for girls Teachers provide girls with more empathetic and supportive teaching in schools Girls’ Behaviour Assumptions & Risks  Girls have access to schools  Schools can adequately accommodate girls Other explanatory factors  Internet access becomes widely available Girls’ Direct Benefit Assumptions/Risks  Girls have time to study  Girls education supported by parents and community  Focus on girls is not disruptive to boy’s education Other explanatory factors  Internet access becomes widely available Supporting activities

timeline Awareness and skills training provided to teachers on girls’ education needs External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Girls learning improves Unintended effect: Boys’ education suffers Teacher Behaviour Change Assumptions & Risks  Training relates to the local conditions  Teachers are assessed in part on their application of the training Capacity Change Assumptions and Risks Most teachers want to better education for girls Reach Assumptions & Risks  Only teachers with a predisposition to the importance of girls’ education attend Impact Assumptions & Risks  Resources and will continue for multi-year involvement Theory of Change for Enhancing Education Outcomes for Girls Girls more engaged in and wanting an education Improved education outcomes for girls Teachers provide girls with more empathetic and supportive teaching in schools Girls’ Behaviour Assumptions & Risks  Girls have access to schools  Schools can adequately accommodate girls Other explanatory factors  Internet access becomes widely available Girls’ Direct Benefit Assumptions/Risks  Girls have time to study  Girls education supported by parents and community  Focus on girls is not disruptive to boy’s education Other explanatory factors  Internet access becomes widely available Supporting activities Teachers Theory of Reach Girls Theory of Reach Parents Theory of Reach

Worst-Off girls learning improves Unintended effect: Boys’ education suffers Worst-Off Girls’ Behaviour & Direct Benefit Change Assumptions & Risks  Worst-off girls have access to schools  Worst-off girls education supported by parents and community  Worst-off girls have time to study  Schools can adequately accommodate girls  Focus on girls is not disruptive to boy’s education Other explanatory factors  Internet access becomes widely available Impact Assumptions & Risks  Resources and will continue for multi-year involvement Theory of Reach for Enhancing Education Outcomes for Worst-Off Girls Worst-Off girls more engaged in and wanting an education Improved education outcomes for worst- off girls Teachers provide (all) girls with more empathetic and supportive teaching in schools Worst-off girls are provided transportation to schools Engaged with parents & communities with worst-off girls Support Activities intervention activities results timeline Engagement with governments Engagement with NGOs Partner Activities Nested theory of reach for access by worst-off girls

Reach & Reaction Capacity Changes Behaviour Changes Impact Direct Benefits Assumptions & Risks Other Explanatory Factors Behavioural Change Assumptions & Risks Other Explanatory Factors Capacity Change Assumptions & Risks Other Explanatory Factors Reach Assumptions & Risks External Influences Supporting Activities and Outputs Direct Benefit Changes Impact Assumptions & Risks Other Explanatory Factors Timeline Activities and Outputs Timeline A Basic Theory of Change for Comprehensive Interventions Reach, Capacity change Behavioural Change

Reach & Reaction Capacity Changes Behaviour Changes Impact Direct Benefits Assumptions & Risks Other Explanatory Factors Behavioural Change Assumptions & Risks Other Explanatory Factors Capacity Change Assumptions & Risks Other Explanatory Factors Reach Assumptions & Risks External Influences Supporting Activities and Outputs Direct Benefit Changes Impact Assumptions & Risks Other Explanatory Factors Timeline Activities and Outputs Timeline A Basic Theory of Change for Comprehensive Interventions Reach, Capacity change Behavioural Change Different Beneficiary Nested ToC Different Intermediary Nested ToC

Using Theories of Change 5.Designing monitoring systems – likely need more than just a focus on results—the whole ToC, including assumptions and risks 6.Adaptive management – Identify what to be monitored – On a regular basis, reflect on what is being observed and on your ToC – Revise your intervention activities and your ToC as you learn

Using Theories of Change 7.Reporting performance – A verified ToC is your performance story 8.Designing evaluation questions, methods, tools – Identifies questions – Adds power to case studies by confirming the ToC – Basis for surveys et al

Behaviour changes Activities and Outputs with respect to beneficiaries External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction Direct Benefits Impacts There are potential Evaluation Questions associated with each box and arrow Timeline Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes? Risks: Risks to the link not occurring. Other Explanatory Factors: Peer or trend pressure; other interventions Behaviour Change What behavioural (practice) changes are observed? Causal Question To what extent are the behavioural changes the result of the capacity changes? Capacity Change What capacity) changes are observed? Assumption Questions Did the events/conditions occur? Did risks materialize? Are other explanations plausible?

Using Theories of Change 9.Making causal claims – Based on generative causality – ToC are models of causality – Approaches such as contribution analysis demonstrates causality

Theories of Change as Causal Packages ToC is a model of the intervention as a contributing cause – ToC identifies supporting factors (assumptions) and confounding factors (risks) – Intervention activities plus the assumptions are sufficient to bring about the intended impacts – Explains how and why intended impacts are expected to occur

Reach & Reaction Capacity Changes Behaviour Changes Impact Direct Benefits Assumptions & Risks Other Explanatory Factors Behavioural Change Assumptions & Risks Other Explanatory Factors Capacity Change Assumptions & Risks Other Explanatory Factors Reach Assumptions & Risks External Influences Supporting Activities and Outputs Direct Benefit Changes Impact Assumptions & Risks Other Explanatory Factors Timeline Activities and Outputs Timeline A Basic Theory of Change Causal package

Contribution Analysis Using a generative perspective on causality, CA shows that an intervention is a contributory cause: The expected result occurred The ToC (causal package) is sufficient – supporting factors (assumptions) occurred and any other supporting factors have been included in the ToC – plausible rival explanations have been accounted for The intervention is necessary for the ToC to be sufficient And can explore the role the intervention played, such as a trigger

Using Theories of Change 10.Generalizing – What works where and when? – ToC explicitly includes the context and the causal mechanisms – Confirming a ToC provides significant evidence on if and how the intervention would work elsewhere

Messages ToC are worth the effort to develop Not a throw away, but a powerful tool to use in evaluation and in managing Lots of fun … and lots of debate!

References Mayne, J. (forthcoming). Using Theories of Reach to Enhance Equity Considerations in Evaluation. In Evaluation and Equity. K. Forss and M. Marra, Eds: Transaction. Mayne, J. (2013). Making Causal Claims. Connections, June. Available at june.pdf. june.pdf Mayne, J. (2012). Contribution Analysis: Coming of Age? Evaluation 18(3), Mayne, J. (2012). Making Causal Claims, ILAC Brief No. 26: The Institutional Learning and Change Initiative. Available at ilac.org/files/publications/mayne_making_causal_claims_ilac_brief_26.pdfhttp:// ilac.org/files/publications/mayne_making_causal_claims_ilac_brief_26.pdf Stern, E, Stame, N., Mayne, J., Forss, K, Davies, R. and Befani, B. (2012). Broadening the Range of Designs and Methods for Impact Evaluation, Working Paper 38, Department for International Development, London. Available at Mayne, J. (2008). Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16. Available at ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdfhttp:// ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf