1. Making Causal Claims in Non-Experimental Settings
IPDET 2016
Ottawa
John Mayne, PhD
Advisor on Public Sector Performance
I hope to leave you with a few ideas of how to making cause-effect claims in non-experimental settings.

2. The Context
Interventions are aimed at bringing about development impacts
Interventions are not acting on their own. There may be:
other events and conditions at play
other interventions at work
other relevant contextual factors
Development impacts result from a mix of such actions and context

3. The challenge
We still need to understand and make credible claims about the contribution interventions (programs and projects) are making to development outcomes and impacts
Experimental designs for these interventions often not feasible or practical

4. The Issue
In this context, what can we say about the relationship between the intervention and observed development results?
The intervention ‘caused’ the results?
The intervention made a difference?
The intervention contributed to the results?
The intervention can be attributed to a specific net result?
What makes sense?

5. Conceptualizing Causality
What kind of causal relation exists then between a development intervention (X) and an impact (Y)?
Can we say X causes Y? No
Is X necessary for Y? No
Is X sufficient for Y? No
But we clearly want to make some causal link between the intervention and the impact

6. Conceptualizing Causality
As might be imagined, these conundrums have been well addressed in the vast literature on causality. There is an answer!
An intervention works as part of a broader causal package of other events and conditions. And if it works, then this causal package is indeed sufficient to bring about the impact.
Further, if the intervention is ‘working’—doing its job—then it is an essential part of this causal package.

7. INUS Causality
This is the very useful concept of INUS causality and INUS conditions.
An Insufficient but Necessary part of a condition that is itself Unnecessary but Sufficient for the occurrence of the effect (Mackie 1974).
Mackie argued that much of the time, when we are talking about causality we are in fact talking about INUS causality.

8. Intervention Causality
Thus, an intervention “made a difference” when:
The intervention causal package was sufficient to bring about the impact, and
The intervention was a necessary component of the causal package
The intervention in this case is a contributory cause. On its own it is neither necessary nor sufficient.
Smoking and lung cancer is an example.

9. The Intervention as Trigger
An intervention then is one among several ‘causes’. But is that all? We probably expect more, that the intervention:
acts as a trigger to start the causal chain (the spark that lights the fire)
and may act as sustaining support for change along the way (gasoline to keep the fire going)
A principal contributory cause

10. Meaningful Causal Questions
Has the intervention made a difference?
Is the intervention a contributory cause?
What contribution has the intervention made?
Why has the impact occurred?
How did the causal factors bring about the result?
What was the context and the mechanisms?
What role did the intervention play?
The question ‘What contribution did the intervention play?’ is asking about whether the intervention played, for example, a trigger role or a more modest role, with something else as a trigger.

11. Demonstrating Contributory Causes
How then to show that the intervention made a difference?
Connecting to theory-based evaluation approaches
Sufficiency through generative (process) causality approaches
One approach: contribution analysis
There are other approaches:
Other T-B approaches
QCA

12. Theories of Change Results chains/logic models
Impact pathways — the sequence of steps to impact
Theories of change — pathways with causal link assumptions
Causal link assumption — events and/or conditions needed to make the causal link work

13. intermediate outcomes
Traditional Theory of Change
Impact
intermediate outcomes
Immediate outcomes
outputs
activities

14. A Generic Useful Theory of Change
Wellbeing
Wellbeing Assumptions  
Direct
Benefits
Direct Benefits Assumptions
Behaviour changes
External Influences
Behaviour Change Assumptions
Capacity changes in knowledge, attitudes skills, & opportunities
Note the External Influences
Can also include possible Unintended Effects in the ToC (not shown)
Key to behaviour change models is that all of the capacity change elements ae needed to bring about behaviour change.
Capacity Change Assumptions
Reach &
Reaction
Reach Assumptions
Timeline
Activities and Outputs with respect to beneficiaries

15. Theories of Change and Causal Packages
Theories of change are causal packages, and more:
ToC identify supporting factors (assumptions)
ToC set out the relationship between the supporting factors and the intervention
ToC is a model of the intervention as a contributing cause

16. A Generic Useful Theory of Change
Wellbeing
Wellbeing Assumptions  
Direct
Benefits
Direct Benefits Assumptions
Behaviour changes
External Influences
Behaviour Change Assumptions
Capacity changes in knowledge, attitudes skills, & opportunities
Note the External Influences
Can also include possible Unintended Effects in the ToC (not shown)
Key to behaviour change models is that all of the capacity change elements ae needed to bring about behaviour change.
Capacity Change Assumptions
Reach &
Reaction
Reach Assumptions
Timeline
Activities and Outputs with respect to beneficiaries
Causal package

17. Four approaches to causal attribution
Regularity frameworks that depend on the frequency of association between cause and effect - basis for statistical approaches
Counterfactual frameworks that depend on the difference between two otherwise identical cases –basis for experimental and quasi experimental approaches
Comparative frameworks that depend on combinations of causes that lead to an effect - basis for ‘configurational’ approaches, such as QCA
Generative frameworks that depend on identifying the causal links and ‘mechanisms’ that explain effects –basis for ‘theory based’ and ‘realist’ approaches.

18. Generative Causation Process or mechanistic causality
Tracing the links in between causal events
Everyday causality: auto mechanic, air crashes, forensic work, doctors
The basis for theory-based evaluation approaches
The causal analysis approach that is behind contribution analysis.
Concluding on cause and effect by tracing the smaller intermediate steps between the cause and the effect.

19. Contribution analysis: the practice
Set out the causal question
Critically develop the expected (robust) theory of change
Gather the existing evidence verifying (or not) the theory of change
Assess the resulting contribution story
Seek out additional evidence
Revise & strengthen the contribution story
What is the attribution question? [Not all questions are useful to pursue.]
Traditional causality questions (Often don’t make sense in multi-cause situations)
Has the program caused the outcome?
To what extent quantitatively has the program caused the outcome?
Contribution questions
Has the program made a difference? That is, has the program made an important contribution to the observed result? Has the program influenced the observed result?
Why has the result occurred? What role did the intervention play?
Management questions
Is it reasonable to conclude that the program has made a difference?
What does the preponderance of evidence say about how well the program is making a difference?
What conditions are needed to make this type of program succeed?
Develop a robust ToC based on documents, stakeholder views, prior research
Gather existing evidence on the links and assumptions in the ToC (M&E, prior studies)
Assessing the contribution story: Where are the weaknesses? Do a theory of change analysis. Identify weak links; look for prior research; which are contested?
Additional evidence: refined ToC, more measurement
Revise and strengthen: design survey and case study tools based on ToC; triangulate

20. Contribution Analysis
CA shows that an intervention is a contributory cause:
The ToC is robust and hence credible
The expected result occurred
The causal package is sufficient
supporting factors (assumptions) occurred
The intervention is necessary for the package to be sufficient
In short, the robust ToC is verified
that the expected result occurred
that the causal package is sufficient
the supporting factors—the assumptions for each link in the theory of change—have occurred and together provide a reasonable explanation for the occurrence of the results, and
any other identified supporting factor that occurred has been included in the causal package, thereby revising the theory of change,
any plausible rival explanations—external causal factors—have been accounted for, and
that the intervention is a necessary part of the causal package
without the activities and outputs of the intervention, the supporting factors alone are not sufficient to bring about the result—knowing that with the intervention the causal package was sufficient.

21. Contribution Analysis in Complex Settings
CA can be data and analysis demanding
In complex settings, ToCs are much more complex and challenging
Applying CA in those settings not straightforward
Stay tuned!!

22. Main Messages
We expect most interventions are principal contributory causes
The intervention causal package is sufficient & the intervention is essential to the package
Want to also know why the impact occurred; to be able to explain
ToC are models of the intervention as a contributory cause
Contribution analysis and other T-B approaches can be used to show cause-effect relationships

23. Some References
DFID (2012). Broadening the Range of Designs and Methods for Impact Evaluation, London. Available at
Mayne, J. (ed) (2012). Contribution Analysis. Evaluation, Special Issue, 18(3).
Mayne, J. (2008). Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16. Available at
Mayne, J. (2012). Making Causal Claims, ILAC Brief No. 26: The Institutional Learning and Change Initiative. Available at
Mayne (2015). Useful Theory of Change Models. Canadian Journal of Evaluation, 30 (2),