1. Deborah DiLiberto, Charles Opondo, Diana Elbourne, Elizabeth Allen
Mediation Analysis to Explore Causal Mechanisms in Trials of Complex Interventions
Deborah DiLiberto, Charles Opondo, Diana Elbourne, Elizabeth Allen

2. Evaluating Complex interventions
MRC 2014 Guidance – RCTs should be complemented by process evaluations to provide evidence of the causal mechanisms that produce intervention effects
Process evaluation recommendation includes:
Mechanisms of change: causal processes underlying the intervention theory of change
Theory of change / Logic model: a description of how the intervention inputs, processes and context are hypothesised to produce the intended outcome
How to combine these in a comprehensive assessment of the intervention?

3. Statistical mediation analysis
MRC guidelines suggests that statistical mediation analysis can be used to assess mechanisms and provide “valuable insights into how an intervention produces impacts” (Moore et al. 2014:43) The aim of mediation analysis is to isolate the causal mechanisms through which the intervention produces the outcome of interest.

4. Approaches to mediation analysis
The traditional framework for mediation analysis applies structural equation modelling (SEM) based on the approach popularised by the seminal work of Baron and Kenny (1986).
Recent advances in mediation theory have shown the SEM approach has theoretical limitations
An alternative nonparametric approach is based on the ‘potential outcomes framework’ and applies the logic of counterfactuals to identify direct and indirect effects (Holland 1986; Rubin 1974; Imai et al 2011)
SEM – approach is based on a linear framework; it is extremely limited for any non-linear models

5. Potential Outcomes Framework
The counterfactual framework is familiar in traditional epidemiological analyses as it forms the inference framework for RCTs
A mediator is potential until the intervention is implemented. Once the intervention is implemented, the mediator in the intervention arm can be observed.
In the counterfactual scenario, participants are exposed to the intervention, but the mediator remains unobserved and represents what would have been observed had participants been in the control arm.
The indirect effect of the mediator on the outcome is the difference between the observed and the unobserved counterfactual outcomes
The direct effect is the effect of all other potential mediators
Unlike RCTs it is not possible to observe the counterfactual mediator
Observed mediator data needs to be connected to the unobservable counterfactual mediator quantities to compare the observed and counterfactual scenarios
To make this connection, strong assumptions are needed - the assumptions of sequential ignorability

6. Assumptions of sequential ignorability
Assumption 1 requires that assignment to the intervention and control trial arms is unaffected by potential mediators and outcomes.
This is satisfied when participants are randomised; mediators do not influence how participants are randomised.
Assumption 2 requires that the mediator must be distributed between trial arms as if it were randomised
This is satisfied when there is no unmeasured confounding anywhere along the causal pathway
Therefore the complete set of covariates that could confound the relationship between the mediator and outcome must be measured and these covariates must not be affected by the intervention.
1. No confounding
2. to ensure that there is no confounding of the relationship between the intervention and outcome, and between the mediator and outcome. This assumption must be evaluated through statistical sensitivity analyses which can provide an indication of whether results may be violating sequential ignorability (Keele 2015). - SEM approach does not permit sensitivity analyses
The validity of commonly used mediation analysis based on structural equation modeling (SEM) also critically relies upon this sequential ignorability assumption and isn’t

7. Motivating case
PRIME: A complex intervention to improve the management of malaria at rural health centres in Uganda
Intervention design and evaluation informed by MRC framework:
Cluster RCT
To evaluate the population-level the impact of PRIME on malaria-related health outcomes in children
(Staedke et al. 2013)
Mixed methods study
To assess the implementation, change processes, context, and wider impact of PRIME
(Chandler et al. 2013)

8. The PRIME intervention
PRIME intervention to improve care for malaria at health centres in Uganda
Training health workers in fever case management with RDTs
Training in-charges in health centre management
Training health workers in patient-centered services
Support of supply of AL & RDTs
“Complex health
services intervention”

9. PRIME Logic Model

10. Assessment of mechanisms
To evaluate the PRIME intervention theory of change by assessing causal mechanisms using statistical mediation analysis
Operationalisation of the logic model into conventions necessary for mediation analysis
directed acyclic graph - all pathways between variables must be drawn in a single forward direction
temporal primacy - all mediators must follow in a forwards chronological order from the exposure variable
Mediation analysis of logic model pathways following Imai et al’s (2011) potential outcomes framework
a mediator farthest from the exposure could not have influenced a mediator closer to the exposure

11. Operationalised logic model
Intervention
Primary outcomes
Mechanisms
Community
Health centre
Anaemia
Health worker attitudes
PRIME Intervention
Patient satisfaction
Appropriate treatment of fever
Stocks of AL
Stocks of mRDTs

12. Analysis and Results
Initially looked at the effect of the intervention on the mechanism measures - some evidence of impact of the intervention on RDT stock outs and health worker attitudes
Looked at mediated logic model pathways- no evidence of any mediated effects
The strong assumptions required in mediation analysis may limit what can be known about interventions and how they function.

13. Issues
Complex interventions are generally accepted to be multidimensional and synergistic activities implemented into dynamic and unpredictable contexts - the assumption that the causal pathway is linear and non-recursive may not be appropriate
The operationalised logic model is a very simplified version of the theory of change and represents just one of many potential interpretations of the intervention change process.

14. Issues The effect of confounding is amplified in mediation analyses.
The introduction of a mediator introduces a second consideration – that there is no confounding between the mediator and the outcome.
In a complex intervention it is likely that there will be many sources of confounding between the mediator and outcome that cannot be measured and accommodated in analyses of mediation.
Independence of mediators is unlikely
negates the intention of using mediation analysis to conduct a comprehensive analyses modelling the effect of all hypothesised mediators at once
most researchers conducting mediation analysis proceed with a single mediator model which assumes that there is no relationship among the multiple mediators and analyse pathways separately (Imai and Yamamoto 2013)
Consider the causal pathway between the intervention and patient satisfaction mediated by stocks of mRDTs: Some health centres received supplies of mRDTs from the government throughout the study period. This activity would have had an effect on the mediator by decreasing stock-outs of mRDTs thereby confounding the outcome of improved appropriate treatment of fevers if health workers used the mRDTs to diagnose patients
In the crowded landscape of low resource settings where numerous government, NGO and research initiatives are working to improve health and health services (Whyte et al. 2013), it is inevitable that initiatives will interact with each other in significant and ‘confounding’ ways
and severely limits interpretation of any effects generated as interactions between multiple mediators can introduce unexpected sources of bias and confounding into analyses
As Daniel et al (2015) demonstrate, when applying the potential outcomes framework to the analysis of two causally related mediators, there are 24 possible ways to decompose the total causal effect into direct and indirect effects

15. Recommendations
Researchers should be made aware that the standard RCT is not the ideal study design for assessments of mediation because mediators are not randomized and therefore the mediators themselves may be subject to confounding.
Researchers must decide their primary objective – either analysis of intervention effect or analysis of mediation – and design their study to maximise this objective.
Researchers should consider how their interventions and accompanying theories of change might be conceptualised in such a way as to align with the acyclical processes necessary for statistical mediation analyses
Imai et al (2013) have designed a set of alternative study designs which address this issue by randomising assignment to the intervention as well as the mediator and comparing outcomes with outcomes from a standard RCT. These are new designs and have not been widely applied, although there appears to be a growing interest for alternative types of designs and analyses for the evaluation of complex interventions (Cousens et al. 2011; Lamont et al. 2016).
the randomization of both treatment and mediator in general does not satisfy this sequential ignorability assumption. Therefore, the randomization of both treatment and mediators is insufficient for identifying mediation effects unless an additional assumption (e.g., no interaction effect between treatment and mediator) is imposed
For example, Angeles et al (2013) present an approach to intervention design that is based on identifying variables (independent, dependent, mediating, moderating, and control), postulating how these variables are related, and developing a logic model by linking the variables in a series of if-then logic statements.

16. Recommendations
Instead of attempting to assess the complete causal pathway between intervention, mediator and outcome, researchers should consider the suggestion from within the mediation literature of limiting their analyses to only modelling the intervention effects on mediators, but not extending the analyses to include outcome measures (Keele 2015).
Finally, researchers are urged to engage with and learn from the growing literature on mediation
there is a risk that researchers and funders will continue to naïvely promote the use of mediation analysis despite several significant statistical and conceptual challenges that have yet to be resolved.
This chapter is one of several attempts amongst a group of researchers who have subsequently abandoned their mediation analyses due to conceptual and statistical limitations, and lack of results.
While mediation analysis may appear a promising methodology, researchers must be prepared to first tackle the theoretical underpinnings of the methodology and then contribute empirical examples scrutinising its statistical and conceptual challenges in order to determine the usefulness of applying this methodology to the evaluation of complex health interventions.

17. References
Baron, Reuben M., and David A. Kenny “The Moderator–mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations.” Journal of personality and social psychology 51(6):1173.
Imai, Kosuke, Luke Keele, and Dustin Tingley “A General Approach to Causal Mediation Analysis.” Psychological methods 15(4):309–34.
Imai, Kosuke, Luke Keele, Dustin Tingley, and Teppei Yamamoto “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies.” American Political Science Review 105(04):765–89.
Imai, Kosuke, Luke Keele, and Teppei Yamamoto “Identification, Inference and Sensitivity Analysis for Causal Mediation Effects.” Statistical Science 25(1):51–71.
Imai, Kosuke, Dustin Tingley, and Teppei Yamamoto “Experimental Designs for Identifying Causal Mechanisms.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 176(1):5–51.

18. References
Imai, Kosuke, and Teppei Yamamoto “Causal Inference with Differential Measurement Error: Nonparametric Identification and Sensitivity Analysis.” American Journal of Political Science 54(2):543–60.
Imai, Kosuke, and Teppei Yamamoto “Identification and Sensitivity Analysis for Multiple Causal Mechanisms: Revisiting Evidence from Framing Experiments.” Political Analysis 21(2):141–71.
Keele, Luke “Causal Mediation Analysis: Warning! Assumptions Ahead.” American Journal of Evaluation 1–14.
Moore, Graham et al “Process Evaluation of Complex Interventions: Medical Resource Council Guidance.” Retrieved January 15, 2015 ( phsrn-process-evaluation-guidance-final/).