1. Narrowing the evaluation gap
Session 2.B.
Beyond Randomised Controlled Trials – How non-experimental methods can contribute to educational research
2018 EEF Evaluators’ Conference
Narrowing the evaluation gap
#EEFeval18
@EducEndowFound

2. University of Warwick and Institute for Fiscal Studies
Claire Crawford
University of Warwick and Institute for Fiscal Studies

3. What do we mean by non-experimental methods?
Methods that take advantage of “natural” experiments which produce close to random assignment of treatment (conditional on covariates)
(Or that can account for some types of unobserved heterogeneity)
e.g.
RDD (sharp/fuzzy, i.e. partnered with IV)
DiD
(Panel data methods?)
(OLS/matching not typically included in this group)

4. How might nonexperimental methods be used?
Initial piloting of policies (DiD)
e.g. EMA, free school meals
Geographic rollout of policies (DiD)
e.g. expansion of free nursery places
Administrative rules used for implementation of policies (RDD)
e.g. date of birth cut-offs determining entitlement to start nursery/school
e.g. rules to regulate class sizes in Israel
Key assumption required (vs. RCTs) is that these “natural” experiments mimic random assignment; credibility relies on this assumption

5. University of Westminster
Richard Dorsett
University of Westminster

6. A role for non-experimental methods in EEF research
Where an RCT is impractical/impossible
perhaps logistical challenges
programmes already underway/scaling up
Where some early/quick/rough evidence is needed
How much of a change is this?
Will involve a different approach to impact estimation
May require different data collection
May leave process study unchanged (if prospective, not if retrospective)
not clear what happens to costs

7. RCT vs non-experimental estimators
RCTs
gold standard in the hierarchy of evidence
Face validity – easy to understand/communicate/persuade
Non-experimental estimators
Cover a range of approaches…
… which vary in their underlying assumptions (some credible, others less so).
Often lower face validity (are data rich enough? is instrument valid?)…
but no issue with randomisation bias (possible external validity advantage).
In practice
Practical issues can reduce purity of RCT design (drop-out, attrition, fidelity)
Non-experimental estimators may form part of RCT analysis (e.g. CACE, imputation)
spectrum

8. Going forward - some considerations
Non-experimental estimators need careful consideration of underlying assumptions
How to assess non-experimental estimators’ security alongside RCT security
When is a careful non-experimental estimator better than a flawed RCT?
Take best practice elements from RCTs
Protocol/SAP – need justification for choice of approach
Power calculations – not traditionally a focus of nonexperimental analysis
Blinding – where feasible
Greater role for sensitivity analysis?
Understanding more about non-experimental estimators in this space:
Revisit completed RCTs using nonexperimental estimators
Understanding general suitability of commonly-used data (NPD), e.g.:
Stability of school trends over time
Situations where selection on observed characteristics is plausible

9. Table Discussions
1.- How can we assess the quality and credibility of non-experimental impact estimates?
How credible are non-experimental estimates? Is there are hierarchy of evidence?
How can we assess the validity and credibility of non-experimental estimates?
How can we assess the stability of results in one study? What is the role of using multiple specifications and methods? How can we combine those results?
2.- How can we ensure that non-experimental estimates are comparable – to each other and to experimental estimates?
Is it possible to specify a prospective protocol/SAP for non-experimental methods? To what extent will these overlap or differ by method?
What is the level of detail that should be expected in prospective documents? Should they include robustness checks and falsification tests? Should prospective documents and reporting follow any specific standards? If yes, which ones?
Under what circumstances should analyses that deviate from those set out in prospective documents be considered? Are these circumstances the same as for RCTs or different?