1. Statistical Analysis Plans EEF Evaluators’ Conference 2016 Dr Ben Styles Head of NFER’s Education Trials Unit

2. The replicability crisis Many research findings are false or exaggerated (Ioannidis, 2014) Suggestions to make more research findings true: replication culture, reproducibility practices, better statistical methods and standardisation of definitions and analyses Ioannidis JPA (2014) How to Make More Published Research True. PLoS Med 11(10): e1001747.doi:10.1371/journal.pmed.1001747 e1001747.doi:10.1371/journal.pmed.1001747

3. Education trials Potentially even more vulnerable: median effect size of 0.1 (Sanders and Chonaire, 2015); very little replication; diversity of approaches to design and analysis Distribution of effect sizes in education research – Sanders and Chonaire (2015) http://www.bris.ac.uk/media-library/sites/cmpo/documents/WP15337_Web_Version.pdf http://www.bris.ac.uk/media-library/sites/cmpo/documents/WP15337_Web_Version.pdf

4. EEF approach Strategy to maximise replicability EEF approach Replication cultureIndependent evaluation…wider availability of trial data and analysis code…occasional re-trial Better statistical methodsAnalysis guidance Standardisation of analysesR-package and forthcoming peer review of SAPs

5. What happens without a SAP? All levels of randomisation Which outcome variable? Which covariates? How much missing data prompts MI/sensitivity analyses? What type of MI/sensitivity analysis? Comparisons when >2 arms? Interim analyses >1 follow-up?

6. What happens without a SAP? Randomised block/multi- site Cluster randomised Fixed or random effect for school? School by treatment interaction? School means/MLM/GEE/Huber- White standard errors

7. What happens without a SAP? = possible cherry picking!

8. Why SAPs To limit ‘cherry picking’ To distinguish between planned and exploratory analyses To provide the opportunity for developer/external challenge before the analysis is carried out Analysis represents a small percentage of the overall trial budget yet the decisions a statistician makes are critical Clinical trials have SAPs

9. Opposition ConcernRepost SAPs kill innovationSAPs do not preclude exploratory analyses Unnecessary bureaucratic hurdleAnalysis has to be written up anyway; EEF template Damaging inflexibility e.g. what if a less biased/more efficient analysis method is proposed in the interim Both analyses are performed and it is explained why one method is superior Green DP (2015) Pre-Analysis Plans: Pros and Cons of Limiting Discretion in the Analysis of RCTs. RCTs in the Social Sciences Conference 2015: Keynote Address

10. Suggested headings for EEF SAP template Introduction Trial design Randomisation Sample size calculations Extent of follow-up Primary outcome Secondary outcomes Protocol changes Primary outcome analysis (ITT) Effect size calculation Missing data analysis strategy On-treatment/per-protocol analysis Secondary outcome analyses Subgroup analyses Additional analyses Interim analyses Analysis of multiple follow-up Sample representation School/student characteristics Test reliability MDES calculation on the basis of parameters seen Baseline effect size for analysed groups (re attrition) Student characteristics of analysed groups (re attrition) List of tables and figures Format of tables and figures

11. SAP sign-off EEF (through peer review) Developer

12. Next steps Write your SAP at least three months before analysis EEF will publish as an addendum to the protocol Volunteer with EEF to review SAPs (small payment, Durham workshop in autumn) In the future: analysis syntax pre-written

13. Workshop on analysis http://www.york.ac.uk/healthsciences/research/trials/rct2016/

14. NFER provides evidence for excellence through its independence and insights, the breadth of its work, its connections, and a focus on outcomes. National Foundation for Educational Research The Mere, Upton Park Slough, Berks SL1 2DQ T: 01753 574123 F: 01753 691632 E: enquiries@nfer.ac.uk www.nfer.ac.uk