1. Impact of two teacher training programmes on pupils’ development of literacy and numeracy ability: a randomised trial Jack Worth National Foundation for Educational Research 9 th September 2015 RCTs in the Social Sciences, York

2. Overview Introduction Methods Results Discussion

3. Introduction Randomised trial commissioned by the Education Endowment Foundation Motivations: General identify interventions that raise the attainment of pupils, particularly in literacy and numeracy, and for disadvantaged pupils Specific evaluate the impact of University of Oxford’s Improving Numeracy and Literacy programme on numeracy and literacy

4. Two interventions Mathematics and Reasoning develop children’s understanding of the logical principles underlying mathematics Literacy and Morphemes improve spelling and reading comprehension by teaching children about sentence structure and morphemes Aimed at teachers of Year 2 pupils (age 7) One day of teacher training and a researcher visit Lesson materials for 10 whole-class lessons Accompanying computer games for in-class/home use by pupils Previous researcher-delivered interventions had effect sizes 0.3 - 1.2

5. Trial design School-randomised controlled trial Three arms to evaluate two interventions Eligibility State-funded mainstream infant and primary schools (Teachers of) Year 2 pupils eligible to participate Opt-out consent offered to parents

6. Outcomes Primary outcome Raw score in Progress in Maths / Progress in English tests Test administered by NFER test administrator, blind to group allocation and instructed not to discuss group allocation or the intervention with teachers Secondary outcomes Free school meal sub-group, and interaction analysis Prior attainment interaction analysis English as an additional language interaction analysis Sub-domain scores in relevant areas Key Stage 1 teacher assessment levels (maths and reading/writing) Transfer effects (numeracy intervention on literacy, and v.v.)

7. Sample size Design Aimed to recruit 60 schools, 20 in each group Average 45 pupils per school Intra-cluster correlation = 0.15 Correlation between pre- and post-test = 0.8 Two-tailed significance = 5%, power = 80% Minimum detectable effect size = 0.22

8. Sample size ▼ Number of schools = 55 (60) ▼ Pupils per school = 35 (45) ▼ Intra-cluster correlation = 0.09 (0.15) ▲ Correlation between pre- and post-test = 0.81 (0.80) ▼ Actual MDES = 0.18

9. Randomisation Simple randomisation by NFER First block of 51 schools randomised (17 each) Block of three late-recruited schools randomised Block of two late-recruited schools randomised to two randomly selected groups School told of group allocation after its pre-test researcher visit One randomised school dropped out before testing, but randomisation outcomes retained

10. Participants S = 55 1719 Allocation 1719 Follow-up 1719 Analysis 972652593 890594537 517 (87%)577 (88%)850 (87%) P = 2217 ControlLiteracyNumeracy

11. Participants School-level averagesControl group Literacy group Numeracy group Pre-test literacy score23.123.823.6 (8.9) (9.2) Pre-test numeracy score20.221.1 (4.7)(4.8) School-level averagesControl group Literacy group Numeracy group Female (%)47.949.746.3 Free school meals (%)21.014.010.1 Special educational needs (%)16.415.612.2 English as an additional language (%)16.221.014.1

12. Participants Multilevel model of pre-test imbalanceEffect size95% CI Pre-test literacy score0.08-0.12 – 0.27 Pre-test numeracy score0.20-0.03 – 0.43 School-level averagesControl group Literacy group Numeracy group Pre-test literacy score23.123.823.6 (8.9) (9.2) Pre-test numeracy score20.221.1 (4.7)(4.8)

13. Analysis Two-level multilevel model (school, pupil) Post-test raw score as dependent variable Pre-test raw score as covariate Literacy/ Numeracy group indicators (forced into final models) Additional covariates (backwards selection) Age in months at post-test Gender Randomisation block Hedges’ g effect size calculated by: group indicator coefficient pupil-level standard deviation from MLM

14. Results Effect size95% confidence interval Numeracy intervention0.200.02 – 0.37 Literacy intervention-0.05-0.18 – 0.08 Numeracy intervention (FSM)0.14-0.08 – 0.37 Literacy intervention (FSM)0.10-0.13 – 0.34 No significant impact on Key Stage 1 teacher assessments No significant transfer effects No significant differential impact by prior ability, or for EAL pupils No significant differential impact on sub-domain scores

15. Further results FSM interaction analysis Effect size95% confidence interval Literacy intervention (FSM sub-group)0.10-0.13 – 0.34 Literacy intervention (FSM interaction)0.260.08 – 0.43

16. Further results Effect size95% confidence interval Total number of computer games played (numeracy) 0.19 per 20 games 0.10 – 0.27 Total number of computer games played (literacy) 0.05 per 15 games -0.01 – 0.11 Note: 15 and 20 games played were the interquartile ranges for literacy and numeracy respectively. The effect size shows the effect of moving from the 25th percentile to the 75th percentile in terms of the number of games played. Computer games “on-treatment” analysis

17. Discussion Numeracy intervention has shown promise at raising attainment. Literacy intervention has not. Both interventions are very low cost £10 per pupil per year over 3 years Plus a day of teacher training Will it work at a larger scale? This was an efficacy trial to demonstrate impact in ideal conditions Now an effectiveness trial with a scalable delivery model

18. NFER provides evidence for excellence through its independence and insights, the breadth of its work, its connections, and a focus on outcomes. National Foundation for Educational Research The Mere, Upton Park Slough, Berks SL1 2DQ T: 01753 574123 F: 01753 691632 E: enquiries@nfer.ac.uk www.nfer.ac.uk Worth, J., Sizmur, J., Ager, R. and Styles, B. (2015). Improving Numeracy and Literacy: Evaluation Report and Executive Summary. London: Education Endowment Foundation https://educationendowmentfoundation.org.uk/projects/improving-numeracy-and-literacy/ Any questions?