1. Building Evidence in Education: Conference for EEF Evaluators 11 th July: Theory 12 th July: Practice www.educationendowmentfoundation.org.uk

2. Panel session 4: Analyse this

3. Testing younger pupils Amy Skipp Amy.Skipp@natcen.ac.uk

4. Pupil testing in the Children and Young people team ARK Maths Mastery (40 schools of year 2 pupils) - IoE evaluating - ‘Singapore method’ of maths tuition - Pre and post test of maths, with waiting control Creative Futures (19 schools of 900 year 2 pupils) - NatCen evaluating -3 arm RCT within classes of Sing, Play, Act - Pre and post testing of maths and literacy

5. Challenges of testing Getting good data out of 6 year olds Measuring the correct outcomes for the intervention Need to minimise burden on schools – time, resource and ‘extras’ SEN recording

6. Choosing a test Age and ability appropriate – suitable entry level but capturing top end Simple and quick to administer (no specialist knowledge) Group administration Standardised outcomes Paper based or PC/online Creative Futures = PIPS ARK Maths = Number Knowledge Test

7. Lessons learnt

8. Children vs testing Difficult to get a quiet private space in primary schools Need to factor in toilet breaks Children like to copy and see how you’re scoring them Little experiencing of ‘being tested’ at this age Time taken to get to test

9. Helping pupils give their best Use of appropriate words Group by matched ability Language support workers

10. Issues with schools Block bookings Changes of staff / school location / IT Contacting correct staff / getting past the receptionist - Web pages - Over recruitment - Value of intervention and EEF

11. Testing children – our specialism Group of experienced ‘testers’ All full DBS clearance Many former teachers Familiarity with tests Enthusiastic about new interventions

12. Panel session 4: Analyse this Trials and tribulations – evaluation of intervention programmes Beng Huat See, Stephen Gorard and Nadia Siddiqui Durham University

13. Randomising within clusters Implications for analysis July 2013

15. Some definitions Types of randomisation (see CONSORT): Simple Restricted Stratified Blocked Paired

16. Restricted randomisation in EEF transitions trials All pupil randomised: Chatterbooks (block=school) Rhythm for Reading (block=school; not pair!) Speaking and Listening (block=timetable group) Vocabulary Enrichment (block=timetable group or national curriculum level)

17. Are we justified in restricting? To improve balance of important covariates For practical reasons No – can adjust for covariates in analysis Yes – teachers need to know numbers for planning purposes

18. Fig 1 Correlation in mean survival time between treatment groups under simple and stratified randomisation (simulated data). Kahan B C, and Morris T P BMJ 2012;345:bmj.e5840 ©2012 by British Medical Journal Publishing Group

19. What’s the problem? Introduces correlation between treatment groups Violates statistical assumption of independence P-values too large and confidence intervals too wide More likely to miss a genuine effect

20. How big is the problem? Only 26% of a recent sample of medical trials made adequate adjustments (Kahan and Morris, 2012) Of course it is difficult to work out the extent of the type I error since making and not making adequate adjustments was NOT RANDOMISED

21. How do we solve it? We need to include the stratification variable as a covariate in the analysis ANCOVA with dummy variables to identify school Multi-level model And this is why pairing before randomisation is a BAD IDEA

22. One last thing Is adjustment necessary for straightforward blocking during rolling randomisation?

23. To remember! If you have restricted your randomisation using a factor that is associated with the outcome (e.g. school) THEN INCLUDE THE FACTOR AS A COVARIATE IN YOUR ANALYSIS