Building Evidence in Education: Conference for EEF Evaluators 11 th July: Theory 12 th July: Practice

Panel session 4: Analyse this

Testing younger pupils Amy Skipp

Pupil testing in the Children and Young people team ARK Maths Mastery (40 schools of year 2 pupils) - IoE evaluating - ‘Singapore method’ of maths tuition - Pre and post test of maths, with waiting control Creative Futures (19 schools of 900 year 2 pupils) - NatCen evaluating -3 arm RCT within classes of Sing, Play, Act - Pre and post testing of maths and literacy

Challenges of testing Getting good data out of 6 year olds Measuring the correct outcomes for the intervention Need to minimise burden on schools – time, resource and ‘extras’ SEN recording

Choosing a test Age and ability appropriate – suitable entry level but capturing top end Simple and quick to administer (no specialist knowledge) Group administration Standardised outcomes Paper based or PC/online Creative Futures = PIPS ARK Maths = Number Knowledge Test

Lessons learnt

Children vs testing Difficult to get a quiet private space in primary schools Need to factor in toilet breaks Children like to copy and see how you’re scoring them Little experiencing of ‘being tested’ at this age Time taken to get to test

Helping pupils give their best Use of appropriate words Group by matched ability Language support workers

Issues with schools Block bookings Changes of staff / school location / IT Contacting correct staff / getting past the receptionist - Web pages - Over recruitment - Value of intervention and EEF

Testing children – our specialism Group of experienced ‘testers’ All full DBS clearance Many former teachers Familiarity with tests Enthusiastic about new interventions

Panel session 4: Analyse this Trials and tribulations – evaluation of intervention programmes Beng Huat See, Stephen Gorard and Nadia Siddiqui Durham University

Randomising within clusters Implications for analysis July 2013

Some definitions Types of randomisation (see CONSORT): Simple Restricted Stratified Blocked Paired

Restricted randomisation in EEF transitions trials All pupil randomised: Chatterbooks (block=school) Rhythm for Reading (block=school; not pair!) Speaking and Listening (block=timetable group) Vocabulary Enrichment (block=timetable group or national curriculum level)

Are we justified in restricting? To improve balance of important covariates For practical reasons No – can adjust for covariates in analysis Yes – teachers need to know numbers for planning purposes

Fig 1 Correlation in mean survival time between treatment groups under simple and stratified randomisation (simulated data). Kahan B C, and Morris T P BMJ 2012;345:bmj.e5840 ©2012 by British Medical Journal Publishing Group

What’s the problem? Introduces correlation between treatment groups Violates statistical assumption of independence P-values too large and confidence intervals too wide More likely to miss a genuine effect

How big is the problem? Only 26% of a recent sample of medical trials made adequate adjustments (Kahan and Morris, 2012) Of course it is difficult to work out the extent of the type I error since making and not making adequate adjustments was NOT RANDOMISED

How do we solve it? We need to include the stratification variable as a covariate in the analysis ANCOVA with dummy variables to identify school Multi-level model And this is why pairing before randomisation is a BAD IDEA

One last thing Is adjustment necessary for straightforward blocking during rolling randomisation?

To remember! If you have restricted your randomisation using a factor that is associated with the outcome (e.g. school) THEN INCLUDE THE FACTOR AS A COVARIATE IN YOUR ANALYSIS