Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell (CEM)

Background on evidence- based education What research use experts said: How are you going to measure outcomes? Can you expect to see any difference?

Research dissemination: competing ideas Passive methods are rarely effective Passive methods are cheap i.e. a tiny effect is still cost effective Measurement of attainment is cheap (National Pupil Database) Measurement of research use is complex and expensive Passive methods may not require consent Several interventions trying to achieve the same goal Active methods can be expensive

Trial design Two RCTs focused on improving Key Stage 2 literacy through teachers’ research use: Active trial- 780 primary schools randomised to experience one of varied activities from CPD to online communities (or control) Passive trial- Most other English primary schools randomised to receive one of various research-based communications (or control)

Outcomes ACTIVE TRIALPASSIVE TRIAL Primary outcomeAttainment in literacy at age 11 Attainment in literacy age 11 Secondary outcomesTeachers’ awareness, understanding and action in relation to research use n/a

How many arms in the active trial? Four providers of research evidence, each with an active and passive approach Control group CEM need three arms

Active trial IEE Electronic/ paper materials Evidence fair Campaign for Learning/ Train Visual Website SupportCEM Paper materials Light CPD Advanced CPD ResearchEd/ Nat Cen Conference Online community Control group 780 schools recruited

Passive trial IEE Electronic/ paper materials Campaign for Learning/ Train Visual Website CEM Paper materials ResearchEd/ Nat Cen ConferenceControl group 12,500 schools 2500

Timeline Recruitment and baseline research use measures Autumn 2014 Randomisation December 2014 Research dissemination Spring 2015 New teaching methods and future planning Summer 2015 New teaching methods Autumn spring 2016 Research use outcomes Spring 2016 Attainment outcome Summer 2016 Attainment outcome Summer 2017

CEM intervention LEVEL 3 : Evidence-based materials, two CPD sessions, pupil diagnostic information (InCAS), peer observations LEVEL 2: Evidenced-based materials, one CPD session LEVEL 1: Evidence-based materials (includes advice booklet, ‘tips for teachers’ cards and monthly classroom posters)

Challenge 1-Recruitment Starting point- why do schools want to take part in research?: Having their say Doing something good for education The process being easy and sounding easy Trust in the organisation approaching them Security of data Clear concise communication Offer of feedback

Our recruitment approach Promotion at Research Ed and other provider events Word of mouth between schools and providers, NFER, EEF Website with detailed information on the trial and what is expected of schools and form to express an interest Letters, phoning, s targeted response Helping inform schools about RCTs including our guide for schools

Recruitment (continued)

Challenge 2- Secondary outcome measurement Secondary outcomes are ‘awareness’, ‘understanding’ and ‘action’ Measure using a bespoke survey Schools involved in other research projects using this survey cannot be in this trial

Research use outcomes Baseline survey developed without pilot ‘Awareness’, ‘understanding’ and ‘action’ assessed through a series of questions in a survey of teachers Follow-up survey will be piloted in autumn Questions will be retained if they discriminate well and contribute to reliable measures

Intention-to-treat analysis Attainment (multilevel model, attainment at age 7 as covariate) Research use is also hierarchical but low numbers per cluster (research use at baseline as a covariate)

Challenge 3- Coordination of the delivery organisations With 6 delivery partners carrying out different interventions things to consider: Effective, timely communication (inc one main point of contact at NFER and one at each provider) Funder as mediator for more basic updates Monitoring information planned in advance

Acknowledgements Education Endowment Foundation NFER Education Trials Unit

For more information For Literacy Octopus project information: For our guide for schools: For information on RCTs for educational researchers: For the NFER Education Trials Unit with details of our other trials

NFER provides evidence for excellence through its independence and insights, the breadth of its work, its connections, and a focus on outcomes. National Foundation for Educational Research The Mere, Upton Park Slough, Berks SL1 2DQ T: F: E: