1. Evaluation of Switch-on
Learning from an effectiveness trial of the Switch-on intervention
28 June 2016

2. Contents What is Switch-on? Designing an effectiveness trial:
Selecting trial participants
Deciding which pupils to test
Choosing an appropriate outcome measure
Delivering the trial:
Operational success factors

3. Overview 1.

4. Switch-on: the intervention
Intensive, personalised reading intervention
Trained TAs work with individual pupils 1:1
Daily 20 minute sessions over 10 weeks
Each teaching assistant works with 2-4 pupils
Two versions of the intervention
Switch-on Reading
Switch-on Reading and Writing
Both aim to improve reading outcomes

5. Switch-on: the trial Three-arm cluster-randomised effectiveness trial
Train the trainers delivery model
21 trainers in 17 local authorities, training 530 TAs (between 2 and 4 per school)
184 schools, delivering to 770 pupils
Trainers recruit schools in their area
Wait-list (intervention cannot be applied to pupils in the control)
Current stage of trial:
Post-intervention testing complete
Analysis about to begin

6. Design challenges 2.

7. Selecting trial participants
Core issue: is this a real-world or a trial problem?
If real world, reflect in design
If trial, work around it
Real-world: let schools select pupils and TAs
Selection before randomisation in all schools
Findings representative of ‘pupils schools would select’
What does that mean?
Trial: evaluators select pupils at random
Findings representative of population of eligible pupils

8. Deciding who to test Testing only selected participants
Opt-in consent
Validity problems from non-participation, especially with low achievers
Can only examine core trial question
Testing whole class
Opt-out consent
Ethical and cost issue
Can examine spillover effects

9. Choosing the right test
We wanted:
Reading-skill focused outcome test
Paper-based
Non-adaptive
No floor effects with low achievers
No or small ceiling effects for high achievers
Choice was mainly between the New Group Reading Test and the Hodder Group Reading Test

10. Operational challenges3.

11. Minimising attrition
Provisional numbers: we tested 8,285 pupils out of a target of 9,034 over a three week period = 8.3% attrition
Success factors
Wait list approach
Being clear from the start of the trial about the testing requirements
Staying in touch with control schools
Working collaboratively with the developers and trainers

12. Carrying out the testing
A dedicated point of contact
Scheduling calls to agree how testing would take place
Selecting invigilators with the right skills and briefing them carefully
Mop-up testing
Live monitoring of progress

13. Contact Rakhee Patel Research Director T. 020 7549 7002E.
Nico Jabin T E.
Visit us online, natcen.org.uk