ARROW Trial Design Professor Greg Brooks, Sheffield University, Ed Studies Dr Jeremy Miles York University, Trials Unit Carole Torgerson, York University, Ed Studies. Professor David Torgerson York University, Trials Unit

Background Information and communication technology (ICT) is widely used in schools. There is little evidence that it is beneficial. A quasi experiment in Israel showed no change in Hebrew literacy and a decline in maths after implementation of ICT.

Background To look at the effectiveness of ICT on literacy a series of systematic reviews have been undertaken with Dept of Ed Studies, York. These have found no evidence of a benefit of ICT on literacy. A systematic review of trials looking at ICT and spelling showed a modest, non- significant benefit on the learning of spelling. Torgerson & Elbourne. JRR 2002;25:125.

Spelling Meta-analysis Torgerson & Elbourne. JRR 2002;25:125.

Summary of evidence Existing evidence of ICT and spelling relies on 7 non-UK trials the largest of which had only 99 pupils. Evidence for benefit or harm is weak. NEED to do more trials especially in the UK.

ARROW trial The ARROW study will evaluate a computer based literacy package. A computer will read text to pupils who simultaneously read it on screen. The pupil reads the text recording it on the computer and finally they write the text (either on the computer or on paper).

ARROW The ARROW method has been used for many years and pupils appear to make progress with the method. HOWEVER, existing data are only before and after and its use is mainly confined to children with low test scores – thus inviting regression to the mean effects.

Extending Use of ARROW In the academic year 2004 it was decided to offer ARROW to all Year 7 pupils in a large comprehensive school. However, the school head and teachers and developer of ARROW were persuaded to undertake an RCT.

Method All Year 7 pupils will be offered ARROW either at the beginning of the first term or at the end. Pupils will be given a pre-test to assess literacy levels and will be given appropriate level of ARROW instruction.

Sample size The total sample size is dictated by the size of the year group. However, few social science interventions improve outcomes by more than 0.5 standard deviations. Therefore we need at least 128 in our sample to detect this difference. Our sample size will be 157.

Randomisation To avoid subversion of randomisation the children’s names have been sorted in alphabetical order. A computer generated randomisation list is then applied to these names. Two copies of the allocation list are held – an independent researcher visits the school to check the children’s allocation corresponds to the list of random numbers (eg. The 10 th child in the alphabet should correspond to random number 10 on list).

Pre-test assessment Pre intervention literacy levels are assessed by independent researchers BEFORE random allocation. This avoids baseline testing being compromised by children’s knowledge of their group assignment.

Post test assessment 1 day after the intervention group have received ARROW both groups will receive post-tests. 12 weeks after ARROW all groups receive second post-test. Those giving and marking the post-tests will be ‘blind’ to the group allocation of the children.

Avoiding contamination To avoid the control children obtaining ARROW in the first term the programme is delivered on laptop computers that are removed after lessons.

Analysis Main analysis is analysis of covariance, comparing the two groups in terms of their mean test scores at follow up. Statistician will be blind to group allocation (simply A or B).

Weaknesses of trial A single school evaluation – the results may not travel. Results only applicable to the ARROW software. Torgerson & Elbourne meta-analysis found an effect size of 0.35 to have good power, but to demonstrate this would require > 200 participants, so trial is a little smaller than is ideal. Relatively short-term follow-up.

Trial Strengths Trial is pragmatic – evaluating ‘real’ school practice rather than taking place in a psychology lab. Children’s usual teachers giving ARROW. Trial is largest ever done in the field of ICT and spelling, is UK based, and using modern software and hardware.

Conclusion ICT in schools has NEVER been subjected to large robust RCTs. ARROW is the first UK trial to evaluate ICT and spelling. Will inform school policy as to whether to recommend continue using ARROW or not.