1. ∂ What works…and who listens? Encouraging the experimental evidence base in education and the social sciences RCTs in the Social Sciences 9 th Annual Conference Professor Carole Torgerson School of Education Durham University carole.torgerson@durham.ac.uk September 11th 2014

2. ∂

3. ∂

4. ∂ First known RCT [Walters, 1931] “Five seniors, each of whom had a good scholarship record, pleasing personality, excellent health and fine social environment, were chosen to act as personnel counselors for the members of the freshman class, who at the end of the first eight weeks of school happened to be delinquent in scholarship in the School of Mechanical Engineering in I929-30. The 220 delinquent freshmen were divided into two groups by random sampling.” [italics added]

5. ∂ US legislation: Institute of Education Sciences “Scientifically valid educational evaluation employs experimental designs using random assignment, when feasible, and other research methodologies that allow for the strongest possible causal inferences when random assignment is not feasible.” [page 5] [italics added]

6. ∂ Education Endowment Foundation (EEF) “...all EEF-funded projects are independently and rigorously evaluated...The impact of projects on attainment will be evaluated where possible, using randomised controlled trials.” [italics added]

7. ∂ Every Child Counts The independent evaluation [Torgerson et al, 2013]

8. ∂ Distribution of schools

9. ∂ Early reading policy

10. ∂ Two alternative designs Systematic review of different methods to teach reading [Torgerson, Brooks and Hall, 2006] Expert review of different methods to teach reading [Rose, 2006] –Used systematic review design and methods –Used expert review design and methods –Included randomised trials undertaken anywhere –Included examples of best practice, two UK-based trials and expert opinion –Examined the quality of the included trials –Did not assess the quality of the included studies

11. ∂ House of Commons Education and Skills Committee, 18 th July 2005 “… in conducting his review, Jim Rose will have the opportunity to draw on the findings of an independent systematic literature review of phonics use in the teaching and application of reading and spelling which we have commissioned from Professor Greg Brooks and Carole Torgerson. This delivers on the public commitment we made…in 2003 to publish an analysis of existing research on phonics teaching methodologies. The aim … is to identify what is known from existing literature about how effective different approaches to phonics teaching are in comparison with each other, including the specific area of analytic versus synthetic phonics.”

12. ∂ Meta-analysis: Forest plot

13. ∂ Publication bias: Funnel plot Effect Size

14. ∂ Publication bias: Funnel plot Effect Size

15. ∂ Publication bias: Funnel plot Effect Size Missing trials?

16. ∂ Synthetic versus analytic phonics

17. ∂ Policy decision Ruth Kelly, Secretary of State for Education and Skills Times Mar. 21 st 2006 Sir Jim Rose Times Mar. 21 st 2006 “The case for synthetic phonics is overwhelming.” “I am clear that synthetic phonics should be the first strategy in teaching all children to read.”

18. ∂ “Synthetic phonics does look promising,” says Carole Torgerson…one of the report's authors. “We found it had a moderate effect compared with whole-language approaches, but the evidence base for this conclusion was 12 relatively small trials, only one of which was UK-based. This would be an ideal time to do a national evaluation by implementing systematic synthetic phonics in some schools and not in others and then comparing the two.” Economist Mar. 26 th 2006

19. ∂ “When the UK government recently introduced the ‘synthetic phonics’ method of teaching young children to read, they were told by Carole Torgerson…that they could easily bolster the slim evidence base by randomising which schools joined the programme first.” Financial Times Mar.18 th 2010 “In 2007 the Government introduced a new reading strategy for primary schools based on synthetic phonics, which matches sounds to groups of letters. Professor Torgerson urged ministers to start a randomised trial: the introduction of phonics would have been staggered, with schools chosen at random to start it one year or the next. Every child would have received the intervention, but it would have been possible to compare outcomes and establish whether phonics really works.” Times Sept. 24 th 2011

20. ∂ Evidence of impact? ‘KS1 results cast doubt on phonics’ http://www.tes.co.uk/article.aspx?storycode=6021536 Percentage of pupils reaching level 2 or above Subject2006200720082009201020112012 Reading84 85 87 Writing8180 81 83 Speaking and listening 87 88

21. ∂ Study characteristics of educational trials 1990-2002 Torgerson et al, British Educational Research Journal 2006, 31:6.

22. ∂ Annual conference: Randomised controlled trials in the social sciences

23. ∂ Thank you for listening! Rose, J. (2006) Independent review of the teaching of early reading, London: Department for Education and Skills Torgerson, C.J., Wiggins, A., Torgerson, D.J., Ainsworth, H., Hewitt, C. (2013) Every Child Counts: Testing policy effectiveness using a randomized controlled trial, designed, conducted and reported to CONSORT standards, Journal of Research in Mathematics Education: 15(2) Torgerson DJ, Torgerson CJ The design of randomised trials in health, education and the social sciences. Palgrave Macmillan, Basingstoke, 2008 Torgerson, C.J., Brooks, G. & Hall, J. (2006) Literature review of phonics use. London: Department for Education and Skills Torgerson, C. Systematic Reviews, Continuum, London and New York, 2003 Walters, J.E. (1931) The Journal of Higher Education: 2(8), 1931