Improving Education: A triumph of hope over experience Robert Coe Inaugural Lecture, Durham University, 18 June 2013

A triumph of hope over experience Have educational standards really risen? School improvement: Isn’t it time there was some? Can we identify effective schools and teachers? Is ‘evidence-based’ practice and policy the answer? Hope So what should we do (that hasn’t failed yet)? www.cem.org/publications

Have educational standards really risen?

Equivalent change in GCSE grades Hanushek E.A. & Woessmann L. (2010) The High Cost of Low Educational Performance: The Long-run Economic Impact of Improving PISA Outcomes. OECD Programme for International Student Assessment. http://www.oecd.org/dataoecd/11/28/44417824.pdf

(Updated from Coe, 2007) Updated from Coe, R. (2007) Changes in standards at GCSE and A-Level: Evidence from ALIS and YELLIS. Report for the Office of National Statistics, April 2007. Curriculum, Evaluation and Management Centre, Durham University. Available at http://www.cemcentre.org/attachments/ONS%20report%20on%20changes%20at%20GCSE%20and%20A-level.pdf (Updated from Coe, 2007)

ICCAMS (Hodgen et al) Hodgen, J., Brown, M., Coe, R., & Küchemann, D. E. (2012). Why are educational standards so resistant to reform? An examination of school mathematics in England. Paper presented at the 2012 Annual Conference of the American Educational Research Association (AERA), Vancouver. Hodgen, J., Küchemann, D., Brown, M., & Coe, R. (2010). Multiplicative reasoning, ratio and decimals: A 30 year comparison of lower secondary students' understandings. In M. F. Pinto & T. F. Kawaski (Eds.), Proceedings of the 34th Conference of the International Group of the Psychology of Mathematics Education (Vol. 3, pp. 89-96). Belo Horizonte, Brazil. Hodgen, J., Kuchemann, D., Brown, M. and Coe, R. (2009) ‘Children’s understandings of algebra 30 years on’. Research in Mathematics Education, 11, 2, 193-194

School improvement: Isn’t it time there was some?

Mistaking School Improvement (1) (Coe, 2009) Wait for a bad year or choose underperforming schools to start with. Most things self-correct or revert to expectations (you can claim the credit for this). Take on any initiative, and ask everyone who put effort into it whether they feel it worked. No-one wants to feel their effort was wasted. Define ‘improvement’ in terms of perceptions and ratings of teachers. DO NOT conduct any proper assessments – they may disappoint. Only study schools or teachers that recognise a problem and are prepared to take on an initiative. They’ll probably improve whatever you do. Coe, R (2009) ‘School Improvement: Reality and Illusion’ British Journal of Educational Studies, 57, 4, 363-379.

Mistaking School Improvement (2) (Coe, 2009) Conduct some kind of evaluation, but don’t let the design be too good – poor quality evaluations are much more likely to show positive results. If any improvement occurs in any aspect of performance, focus attention on that rather than on any areas or schools that have not improved or got worse (don’t mention them!). Put some effort into marketing and presentation of the school. Once you start to recruit better students, things will improve. Coe, R (2009) ‘School Improvement: Reality and Illusion’ British Journal of Educational Studies, 57, 4, 363-379. See also http://nepc.colorado.edu/newsletter/2013/05/tcr-dirty-dozen and http://www.tcrecord.org/Content.asp?ContentID=17104 for a US perspective.

Can we identify effective schools and teachers?

Problems with school effectiveness research ‘Value-added’ is not effectiveness (Gorard, 2010; Dumay, Coe & Anumendem, 2013) Characteristics of ‘effective schools’ ‘strong leadership’, ‘high expectations’, ‘positive climate’ and a ‘focus on teaching and learning’ Too vague ‘Effects’ are tiny anyway (Scheerens, 2000, 2012) Correlations, not causes (Coe & Fitz-Gibbon, 1998) Can ‘effective’ strategies be implemented? If so, do they lead to improvement? Gorard, S. (2010) Serious doubts about school effectiveness, British Educational Research Journal, 36, Dumay, X., Coe, R., and Anumendem, D. (in press, 2013) ‘Stability over time of different methods of estimating school performance’. School Effectiveness and School Improvement, vol , no , pp . http://www.tandfonline.com/doi/full/10.1080/09243453.2012.759599 Scheerens (2000) ‘Improving School Effectiveness’ UNESCO http://unesdoc.unesco.org/images/0012/001224/122424E.pdf Viviane M. J. Robinson, Claire A. Lloyd and Kenneth J. Rowe (2008) The Impact of Leadership on Student Outcomes: An Analysis of the Differential Effects of Leadership Types. Educational Administration Quarterly 2008; 44; 635 Scheerens, J. (ed.) (2012) School leadership effects revisited: review and meta-analysis of empirical studies. Dordrecht: Springer. [average effect size of leadership characteristics on student outcomes is 0.04]

Is ‘evidence-based’ practice and policy the answer?

Toolkit of Strategies to Improve Learning Higgins, S., Katsipataki, M., Kokotsaki, D., Coleman, R., Major, L.E., & Coe, R. (2013). The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit. London: Education Endowment Foundation. [Available at http://www.educationendowmentfoundation.org.uk/toolkit] The Sutton Trust-EEF Teaching and Learning Toolkit http://www.educationendowmentfoundation.org.uk/toolkit/

Impact vs cost Promising May be worth it Not worth it www.educationendowmentfoundation.org.uk/toolkit Impact vs cost Promising 8 May be worth it Feedback Meta-cognitive Peer tutoring Early Years Homework (Secondary) 1-1 tuition Effect Size (months gain) Collaborative Behaviour Small gp tuition Phonics Parental involvement Smaller classes Social ICT Summer schools Individualised learning After school Not worth it Mentoring Homework (Primary) Performance pay Teaching assistants Aspirations Ability grouping £0 £1000 Cost per pupil

Key messages Some things that are popular or widely thought to be effective are probably not worth doing Ability grouping (setting); After-school clubs; Teaching assistants; Smaller classes; Performance pay; Raising aspirations Some things look ‘promising’ Effective feedback; Meta-­cognitive and self regulation strategies; Peer tutoring/peer‐assisted learning strategies; Homework

Clear, simple advice: Choose from the top left Go back to school and do it For every complex problem there is an answer that is clear, simple, and wrong H.L. Mencken

Why not? We have been doing some of these things for a long time, but have generally not seen improvement Research evidence is problematic Sometimes the existing evidence is thin Research studies may not reflect real life Context and ‘support factors’ may matter (Cartwright and Hardie, 2012) Implementation is problematic We may think we are doing it, but are we doing it right? We do not know how to get large groups of teachers and schools to implement these interventions in ways that are faithful, effective and sustainable

So what should we do (that hasn’t failed yet)?

Four steps to improvement Think hard about learning Invest in effective professional development Evaluate teaching quality Evaluate impact of changes

1. Think hard about learning

Impact vs cost Promising May be worth it Not worth it www.educationendowmentfoundation.org.uk/toolkit Impact vs cost Promising 8 May be worth it Feedback Meta-cognitive Peer tutoring Early Years Homework (Secondary) 1-1 tuition Effect Size (months gain) Collaborative Behaviour Small gp tuition Phonics Parental involvement Smaller classes Social ICT Summer schools Individualised learning After school Not worth it Mentoring Homework (Primary) Performance pay Teaching assistants Aspirations Ability grouping £0 £1000 Cost per pupil

Impact vs cost Does your theory of learning explain why … These work? www.educationendowmentfoundation.org.uk/toolkit Impact vs cost Does your theory of learning explain why … 8 Feedback Meta-cognitive These work? Peer tutoring Homework (Secondary) Effect Size (months gain) These don’t? Collaborative Phonics Smaller classes After school Performance pay Teaching assistants Aspirations Ability grouping £0 £1000 Cost per pupil

Poor Proxies for Learning Students are busy: lots of work is done (especially written work) Students are engaged, interested, motivated Students are getting attention: feedback, explanations Classroom is ordered, calm, under control Curriculum has been ‘covered’ (ie presented to students in some form) (At least some) students have supplied correct answers (whether or not they really understood them or could reproduce them independently)

A simple theory of learning Learning happens when people have to think hard

Hard questions about your school How many minutes does an average pupil on an average day spend really thinking hard? Do you really want pupils to be ‘stuck’ in your lessons? If they knew the right answer but didn’t know why, how many pupils would care?

2. Invest in effective CPD

How do we get students to learn hard things? Place value Persuasive writing Music composition Balancing chemical equations Explain what they should do Demonstrate it Get them to do it (with gradually reducing support) Provide feedback Get them to practise until it is secure Assess their skill/ understanding

How do we get teachers to learn hard things? Using formative assessment Assertive discipline How to teach algebra Explain what they should do See www.geoffpetty.com/downloads/WORD/Differentiation /Makingithappen1.doc‎

What (probably) makes CPD effective? Intense: at least 15 hours, preferably 50 Sustained: over at least two terms Content focused: on teachers’ knowledge of subject content & how students learn it Active: opportunities to try it out & discuss Supported: external feedback and networks to improve and sustain Evidence based: promotes strategies supported by robust evaluation evidence Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007–No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. Retrieved from http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/rel_2007033.pdf Wei, R. C., Darling-Hammond, L., Andree, A., Richardson, N., Orphanos, S. (2009). Professional learning in the learning profession: A status report on teacher development in the United States and abroad. Dallas, TX. National Staff Development Council. http://www.becker.k12.mn.us/sites/beckerschools/files/departments/2012/NSDCstudytechnicalreport2009_0.pdf Cordingley, P. & Bell, M. (2012) Understanding What Enables High Quality Professional Learning: A report on the research evidence. Centre for the Use of Research Evidence in Education (CUREE); Pearson School Improvement http://www.pearsonschoolmodel.co.uk/wp-content/uploads/2011/09/CUREE-Report.pdf “The new theory of evolution” http://www.teacherdevelopmenttrust.org/the-new-theory-of-evolution/ Joyce and Showers (2002) ‘Student Achievement through Staff Development’ 3rd ed. ASCD www.ascd.org [excellent summary at www.geoffpetty.com/downloads/WORD/JoyceNshowers.doc‎. ]

3. Evaluate teaching quality

Every teacher needs to improve, not because they are not good enough, but because they can be even better. Dylan Wiliam Dylan Wiliam, in a speech to the SSAT National Conference, 4 Dec, 2012. http://www.ssatuk.co.uk/wp-content/uploads/2012/12/Dylan-Wiliam_keynote.pdf Hattie, J. (2003) Teachers Make a Difference: What is the research evidence? Australian Council for Educational Research, October 2003

Identifying the best teachers Sources of evidence: Colleagues (peers, SMs) observing lessons Trained outsiders observing lessons Pupils’ test score gains Progress in NC levels (from teacher assessment) Pupils’ ratings of teacher/lesson quality Teacher qualifications Tests of teachers’ content knowledge Parents’ ratings Ofsted ratings Colleagues’ (including senior managers) perceptions Teachers’ self-evaluation Observation: Robert C. Pianta and Bridget K. Hamre Conceptualization, Measurement, and Improvement of Classroom Processes: Standardized Observation Can Leverage Capacity Educational Researcher, March 2009; vol. 38, 2: pp. 109-119. Heather C. Hill, Charalambos Y. Charalambous, and Matthew A. Kraft (2012) When Rater Reliability Is Not Enough: Teacher Observation Systems and a Case for the Generalizability Study. Educational Researcher, March 2012; vol. 41, 2: pp. 56-64. Ho, A.D. and Kane, T. J. (2013). ‘The Reliability of Classroom Observations by School Personnel’. Research Paper. MET Project. Bill & Melinda Gates Foundation. Available at http://www.metproject.org/downloads/MET_Reliability%20of%20Classroom%20Observations_Research%20Paper.pdf

Next generation of CEM systems … Assessments that are Comprehensive, across the full range of curriculum areas, levels, ages, topics and educationally relevant abilities Diagnostic, with evidence-based follow-up Interpretable, calibrated against norms and criteria High psychometric quality Feedback that is Bespoke to individual teacher, for their students and classes Multi-component, incorporating learning gains, pupil ratings, peer feedback, self-evaluation, … Constant experimenting

4. Evaluate impact of changes

Bad reasons not to evaluate We are sure this works This is so important we need it to work Everyone is working really hard and fully committed to this Evaluating would be a lot of work We don’t have the data to be able to evaluate We don’t know how to evaluate We can’t do a really good evaluation, so what is the point of doing it badly? We do happy sheets and ask people what they thought of it; isn’t that enough? You can’t do randomised trials in education What works is different in different schools or contexts

Key elements of good evaluation Clear, well defined intervention Good assessment of appropriate outcomes Well-matched comparison group Coe, R., Kime, S., Nevill, C. and Coleman, R. (2013) ‘The DIY Evaluation Guide’. London: Education Endowment Foundation. [Available at http://educationendowmentfoundation.org.uk/library/diy-evaluation-guide]

A triumph of hope over experience So far, we haven’t cracked it: don’t keep doing the same things Hope Think hard about learning Invest in effective professional development Evaluate teaching quality Evaluate impact of changes