Presentation on theme: "Improving Education: A triumph of hope over experience"— Presentation transcript:
1 Improving Education: A triumph of hope over experience Robert CoeInaugural Lecture, Durham University, 18 June 2013
2 A triumph of hope over experience Have educational standards really risen?School improvement: Isn’t it time there was some?Can we identify effective schools and teachers?Is ‘evidence-based’ practice and policy the answer?HopeSo what should we do (that hasn’t failed yet)?
5 Equivalent change in GCSE grades Hanushek E.A. & Woessmann L. (2010) The High Cost of Low Educational Performance: The Long-run Economic Impactof Improving PISA Outcomes. OECD Programme for International Student Assessment.
6 (Updated from Coe, 2007) Updated from Coe, R. (2007) Changes in standards at GCSE and A-Level: Evidence from ALIS and YELLIS. Report for the Office of National Statistics, April Curriculum, Evaluation and Management Centre, Durham University. Available at(Updated from Coe, 2007)
7 ICCAMS (Hodgen et al)Hodgen, J., Brown, M., Coe, R., & Küchemann, D. E. (2012). Why are educational standards so resistant to reform? An examination of school mathematics in England. Paper presented at the 2012 Annual Conference of the American Educational Research Association (AERA), Vancouver.Hodgen, J., Küchemann, D., Brown, M., & Coe, R. (2010). Multiplicative reasoning, ratio and decimals: A 30 year comparison of lower secondary students' understandings. In M. F. Pinto & T. F. Kawaski (Eds.), Proceedings of the 34th Conference of the International Group of the Psychology of Mathematics Education (Vol. 3, pp ). Belo Horizonte, Brazil.Hodgen, J., Kuchemann, D., Brown, M. and Coe, R. (2009) ‘Children’s understandings of algebra 30 years on’. Research in Mathematics Education, 11, 2,
8 School improvement: Isn’t it time there was some?
9 Mistaking School Improvement (1) (Coe, 2009) Wait for a bad year or choose underperforming schools to start with. Most things self-correct or revert to expectations (you can claim the credit for this).Take on any initiative, and ask everyone who put effort into it whether they feel it worked. No-one wants to feel their effort was wasted.Define ‘improvement’ in terms of perceptions and ratings of teachers. DO NOT conduct any proper assessments – they may disappoint.Only study schools or teachers that recognise a problem and are prepared to take on an initiative. They’ll probably improve whatever you do.Coe, R (2009) ‘School Improvement: Reality and Illusion’ British Journal of Educational Studies, 57, 4,
10 Mistaking School Improvement (2) (Coe, 2009) Conduct some kind of evaluation, but don’t let the design be too good – poor quality evaluations are much more likely to show positive results.If any improvement occurs in any aspect of performance, focus attention on that rather than on any areas or schools that have not improved or got worse (don’t mention them!).Put some effort into marketing and presentation of the school. Once you start to recruit better students, things will improve.Coe, R (2009) ‘School Improvement: Reality and Illusion’ British Journal of Educational Studies, 57, 4,See also and for a US perspective.
11 Can we identify effective schools and teachers?
12 Problems with school effectiveness research ‘Value-added’ is not effectiveness (Gorard, 2010; Dumay, Coe & Anumendem, 2013)Characteristics of ‘effective schools’‘strong leadership’, ‘high expectations’, ‘positive climate’ and a ‘focus on teaching and learning’Too vague‘Effects’ are tiny anyway (Scheerens, 2000, 2012)Correlations, not causes (Coe & Fitz-Gibbon, 1998)Can ‘effective’ strategies be implemented?If so, do they lead to improvement?Gorard, S. (2010) Serious doubts about school effectiveness, British Educational Research Journal, 36,Dumay, X., Coe, R., and Anumendem, D. (in press, 2013) ‘Stability over time of different methods of estimating school performance’. School Effectiveness and School Improvement, vol , no , pp .Scheerens (2000) ‘Improving School Effectiveness’ UNESCOViviane M. J. Robinson, Claire A. Lloyd and Kenneth J. Rowe (2008) The Impact of Leadership on Student Outcomes: An Analysis of the Differential Effects of Leadership Types. Educational Administration Quarterly 2008; 44; 635Scheerens, J. (ed.) (2012) School leadership effects revisited: review and meta-analysis of empirical studies. Dordrecht: Springer. [average effect size of leadership characteristics on student outcomes is 0.04]
13 Is ‘evidence-based’ practice and policy the answer?
14 Toolkit of Strategies to Improve Learning Higgins, S., Katsipataki, M., Kokotsaki, D., Coleman, R., Major, L.E., & Coe, R. (2013). The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit. London: Education Endowment Foundation. [Available atThe Sutton Trust-EEF Teaching and Learning Toolkit
15 Impact vs cost Promising May be worth it Not worth it Impact vs costPromising8May be worth itFeedbackMeta-cognitivePeer tutoringEarly YearsHomework (Secondary)1-1 tuitionEffect Size (months gain)CollaborativeBehaviourSmall gp tuitionPhonicsParental involvementSmaller classesSocialICTSummer schoolsIndividualised learningAfter schoolNot worth itMentoringHomework (Primary)Performance payTeaching assistantsAspirationsAbility grouping£0£1000Cost per pupil
16 Key messagesSome things that are popular or widely thought to be effective are probably not worth doingAbility grouping (setting); After-school clubs; Teaching assistants; Smaller classes; Performance pay; Raising aspirationsSome things look ‘promising’Effective feedback; Meta-cognitive and self regulation strategies; Peer tutoring/peer‐assisted learning strategies; Homework
17 Clear, simple advice: Choose from the top left Go back to school and do itFor every complex problem there is an answer that is clear, simple, and wrongH.L. Mencken
18 Why not?We have been doing some of these things for a long time, but have generally not seen improvementResearch evidence is problematicSometimes the existing evidence is thinResearch studies may not reflect real lifeContext and ‘support factors’ may matter (Cartwright and Hardie, 2012)Implementation is problematicWe may think we are doing it, but are we doing it right?We do not know how to get large groups of teachers and schools to implement these interventions in ways that are faithful, effective and sustainable
22 Impact vs cost Promising May be worth it Not worth it Impact vs costPromising8May be worth itFeedbackMeta-cognitivePeer tutoringEarly YearsHomework (Secondary)1-1 tuitionEffect Size (months gain)CollaborativeBehaviourSmall gp tuitionPhonicsParental involvementSmaller classesSocialICTSummer schoolsIndividualised learningAfter schoolNot worth itMentoringHomework (Primary)Performance payTeaching assistantsAspirationsAbility grouping£0£1000Cost per pupil
23 Impact vs cost Does your theory of learning explain why … These work? Impact vs costDoes your theory of learning explain why …8FeedbackMeta-cognitiveThese work?Peer tutoringHomework (Secondary)Effect Size (months gain)These don’t?CollaborativePhonicsSmaller classesAfter schoolPerformance payTeaching assistantsAspirationsAbility grouping£0£1000Cost per pupil
24 Poor Proxies for Learning Students are busy: lots of work is done (especially written work)Students are engaged, interested, motivatedStudents are getting attention: feedback, explanationsClassroom is ordered, calm, under controlCurriculum has been ‘covered’ (ie presented to students in some form)(At least some) students have supplied correct answers (whether or not they really understood them or could reproduce them independently)
25 A simple theory of learning Learning happens when people have to think hard
26 Hard questions about your school How many minutes does an average pupil on an average day spend really thinking hard?Do you really want pupils to be ‘stuck’ in your lessons?If they knew the right answer but didn’t know why, how many pupils would care?
28 How do we get students to learn hard things? Place valuePersuasive writingMusic compositionBalancing chemical equationsExplain what they should doDemonstrate itGet them to do it (with gradually reducing support)Provide feedbackGet them to practise until it is secureAssess their skill/ understanding
29 How do we get teachers to learn hard things? Using formative assessmentAssertive disciplineHow to teach algebraExplain what they should doSee /Makingithappen1.doc
30 What (probably) makes CPD effective? Intense: at least 15 hours, preferably 50Sustained: over at least two termsContent focused: on teachers’ knowledge of subject content & how students learn itActive: opportunities to try it out & discussSupported: external feedback and networks to improve and sustainEvidence based: promotes strategies supported by robust evaluation evidenceYoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007–No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. Retrieved fromWei, R. C., Darling-Hammond, L., Andree, A., Richardson, N., Orphanos, S. (2009). Professional learning in the learning profession: A status report on teacher development in the United States and abroad. Dallas, TX. National Staff Development Council.Cordingley, P. & Bell, M. (2012) Understanding What Enables High Quality Professional Learning: A report on the research evidence. Centre for the Use of Research Evidence in Education (CUREE); Pearson School Improvement“The new theory of evolution”Joyce and Showers (2002) ‘Student Achievement through Staff Development’ 3rd ed. ASCD [excellent summary at]
32 Every teacher needs to improve, not because they are not good enough, but because they can be even better.Dylan WiliamDylan Wiliam, in a speech to the SSAT National Conference, 4 Dec,Hattie, J. (2003) Teachers Make a Difference: What is the research evidence? Australian Council for Educational Research, October 2003
33 Identifying the best teachers Sources of evidence:Colleagues (peers, SMs) observing lessonsTrained outsiders observing lessonsPupils’ test score gainsProgress in NC levels (from teacher assessment)Pupils’ ratings of teacher/lesson qualityTeacher qualificationsTests of teachers’ content knowledgeParents’ ratingsOfsted ratingsColleagues’ (including senior managers) perceptionsTeachers’ self-evaluationObservation:Robert C. Pianta and Bridget K. HamreConceptualization, Measurement, and Improvement of Classroom Processes: Standardized Observation Can Leverage Capacity Educational Researcher, March 2009; vol. 38, 2: ppHeather C. Hill, Charalambos Y. Charalambous, and Matthew A. Kraft (2012) When Rater Reliability Is Not Enough: Teacher Observation Systems and a Case for the Generalizability Study. Educational Researcher, March 2012; vol. 41, 2: ppHo, A.D. and Kane, T. J. (2013). ‘The Reliability of Classroom Observations by School Personnel’. Research Paper. MET Project. Bill & Melinda Gates Foundation. Available at
34 Next generation of CEM systems … Assessments that areComprehensive, across the full range of curriculum areas, levels, ages, topics and educationally relevant abilitiesDiagnostic, with evidence-based follow-upInterpretable, calibrated against norms and criteriaHigh psychometric qualityFeedback that isBespoke to individual teacher, for their students and classesMulti-component, incorporating learning gains, pupil ratings, peer feedback, self-evaluation, …Constant experimenting
36 Bad reasons not to evaluate We are sure this worksThis is so important we need it to workEveryone is working really hard and fully committed to thisEvaluating would be a lot of workWe don’t have the data to be able to evaluateWe don’t know how to evaluateWe can’t do a really good evaluation, so what is the point of doing it badly?We do happy sheets and ask people what they thought of it; isn’t that enough?You can’t do randomised trials in educationWhat works is different in different schools or contexts
37 Key elements of good evaluation Clear, well defined interventionGood assessment of appropriate outcomesWell-matched comparison groupCoe, R., Kime, S., Nevill, C. and Coleman, R. (2013) ‘The DIY Evaluation Guide’. London: Education Endowment Foundation. [Available at
38 A triumph of hope over experience So far, we haven’t cracked it: don’t keep doing the same thingsHopeThink hard about learningInvest in effective professional developmentEvaluate teaching qualityEvaluate impact of changes