1. Improving Education: A triumph of hope over experience
Robert Coe
Inaugural Lecture, Durham University, 18 June 2013

2. A triumph of hope over experience
Have educational standards really risen?
School improvement: Isn’t it time there was some?
Can we identify effective schools and teachers?
Is ‘evidence-based’ practice and policy the answer?
Hope
So what should we do (that hasn’t failed yet)?

3. Have educational standards really risen?

5. Equivalent change in GCSE grades
Hanushek E.A. & Woessmann L. (2010) The High Cost of Low Educational Performance: The Long-run Economic Impact
of Improving PISA Outcomes. OECD Programme for International Student Assessment.

6. (Updated from Coe, 2007) Updated from
Coe, R. (2007) Changes in standards at GCSE and A-Level: Evidence from ALIS and YELLIS. Report for the Office of National Statistics, April Curriculum, Evaluation and Management Centre, Durham University. Available at
(Updated from Coe, 2007)

7. ICCAMS (Hodgen et al)
Hodgen, J., Brown, M., Coe, R., & Küchemann, D. E. (2012). Why are educational standards so resistant to reform? An examination of school mathematics in England. Paper presented at the 2012 Annual Conference of the American Educational Research Association (AERA), Vancouver.
Hodgen, J., Küchemann, D., Brown, M., & Coe, R. (2010). Multiplicative reasoning, ratio and decimals: A 30 year comparison of lower secondary students' understandings. In M. F. Pinto & T. F. Kawaski (Eds.), Proceedings of the 34th Conference of the International Group of the Psychology of Mathematics Education (Vol. 3, pp ). Belo Horizonte, Brazil.
Hodgen, J., Kuchemann, D., Brown, M. and Coe, R. (2009) ‘Children’s understandings of algebra 30 years on’. Research in Mathematics Education, 11, 2,

8. School improvement: Isn’t it time there was some?

9. Mistaking School Improvement (1) (Coe, 2009)
Wait for a bad year or choose underperforming schools to start with. Most things self-correct or revert to expectations (you can claim the credit for this).
Take on any initiative, and ask everyone who put effort into it whether they feel it worked. No-one wants to feel their effort was wasted.
Define ‘improvement’ in terms of perceptions and ratings of teachers. DO NOT conduct any proper assessments – they may disappoint.
Only study schools or teachers that recognise a problem and are prepared to take on an initiative. They’ll probably improve whatever you do.
Coe, R (2009) ‘School Improvement: Reality and Illusion’ British Journal of Educational Studies, 57, 4,

10. Mistaking School Improvement (2) (Coe, 2009)
Conduct some kind of evaluation, but don’t let the design be too good – poor quality evaluations are much more likely to show positive results.
If any improvement occurs in any aspect of performance, focus attention on that rather than on any areas or schools that have not improved or got worse (don’t mention them!).
Put some effort into marketing and presentation of the school. Once you start to recruit better students, things will improve.
Coe, R (2009) ‘School Improvement: Reality and Illusion’ British Journal of Educational Studies, 57, 4,
See also and for a US perspective.

11. Can we identify effective schools and teachers?

12. Problems with school effectiveness research
‘Value-added’ is not effectiveness (Gorard, 2010; Dumay, Coe & Anumendem, 2013)
Characteristics of ‘effective schools’
‘strong leadership’, ‘high expectations’, ‘positive climate’ and a ‘focus on teaching and learning’
Too vague
‘Effects’ are tiny anyway (Scheerens, 2000, 2012)
Correlations, not causes (Coe & Fitz-Gibbon, 1998)
Can ‘effective’ strategies be implemented?
If so, do they lead to improvement?
Gorard, S. (2010) Serious doubts about school effectiveness, British Educational Research Journal, 36,
Dumay, X., Coe, R., and Anumendem, D. (in press, 2013) ‘Stability over time of different methods of estimating school performance’. School Effectiveness and School Improvement, vol , no , pp .
Scheerens (2000) ‘Improving School Effectiveness’ UNESCO
Viviane M. J. Robinson, Claire A. Lloyd and Kenneth J. Rowe (2008) The Impact of Leadership on Student Outcomes: An Analysis of the Differential Effects of Leadership Types. Educational Administration Quarterly 2008; 44; 635
Scheerens, J. (ed.) (2012) School leadership effects revisited: review and meta-analysis of empirical studies. Dordrecht: Springer. [average effect size of leadership characteristics on student outcomes is 0.04]

13. Is ‘evidence-based’ practice and policy the answer?

14. Toolkit of Strategies to Improve Learning
Higgins, S., Katsipataki, M., Kokotsaki, D., Coleman, R., Major, L.E., & Coe, R. (2013). The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit. London: Education Endowment Foundation. [Available at
The Sutton Trust-EEF Teaching and Learning Toolkit

15. Impact vs cost Promising May be worth it Not worth it
Impact vs cost
Promising 8
May be worth it
Feedback
Meta-cognitive
Peer tutoring
Early Years
Homework (Secondary)
1-1 tuition
Effect Size (months gain)
Collaborative
Behaviour
Small gp tuition
Phonics
Parental involvement
Smaller classes
Social
ICT
Summer schools
Individualised learning
After school
Not worth it
Mentoring
Homework (Primary)
Performance pay
Teaching assistants
Aspirations
Ability grouping £0
£1000
Cost per pupil

16. Key messages
Some things that are popular or widely thought to be effective are probably not worth doing
Ability grouping (setting); After-school clubs; Teaching assistants; Smaller classes; Performance pay; Raising aspirations
Some things look ‘promising’
Effective feedback; Meta-­cognitive and self regulation strategies; Peer tutoring/peer‐assisted learning strategies; Homework

17. Clear, simple advice: Choose from the top left
Go back to school and do it
For every complex problem there is an answer that is clear, simple, and wrong
H.L. Mencken

18. Why not?
We have been doing some of these things for a long time, but have generally not seen improvement
Research evidence is problematic
Sometimes the existing evidence is thin
Research studies may not reflect real life
Context and ‘support factors’ may matter (Cartwright and Hardie, 2012)
Implementation is problematic
We may think we are doing it, but are we doing it right?
We do not know how to get large groups of teachers and schools to implement these interventions in ways that are faithful, effective and sustainable

19. So what should we do (that hasn’t failed yet)?

20. Four steps to improvement
Think hard about learning
Invest in effective professional development
Evaluate teaching quality
Evaluate impact of changes

21. 1. Think hard about learning

22. Impact vs cost Promising May be worth it Not worth it
Impact vs cost
Promising 8
May be worth it
Feedback
Meta-cognitive
Peer tutoring
Early Years
Homework (Secondary)
1-1 tuition
Effect Size (months gain)
Collaborative
Behaviour
Small gp tuition
Phonics
Parental involvement
Smaller classes
Social
ICT
Summer schools
Individualised learning
After school
Not worth it
Mentoring
Homework (Primary)
Performance pay
Teaching assistants
Aspirations
Ability grouping £0
£1000
Cost per pupil

23. Impact vs cost Does your theory of learning explain why … These work?
Impact vs cost
Does your theory of learning explain why … 8
Feedback
Meta-cognitive
These work?
Peer tutoring
Homework (Secondary)
Effect Size (months gain)
These don’t?
Collaborative
Phonics
Smaller classes
After school
Performance pay
Teaching assistants
Aspirations
Ability grouping £0
£1000
Cost per pupil

24. Poor Proxies for Learning
Students are busy: lots of work is done (especially written work)
Students are engaged, interested, motivated
Students are getting attention: feedback, explanations
Classroom is ordered, calm, under control
Curriculum has been ‘covered’ (ie presented to students in some form)
(At least some) students have supplied correct answers (whether or not they really understood them or could reproduce them independently)

25. A simple theory of learning
Learning happens when people have to think hard

26. Hard questions about your school
How many minutes does an average pupil on an average day spend really thinking hard?
Do you really want pupils to be ‘stuck’ in your lessons?
If they knew the right answer but didn’t know why, how many pupils would care?

27. 2. Invest in effective CPD

28. How do we get students to learn hard things?
Place value
Persuasive writing
Music composition
Balancing chemical equations
Explain what they should do
Demonstrate it
Get them to do it (with gradually reducing support)
Provide feedback
Get them to practise until it is secure
Assess their skill/ understanding

29. How do we get teachers to learn hard things?
Using formative assessment
Assertive discipline
How to teach algebra
Explain what they should do
See /Makingithappen1.doc‎

30. What (probably) makes CPD effective?
Intense: at least 15 hours, preferably 50
Sustained: over at least two terms
Content focused: on teachers’ knowledge of subject content & how students learn it
Active: opportunities to try it out & discuss
Supported: external feedback and networks to improve and sustain
Evidence based: promotes strategies supported by robust evaluation evidence
Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007–No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. Retrieved from
Wei, R. C., Darling-Hammond, L., Andree, A., Richardson, N., Orphanos, S. (2009). Professional learning in the learning profession: A status report on teacher development in the United States and abroad. Dallas, TX. National Staff Development Council.
Cordingley, P. & Bell, M. (2012) Understanding What Enables High Quality Professional Learning: A report on the research evidence. Centre for the Use of Research Evidence in Education (CUREE); Pearson School Improvement
“The new theory of evolution”
Joyce and Showers (2002) ‘Student Achievement through Staff Development’ 3rd ed. ASCD [excellent summary at ]

31. 3. Evaluate teaching quality

32. Every teacher needs to improve, not because they are not good enough, but because they can be even better.
Dylan Wiliam
Dylan Wiliam, in a speech to the SSAT National Conference, 4 Dec,
Hattie, J. (2003) Teachers Make a Difference: What is the research evidence? Australian Council for Educational Research, October 2003

33. Identifying the best teachers
Sources of evidence:
Colleagues (peers, SMs) observing lessons
Trained outsiders observing lessons
Pupils’ test score gains
Progress in NC levels (from teacher assessment)
Pupils’ ratings of teacher/lesson quality
Teacher qualifications
Tests of teachers’ content knowledge
Parents’ ratings
Ofsted ratings
Colleagues’ (including senior managers) perceptions
Teachers’ self-evaluation
Observation:
Robert C. Pianta and Bridget K. Hamre
Conceptualization, Measurement, and Improvement of Classroom Processes: Standardized Observation Can Leverage Capacity Educational Researcher, March 2009; vol. 38, 2: pp
Heather C. Hill, Charalambos Y. Charalambous, and Matthew A. Kraft (2012) When Rater Reliability Is Not Enough: Teacher Observation Systems and a Case for the Generalizability Study. Educational Researcher, March 2012; vol. 41, 2: pp
Ho, A.D. and Kane, T. J. (2013). ‘The Reliability of Classroom Observations by School Personnel’. Research Paper. MET Project. Bill & Melinda Gates Foundation. Available at

34. Next generation of CEM systems …
Assessments that are
Comprehensive, across the full range of curriculum areas, levels, ages, topics and educationally relevant abilities
Diagnostic, with evidence-based follow-up
Interpretable, calibrated against norms and criteria
High psychometric quality
Feedback that is
Bespoke to individual teacher, for their students and classes
Multi-component, incorporating learning gains, pupil ratings, peer feedback, self-evaluation, …
Constant experimenting

35. 4. Evaluate impact of changes

36. Bad reasons not to evaluate
We are sure this works
This is so important we need it to work
Everyone is working really hard and fully committed to this
Evaluating would be a lot of work
We don’t have the data to be able to evaluate
We don’t know how to evaluate
We can’t do a really good evaluation, so what is the point of doing it badly?
We do happy sheets and ask people what they thought of it; isn’t that enough?
You can’t do randomised trials in education
What works is different in different schools or contexts

37. Key elements of good evaluation
Clear, well defined intervention
Good assessment of appropriate outcomes
Well-matched comparison group
Coe, R., Kime, S., Nevill, C. and Coleman, R. (2013) ‘The DIY Evaluation Guide’. London: Education Endowment Foundation. [Available at

38. A triumph of hope over experience
So far, we haven’t cracked it: don’t keep doing the same things
Hope
Think hard about learning
Invest in effective professional development
Evaluate teaching quality
Evaluate impact of changes