From Evidence to Great Teaching Robert Coe, Durham University ASCL Annual Conference, 20 March 2015 What can research tell us about the kinds of classroom practices that are likely to create the most learning for students, and the best bets for introducing interventions with greatest cost-effectiveness? Most important – and hardest – of all, how can we best support teachers in improving their classroom practices and realising the full benefits that research seems to promise? In this session Prof Rob Coe will draw on a range of research, including two reviews of which he is an author: the EEF Toolkit and the Sutton Trust’s ‘What makes great teaching?’ report.

What makes great teaching improving education eef toolkit What makes great teaching Coe, R., Aloisi, C., Higgins, S. and Elliot Major, L. (2014) ‘What makes great teaching? Review of the underpinning research’. Sutton Trust, October 2014 http://www.suttontrust.com/researcharchive/great-teaching/ Coe, R. (2013) Improving Education: A triumph of hope over experience. Inaugural Lecture of Professor Robert Coe, Durham University, 18 June 2013. Essay version available at http://www.cem.org/attachments/publications/ImprovingEducation2013.pdf Video at https://vimeo.com/70471076 Higgins, S., Katsipataki, M., Kokotsaki, D., Coleman, R., Major, L.E., & Coe, R. (2013). The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit. London: Education Endowment Foundation. [Available at http://www.educationendowmentfoundation.org.uk/toolkit]

The argument Successful implementation of evidence-based strategies requires deep understanding of the evidence For most people this requires learning Even if you do ‘what works’ it may not work: always evaluate

Evidence Can’t we just bolt it on?

True or false? Reducing class size is one of the most effective ways to increase learning [evidence] Differentiation and ‘personalised learning’ resources maximise learning [evidence] Generous praise encourages learners and helps them persist with hard tasks [evidence] Technology supports learning by engaging and motivating learners [evidence] The best way to raise attainment is to enhance motivation and interest [evidence] Rob Coe EMC blog on class size: http://educationmediacentre.org/researchnews/whats-the-evidence-on-class-size/ John Hattie on individualisation (eg https://cdn.auckland.ac.nz/assets/education/about/research/documents/influences-on-student-learning.pdf). See also Harry Webb’s blog: ‘The Evidence on Differentiation’ http://websofsubstance.wordpress.com/2014/07/ Deborah Stipek on praise: How Do Teachers' Expectations Affect Student Learning http://www.education.com/reference/article/teachers-expectations-affect-learning/ Steve Higgins et al’s summary for EEF: The Impact of Digital Technology on Learning http://educationendowmentfoundation.org.uk/uploads/pdf/The_Impact_of_Digital_Technologies_on_Learning_FULL_REPORT_(2012).pdf Stephen Gorard et al for JRF on aspirations: The impact of attitudes and aspirations on educational attainment and participation http://www.jrf.org.uk/sites/files/jrf/education-young-people-parents-full.pdf

Most promising for raising attainment Small effects / high cost www.educationendowmentfoundation.org.uk/toolkit Impact vs cost Most promising for raising attainment 8 May be worth it Feedback Meta-cognitive Peer tutoring Early Years Homework (Secondary) 1-1 tuition Effect Size (months gain) Collaborative Behaviour Small gp tuition Phonics Parental involvement Smaller classes Social ICT Summer schools Individualised learning After school Small effects / high cost Mentoring Teaching assistants Homework (Primary) Performance pay Aspirations Ability grouping £0 £1000 Cost per pupil

Poor Proxies for Learning Students are busy: lots of work is done (especially written work) Students are engaged, interested, motivated Students are getting attention: feedback, explanations Classroom is ordered, calm, under control Curriculum has been ‘covered’ (ie presented to students in some form) (At least some) students have supplied correct answers, even if they Have not really understood them Could not reproduce them independently Will have forgotten it by next week (tomorrow?) Already knew how to do this anyway

A better proxy for learning? Learning happens when people have to think hard

What makes great teaching? (According to the evidence)

Dimensions of great teaching (Pedagogical) content knowledge (PCK) Quality of instruction Classroom management / behaviour / control Classroom climate / relationships / expectations Beliefs (theory) about subject, learning & teaching Wider professional elements: collegiality, PD, stakeholder relationships Coe, R., Aloisi, C., Higgins, S. and Elliot Major, L. (2014) ‘What makes great teaching? Review of the underpinning research’. Sutton Trust, October 2014 http://www.suttontrust.com/researcharchive/great-teaching/

1. We do that already (don’t we?) Reviewing previous learning Setting high expectations Using higher-order questions Giving feedback to learners Having deep subject knowledge Understanding student misconceptions Managing time and resources Building relationships of trust and challenge Dealing with disruption

2. Do we always do that? Challenging students to identify the reason why an activity is taking place in the lesson Asking a large number of questions and checking the responses of all students Raising different types of questions (i.e., process and product) at appropriate difficulty level Giving time for students to respond to questions Spacing-out study or practice on a given topic, with gaps in between for forgetting Making students take tests or generate answers, even before they have been taught the material Engaging students in weekly and monthly review

For more on what kinds of praise can be harmful and why, see: Richard Bailey @DrDickB 1Nov2014 ‘The Problem With Praise’ https://www.psychologytoday.com/blog/smart-moves/201411/the-problem-praise Nick Rose @Turnfordblog 1Jun2014 Growth mindset: It’s not magic https://evidenceintopractice.wordpress.com/2014/06/01/growth-mindset-its-not-magic/

3. We don’t do that (hopefully) Use praise lavishly Allow learners to discover key ideas for themselves Group learners by ability Encourage re-reading and highlighting to memorise key ideas Address issues of confidence and low aspirations before you try to teach content Present information to learners in their preferred learning style Ensure learners are always active, rather than listening passively, if you want them to remember

If we know what it looks like, can we just do it?

Improving Teaching Teacher quality is what matters We need to focus on teacher learning Teachers learn just like other people Be clear what you want them to learn Get good information about where they are at Give good feedback

Just a check-list of techniques? No! Great teaching involves selecting, integrating, orchestrating, adapting, monitoring, responding, etc, and depends on context, history, personalities, relationships, etc, But without the skills, a teacher’s choices are more limited Developing these skills & techniques takes dedicated, extended practice, with feedback

What CPD benefits students? Promotes ‘great teaching’ PCK, assessment, learning, high expectations, collective responsibility Focuses on student outcomes Supported by External input: challenge and expertise Peer networks: communities of practice School leaders must actively lead Builds teacher understanding and skills Challenges and engages teachers Integrates theory and active skills practice Enough learning time (monthly for min 6 months: 30hrs+) Timperley, H., Wilson, A., Barrar, H. & Fung, I. (2007) Teacher professional learning and development: Best evidence synthesis iteration. Wellington, New Zealand: Ministry of Education. http://www.educationcounts.govt.nz/publications/series/2515/15341 Timperley et al 2007

When ‘what works’ doesn’t work The case for evaluation

Why monitor teaching quality? Good evidence of (potential) benefit from Performance feedback (Coe, 2002) Target setting (Locke & Latham, 2006) Accountability (Coe & Sahlgren, 2014) Individual teachers matter most Teachers typically stop improving after 3-5 years Everyone can improve Assessment is an essential part of learning (including teacher learning) Coe, R. (2002) ‘Evidence on the Role and Impact of Performance Feedback in Schools’ in A.J.Visscher and R. Coe (eds.) School Improvement Through Performance Feedback. Rotterdam: Swets and Zeitlinger Locke, E. A., & Latham, G. P. (2006). New directions in goal-setting theory. Current directions in psychological science, 15(5), 265-268. Lee J. (2008), ‘Is Test-Driven External Accountability Effective? Synthesizing the Evidence From Cross-State Causal-Comparative and Correlational Studies’, Review of Educational Research 78(3):608–644 Coe, R. and Sahgren G.H, (2014) “Incentives and ignorance in qualifications, assessment, and accountability”. In G.H. Sahlgren (ed.) Tests worth teaching to: incentivising quality in qualifications and accountability. Centre for Market Reform of Education http://www.cmre.org.uk/sites/default/files/Tests%20worth%20teaching%20to_web%20text.pdf WILIAM, D. (2010) 'Standardized Testing and School Accountability', Educational Psychologist, 45: 2, 107 — 122 Rockoff, J. E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. The American Economic Review, 94(2), 247-252. Kukla-Acevedo, S. (2009). Do teacher characteristics matter? New results on the effects of teacher preparation on student achievement. Economics of Education Review, 28(1), 49-57. Murphy, R. (2013) ‘Testing Teachers: What works best for teacher evaluation and appraisal’. Sutton Trust. http://www.suttontrust.com/news/publications/testing-teachers-what-works-best-for-teacher-evaluation/

Methods of identifying effectiveness classroom observations by peers, principals or external evaluators ‘value-added’ models (assessing gains in student achievement) student ratings principal (or headteacher) judgement teacher self-reports analysis of classroom artefacts and teacher portfolios

Do we know a good lesson when we see one? http://www.cem.org/blog/414/

Lesson Observation Two teachers observe the same lesson, one rates it ‘Inadequate’. What is the probability the other will agree? a) 10% b) 40% c) 60% d) 80% An observer judges a lesson ‘Outstanding’. What is the probability that pupils are really making sustained, outstanding progress? a) 5% b) 30% c) 50% d) 70% Answers: 1: probably a; best case for a research-validated, quality-assured, rigorous process is b 2: probably a; best case for a research-validated, quality-assured, rigorous process is b See “Classroom observation: it’s harder than you think” http://www.cem.org/blog/414/ and my presentation to Teach First / Teacher Development Trust Event on Lesson Observation, 13 Jan 2014, London Powerpoint (http://www.cem.org/attachments/Lesson%20Observation%2013Jan14.pptx) and video (http://www.l4l.co.uk/?p=13050) www.cem.org/blog

Beware these traps Overconfidence about knowledge of what is effective Focus on teaching rather than learning Thinking that we are doing it already Overconfidence in assessments (even if formative) of teaching quality Thinking that if we assess teaching we must attach consequences to that (cf ‘assessment for learning’)

Problems with assessment criteria If you know what it means, you know what it means (eg from KS1 Performance Descriptors) capital letters for some names of people, places and days of the week (below) capital letters for some proper nouns and for the personal pronoun ‘I’ (towards) capital letters for almost all proper nouns (at) correctly punctuated (mastery) Teaching by numbers (from KS2 2014 guidance) a range of openings, e.g. adverbials (some time later, as we ran, once we had arrived...), subject reference (they, the boys, our gang...), speech. Some variety in subordinating connectives, e.g. because, if, which use of modals to express prediction, possibility, permission, e.g. should, might, could. 2014 Key stage 2 writing – moderation. Exemplification materials for teacher assessment (Level 4 - Annotated) www.education.gov.uk/ks2

Bias in Teacher Assessment (vs standardised tests) Systematic bias against Pupils with SEN, EAL & FSM Pupils with challenging behaviour Reinforcing stereotypes Eg boys perceived to be better at maths ethnic minority / subject combinations Pupil/teacher interaction Bias against pupils whose personality is different from the teacher’s Harlen W (2004) A systematic review of the evidence of reliability and validity of assessment by teachers used for summative purposes. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. [http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=116&language=en-US] Bennett et al., 1993 Peter Tymms: ‘Teachers show bias to pupils who share their personality’. The Conversation 25 Feb 2015 https://theconversation.com/teachers-show-bias-to-pupils-who-share-their-personality-38018 Burgess, S. and Greaves, E (2009) Test Scores, Subjective Assessment and Stereotyping. Centre for Market and Public Organisation, Bristol University. Working Paper No. 09/221. http://www.bristol.ac.uk/media-library/sites/cmpo/migrated/documents/wp221.pdf of Ethnic Minorities

‘Improvement’ often isn’t School/college would have improved anyway Volunteers/enthusiasts improve: misattributed to intervention Chance variation (esp. if start low) Poor outcome measures Perceptions of those who worked hard at it No robust assessment of pupil learning Poor evaluation designs Weak evaluations more likely to show positive results Improved intake mistaken for impact of intervention Selective reporting Dredging for anything positive (within a study) Only success is publicised Coe, R (2009) ‘School Improvement: Reality and Illusion’ British Journal of Educational Studies, 57, 4, 363-379. Coe, R. (2013) Improving Education: A triumph of hope over experience. Inaugural Lecture of Professor Robert Coe, Durham University, 18 June 2013. Essay version available at http://www.cem.org/attachments/publications/ImprovingEducation2013.pdf Video at https://vimeo.com/70471076 See also http://nepc.colorado.edu/newsletter/2013/05/tcr-dirty-dozen and http://www.tcrecord.org/Content.asp?ContentID=17104 for a US perspective. (Coe, 2009, 2013)

Key elements of good evaluation EEF DIY Evaluation Guide Clear, well defined, replicable intervention Good assessment of appropriate outcomes Well-matched comparison group Coe, R., Kime, S., Nevill, C. and Coleman, R. (2013) ‘The DIY Evaluation Guide’. London: Education Endowment Foundation. [Available at http://educationendowmentfoundation.org.uk/library/diy-evaluation-guide] http://educationendowmentfoundation.org.uk/uploads/pdf/EEF_DIY_Evaluation_Guide_(2013).pdf What could you evaluate?

Simple, huh?

“After 30 years of doing such work, I have concluded that classroom teaching…is perhaps the most complex, most challenging, and most demanding, subtle, nuanced, and frightening activity that our species has ever invented…The only time a physician could possibly encounter a situation of comparable complexity would be in the emergency room of a hospital during or after a natural disaster.” Lee Shulman, The Wisdom of Practice