1. Accountability Measures and School League Tables Robert Coe Capita workshop, 15th July 2014

2. ∂ Outline  Evidence on impact of accountability  Typology of accountability systems  Moral leadership  What should we do? 2

3. ∂ Who wants accountability?  Direct incentives drive people’s behaviour –Policymakers –Economists –Parents  Negative side-effects outweigh benefits –Teachers –Education researchers –Parents 3

4. Evidence on impact of accountability Robert Coe 4

5. ∂ Research evidence  Meta-analysis of US studies by Lee (2008) –Small positive effects on attainment (ES=0.08)  Impact of publishing league tables (England vs Wales) (Burgess et al 2013) –Overall small positive effect (ES=0.09) –Reduces rich/poor gap –No impact on school segregation  Other reviews: mostly agree, but mixed findings  Lack of evidence about long-term, important outcomes 5

6. ∂ Evidence from PISA  DfE Accountability response: ‘OECD evidence shows that a robust accountability framework is essential to improving pupils’ achievement’ (DfE, 2013)  What the report actually said: ‘there is no measurable relationship between…various uses of assessment data for accountability purposes and the performance of school systems’ (OECD, 2010, p46) 6

7. ∂ Dysfunctional side effects  Extrinsic replaces intrinsic motivation  Narrowing focus on measures  Gaming (playing silly games)  Cheating (actual cheating)  Helplessness: giving up  Risk avoidance: playing it safe  Pressure: stress undermines performance  Competition: sub-optimal for system 7

8. ∂ Accountability cultures Trust Autonomous Confidence Challenge Supportive Improvement-focus Problem-solving Long-term Genuine quality Evaluation Distrust Controlled Fear Threat Competitive Target-focus Image presentation Quick fix Tick-list quality Sanctions

9. ∂ Accountability and improvement Official Accountability Systems Professional Monitoring Systems If you find a problem with your performance, what do you do? Cover it upExpose it to view. (Tymms, 1999)

10. ∂ Overall evidence-based conclusions  Easy to cherry-pick ‘[E]ducational policy makers and practitioners should be cautioned against relying exclusively on research that is consistent with their ideological positions to support or criticize the current high-stakes testing policy movement’ (Lee, 2008, p. 639)  Direct incentives do drive people’s behaviour; current evidence suggests accountability has small positive effects on attainment  Accountability systems always seem to have some undesirable side-effects  Balance of positive & negative effects likely to depend on a range of factors; current knowledge does not allow us to predict confidently 10

11. Moral leadership 11

12. ∂ 12

13. ∂ Hard questions 1.Imagine there was no accountability. What would you do differently? 2.Would students be better off as a result? a)No – I wouldn’t do anything at all differently b)Not significantly – minor presentational changes only c)Yes – students would be better off without accountability 3.What actually stops you doing this? 13

14. Ways forward 14

15. ∂ Making Accountability Work 1.Reclaim professionalism 2.Experiment to optimise 3.Improve the measures 4.Make teacher assessment robust 5.Uncertainty and unpredictability 6.No substitute for judgement 15 (Coe & Sahlgren, 2014)

16. ∂ 1. Reclaim professionalism  Take the pledge: “We do what’s right for children and young people, not just what Ofsted might want”  Commit to supporting other schools/teachers who suffer as a result –Need evidence of great teaching, from robust evaluation and monitoring: can’t just support any school/teacher judged inadequate –Important that it is not just the ‘failed’ school/teacher that complains –Social media campaigns can be very effective: @OldAndrewUK vs Ofsted 16

17. ∂ 2. Experiment to optimise  Should accountability have –Explicit (eg PRP, schools ‘academised’) or implicit (challenge, compare) incentives? –Performance published or confidential? –Interpreted judgements or objective data? –Improvement through consequences or feedback? –Focus on information for consumers (eg parents) or professionals?  We don’t know, so need to experiment 17

18. ∂ 3. Improve the measures  Choose measures that are genuinely aligned with what is valued (& hard to distort)  Ensure assessments/qualifications are predictive of later success  Measure a wide range of outcomes  Look at distributions, not just thresholds  Use delayed outcomes: eg for 11-16 –% NEET @ 18 –% entering elite university courses  Build in loophole-closing mechanisms (eg re difficulty/value of ‘equivalents’) 18

19. ∂ 4. Make teacher assessment robust  Training in assessment and moderation  Link teacher assessed mark distribution to within-centre exam mark distribution  Spot checks (risk targeted): can students reproduce it?  Support whistle-blowing  Signed declarations from teachers, headteachers and students  Questionnaire audit of practices: ‘too good to be true’ triggers spot check 19

20. ∂ 5. Uncertainty and unpredictability  State general aims, but be vague/flexible about specific targets/measures  Change the targets and monitor who chases  Make assessments less predictable (more capricious?) 20

21. ∂  Combine statistical measures with face-to-face observation & judgement  Require inspectors to demonstrate their ability to make sound judgements about complex data, from observation, etc  Actively look for (and publicise) gaming and unintended consequences; encourage whistle- blowing on counter-productive gaming 6. No substitute for judgement

22. Summary … 1.Evidence on accountability is not great, but suggests small positive impacts 2.Dysfunctional side-effects are also real 3.We need experiments to learn how to optimise 4.Moral leadership is required 1.Evidence on accountability is not great, but suggests small positive impacts 2.Dysfunctional side-effects are also real 3.We need experiments to learn how to optimise 4.Moral leadership is required Robert.Coe@cem.dur.ac.uk www.cem.org @ProfCoe