1. 7 th June 2012 Is it better to fail than to succeed? A quantitative analysis of ‘just’ failing an English school inspection Rebecca Allen, Institute of Education, University of London

2. Research questions  What is the best policy for dealing with under-performing schools?  What happens to schools that are judged unsatisfactory by Ofsted (between 2002 and 2009)?  In principle, the effects of failing an Ofsted inspection could go either way:  inducement to focus on academic performance  spiral of decline

3. The policy treatment  We compare those who ‘just’ fail and are given a notice to improve with those judged as satisfactory “the school requires significant improvement because either: it is failing to provide an acceptable standard of education, but is demonstrating the capacity to improve; or it is not failing to provide an acceptable standard of education but is performing significantly less well in all the circumstances reasonably be expected to perform” (Ofsted, 2011a, page 12)  ‘Light-touch’ judgement, although publicly humiliating?  No operating restrictions  Monitoring inspection within the year and full inspection after a year  Opportunity to attend a school improvement seminar  Expected to amend school plans

4. Identification problem  We aim to estimate the causal impact of being judged by Ofsted as unsatisfactory on school performance  Endogeneity of failure: underperforming schools have different levels and trajectories of achievement, regardless of Ofsted inspections  Estimation approach: regression discontinuity design (RDD) in a panel data context, comparing the performance for schools that are designated as just failing with those just passing  Intuition is that schools around the failure threshold are very similar, except for random measurement of quality by inspectors  A running variable based on sub-criteria judgements captures continuous variation between schools, on top of which is the discontinuity of a discrete judgement of ‘fail’ or ‘pass’

5. Ofsted inspections data Year02/0303/0404/0505/0606/0707/0808/09 Number of school visits4765604529261,106971638 Number of sub-criteria1933 55415865 Rating = Excellent181013n/a Outstanding/Very good1179810998165180151 Good202264190358438417283 Satisfactory114130107348415300167 Unsatisfactory184424122887437 Poor5149n/a Very poor200n/a Proportion failing (%)5.310.47.313.28.07.65.8

6. National Pupil Database (02-11)  National Pupil Database from 2002 onwards, aggregated to school- level variables  The achievement of the year 11 cohorts is measured using:  ‘Capped GCSE’ = average score across all pupils in their best 8 subjects at GCSE, standardised across all pupils as a z- score  ‘%5AC GCSE’ = proportion of pupils achieving five or more ‘good’ GCSEs at grades A*-C  average school grades in English and maths measured on a scale of 0 (=U) to 8 (=A*)  Control variables include free school meals, ethnicity, gender, English mother tongue proportions and average deprivation and prior attainment for cohort

7. Selecting ‘just’ passers and ‘just’ fails From multiple sub-criteria to a continuous, uni-dimensional measure of failure Role of rating variable:  Divides schools into those that actually passed and failed reasonably well  Has enough variation to distinguish between bad fails and very bad fails Our rating variable is prediction from the sub-criteria: fail s = β 0 + β 1 *%fail s + β 2 *%satisfactories s + ε s (centred around zero)

8. Rating variable and bandwidth 8

9. Fuzzy regression discontinuity Instrumented using threshold dummy (rating>0) and quadratic of rating variable on each side of threshold Level and change in control variables: -Prior attainment x 3 -FSM and deprivation -EAL, ethnicity, female Change in school GCSE outcomes at t+1, t+2, t+3, t+4 minus t-1 Prior trend in GCSE outcome variable Inspection year dummies

10. Illustration of impact of failing 10

11. Fuzzy RDD results Difference in GCSE:(t+1) – (t-1)(t+2) – (t-1)(t+3) – (t-1)(t+4) – (t-1) All observations 0.057***0.104***0.112***0.123*** (0.012)(0.015)(0.016)(0.022) N 4004396633132359 Broad bandwidth 0.043*0.069**0.092***0.135*** (0.022)(0.027)(0.031)(0.043) N 467466421325 Narrow bandwidth 0.0460.102***0.121**0.140** (0.032)(0.036)(0.044)(0.055) N 315314283232 V narrow bandwidth 0.0220.0210.1350.082 (0.061)(0.067)(0.084)(0.100) N 156 139119

12. How is the change achieved?  Do schools:  simply try to raise teaching effectiveness, or  Game by introducing a lot of GCSE-equivalents?  Do schools focus:  Just on marginal pupils, or  All pupils?

13. Different outcome variables Difference in GCSE: (t+1) – (t- 1) (t+2) – (t- 1) (t+3) – (t- 1) (t+4) – (t- 1) Capped mean GCSE score 0.0460.102***0.121**0.140** (0.032)(0.036)(0.044)(0.055) N 315314283232 Fraction with least 5 A*-C GCSE 0.0240.037*0.050**0.058** (0.019)(0.021)(0.025)(0.029) N 315314283232 Mean English GCSE score 0.139**0.164**0.141*0.094 (0.062)(0.068)(0.079)(0.082) N 315314283232 Mean Maths GCSE score 0.146**0.114*0.1060.074 (0.059)(0.064)(0.074)(0.081) N 315314283232

14. Marginal pupils versus others Difference in GCSE: (t+1) – (t-1)(t+2) – (t-1)(t+3) – (t-1)(t+4) – (t-1) Lower ability students0.0100.075*0.117**0.118* (0.037)(0.045)(0.048)(0.062) Marginal students0.082*0.093*0.113**0.157** (0.044)(0.048)(0.053)(0.064) Higher ability students0.085*0.106**0.095*0.216*** (0.044)(0.050)(0.055)(0.069)

15. Conclusions  Findings  Schools failing their Ofsted inspections improve their subsequent performance relative to the score in the pre-visit year  The magnitudes are quantitatively very significant: around 10% of a (student-level) SD  The main impact arises two years after the visit in this data  Effects are consistent across individual subjects  Why do they improve?  Threat from re-inspection?  Information about relative performance?  Public stigma of failure?  Policy implications  Cost compared to alternatives  What to do with satisfactory schools?

16. Full report available at: http://repec.ioe.ac.uk/repec/pdf/qsswp1202.pdf