The mathematics skills of school children: How does England compare to the high performing East Asian jurisdictions?

PISA maths – England versus East Asia By age 15, children in East Asian counties are approximately 1.5 years of schooling ahead of children in England

Huge policy interest in this…. These regions and nations – from Alberta to Singapore, Finland to Hong Kong, Harlem to South Korea - should be our inspiration’ (Secretary of State for Education in England, Michael Gove) ‘we must learn from high-performing nations like Japan’ (Shadow Secretary of State for Education in England, Stephen Twigg) ‘We have to see this as a wake-up call….. We can quibble, or we can face the brutal truth that we’re being out-educated’ (US Secretary of Education, Arne Duncan)

WHY!?....we don’t really know

Possible explanations Biology / genetics Cultural differences - Commitment and highly value education (‘Tiger mums’) ‘Failing’ secondary schools Teaching methods Design of the curriculum Many more!!

Possible explanations Biology / genetics - Can’t do anything about Cultural differences – Hard to change in short-run - Commitment and highly value education (‘Tiger mums’) ‘Failing’ secondary schools Teaching methods Things we can change Design of the curriculum Many more!!

...but we do not know the relative importance of each of these factors Policy – hard to know what we need to do to catch up!

Current research Lots of research looking at the ‘high-performing’ systems (particularly East Asia) E.g. Review of the curriculum tionDetail/Page1/DFE-RR178 E.g. Teaching observations in Shanghi Pretty much all qualitative This paper: Think what quantitative research can add to the debate

But there is a MAJOR problem for both qualitative and quantitative research East Asian countries have a lot of East Asian children in them! ….Big confounding factor

Implications of this confounding Qualitative research – We observe differences in curriculum – We observe differences in teaching methods – BUT we don’t know if this has causal impact – Really the reason why East Asian countries doing better? Quantitative research – Can’t easily ‘control away’ cross-country confounders – Therefore also difficult to establish causal impacts – PISA can tell us how far we are behind…… – …..But it can’t tell us why!

My paper….. What else can be done to inform the debate???? Starting point – How do we judge school performance? – Widely recognised it should not be on % 5 A*- C grades – Should be judged on ‘value-added’ (how much children improve) International application – Should think same way about countries / school systems – Don’t judge UK secondary schools by PISA test scores – Judge them by value-added – How much do UK children improve in secondary school (relative to East Asian children)

We try to start thinking about this….. (1)How do average test scores change between age 10, 14 and 16 in England compared to East Asia? (2)How does the distribution of test scores change between 10, 14 and 16 in England and East Asia? (3)How does the socio-economic gap in children’s test scores change over age in England compared to East Asia?

Data

Ideal data → Longitudinal → Follow same kids over time → Test scores designed to be comparable over time and across countries → E.g. A longitudinal PISA This type of very rich data does not exist Therefore have to think about an alternative

Data we do have…. → TIMSS th grade (age 9/10) → TIMSS th grade (age 13/14) → PISA 2009 (age 15 year 3 months – 16 years 2 months) – Think of this as ‘repeated cross-sectional data’ – Investigate change over time at the population level under certain assumptions: Samples drawn from same population at each time point The same skill is being measured at each time point Argument: Do these assumptions hold for TIMSS / PISA data? Argue they provide an OK approximation….

An aside – how similar are PISA / TIMSS? Often argued that PISA / TIMSS measure ‘different skills’ PISA = ‘Functional ability’ TIMSS = ‘Curriculum’ based skills Qualitative research: Analysis of questions suggests there are differences.. …but unable to say of this has major impact Other statistical differences Item-response model used Target populations etc But really how different are PISA and TIMSS??

PISA 2009 vs TIMSS 2011 (maths) Strong correlation between x-national results (r = 0.88) Actually tell us pretty similar things about performance

PISA 2003 vs Key stage 3 test scores (England) Correlation PISA maths with Key Stage 3 average scores = 0.83 Correlation PISA maths with Key Stage 3 math scores = 0.85

Previous graphs suggest that key assumptions of the method used may approximately hold

International z-scores 20 Nevertheless …….. PISA and TIMSS raw test scores are not directly comparable – based on a different array of countries. Convert into international z-scores. Each country’s mean test score (for each wave of the survey) is adjusted by subtracting the mean score achieved amongst all children in the countries for that particular survey and dividing by the standard deviation Estimates refer to English pupils’ test performance relative to other countries

Countries included in this study 21 Include all countries that took part in TIMSS 4 th grade (2003), TIMSS 8 th grade (2007) and PISA 2009 Leaves 13 countries: -England (Focus) -Asian Tigers (Hong Kong, Japan, Singapore, Taiwan) -Other developed (e.g. Italy, Norway, US)

Measures 22 ‘Average’ = Mean z-score ‘Inequality’ = Standard deviation of test scores ‘Low achievers’ = 10 th percentile of test distribution ‘High achievers’ = 90 th percentile of test distribution ‘Low SES’ = 0 – 25 books in the home ‘High SES’ = More than 200 books in home (i)Measuring construct of interest? (ii)Measurement error?

Methods 23 ‘Average’ = OLS regression (allowing for clustering). Two-sample t-test for statistical significance ‘Low achievers’ = Quantile regression at 10 th percentile. Bootstrapped standard errors to allow for clustering. ‘High achievers’ = Quantile regression at 90 th percentile. Bootstrapped standard errors to allow for clustering. SES differences in achievement = OLS regression. Maths test scores as response. Books in home as key explanatory variable. Controls for gender and immigrant status.

Results 24

…. we are just as far behind at age 9/10 as at 15/16 (on average) Gap between England and East Asia does not widen between 9/10 and 15/16 (in terms of our relative position)

….(some evidence) low achievers catch-up (age 10 – age 16) Low achievers in England may improve relative to low achievers in East Asian countries between primary and secondary school

…. But the high achievers may lose some ground High achievers in England may fall back relative to those in East Asian countries between primary and secondary school

SES gap in math test scores Big gap in England…. …seems to increase between 10 and 16 ….but this holds true in all countries Caution needed Difficulties with measurement of SES (and with books in the home)

Conclusions 29

Suggests (but does not conclusively prove) that.... ‘Poor’ PISA ranking not about failing secondary schools... - Just as far behind East Asian countries before children each secondary school (on average) though still possible some changes are needed - E.g. Curriculum / incentives to stretch the highest achievers more? Focus should perhaps be earlier years??? - Primary school? Early childhood?.... but still can’t rule out ‘culture’ as the dominant force? - Likely to take hold early in life. - Hence consistent with the large cross-national differences at age 10.