1 1 National Conference on Student Assessment How the U.S. and Other Countries Compare: Lessons Learned International Data CCSSO – San Diego June 22, 2015 Andreas Schleicher

Increased likelihood of positive outcomes among adults with higher literacy skills 2 (scoring at Level 4/5 compared with those scoring at Level 1 or below) Odds ratio

3 3 The dilemma for educators The kinds of things that are easy to teach and test are also easy to digitise, automate and outsource

4 The modern world no longer rewards people just for what they know, but for what they can do with what they know Source: Autor, David H. and Brendan M. Price "The Changing Task Composition of the US Labor Market: An Update of Autor, Levy, and Murnane ( 2003)." MIT Mimeograph, June.

5 PISA in brief Over half a million students… – representing 28 million 15-year-olds in 65 countries/economies … took an internationally agreed 2-hour test… – Goes beyond testing whether students can reproduce what they were taught… …to assess students’ capacity to extrapolate from what they know and creatively apply their knowledge in novel situations – Mathematics, reading, science, problem-solving, financial literacy – Total of 390 minutes of assessment material …and responded to questions on… – their personal background, their schools and their engagement with learning and school Parents, principals and system leaders provided data on… – school policies, practices, resources and institutional factors that help explain performance differences.

High mathematics performance Low mathematics performance … Shanghai-China performs above this line (613) Average performance of 15-year-olds in Mathematics (PISA) Fig I.2.13 Below PISA Level 2 Massachusetts 26% of American 15-year-olds do not reach PISA Level 2 (OECD average 23%, Shanghai 4%, Japan 11%, Canada 14% )

Low mathematics performance * Substituted from TIMSS

Regular but moderate physical exercise is good for our health What happens when muscles are exercised? Circle “Yes” or “No” for each statement. Does this happen when muscles are exercised?Yes or No? Muscles get an increased flow of blood.Yes / No Fats are formed in the muscles.Yes / No Answering this question correctly corresponds to a difficulty of 386 score points on the PISA science scale. Across OECD countries, 82% of students answered correctly. This question assesses students’ competency of explaining phenomena scientifically.Answering this question correctly corresponds to a difficulty of 386 score points on the PISA science scale. Across OECD countries, 82% of students answered correctly. This question assesses students’ competency of explaining phenomena scientifically. Answering this question correctly corresponds to a difficulty of 386 score points on the PISA science scale. Across OECD countries, 82% of students answered correctly. This question assesses students’ competency of explaining phenomena scientifically.Answering this question correctly corresponds to a difficulty of 386 score points on the PISA science scale. Across OECD countries, 82% of students answered correctly. This question assesses students’ competency of explaining phenomena scientifically. Answering this question correctly corresponds to a difficulty of 386 score points on the PISA science scale. Across countries, 82% of students answered correctly. This question assesses students’ competency of explaining phenomena scientifically. % students by country who answered correctly Finland93 Hungary91 Russian Federation90 Slovenia89 Latvia88 Czech Republic88 Iceland88 Greece87 Portugal87 Croatia86 Spain86 Italy85 Liechtenstein85 Hong Kong- China85 Australia85 Canada84 Denmark84 Serbia84 New Zealand84 Belgium84 Poland84 Netherlands84 Tunisia83 Slovak Republic83 United Kingdom83 OECD average82 Sweden82 Switzerland82 Chile82 Turkey82 Thailand81 Macao-China81 Bulgaria81 Jordan80 Israel80 Japan80 Luxembourg79 Austria79 France79 Mexico78 Germany77 Estonia77 Chinese Taipei77 Norway76 United States76 Romania76 Montenegro76 Ireland76 Argentina75 Lithuania73 Azerbaijan72 Brazil71 Korea68 Colombia63 Kyrgyzstan57 Indonesia54 Qatar53

Mei-Ling from Singapore was preparing to go to South Africa for 3 months as an exchange student. She needed to change some Singapore dollars (SGD) into South African rand (ZAR). Question: Mei-Ling found out that the exchange rate between Singapore dollars and South African rand was: 1 SGD = 4.2 ZAR Mei-Ling changed 3000 Singapore dollars into South African rand at this exchange rate. How much money in South African rand did Mei-Ling get? Answer: ________________________ % students by country who answered correctly Liechtenstein95 Macao- China93 Finland90 France89 Hong Kong-China89 Sweden89 Austria87 Switzerland87 Belgium87 Czech Republic87 Canada86 Slovak Republic86 Iceland86 Denmark85 Russian Federation85 Luxembourg85 Netherlands85 Hungary84 Ireland83 Germany83 Australia81 Korea81 Latvia80 New Zealand80 OECD average80 Japan79 Spain79 Serbia79 Norway77 Poland77 Portugal74 United Kingdom74 Greece73 Italy71 Uruguay71 Mexico60 Thailand60 Turkey60 Indonesia59 Tunisia55 United States54 Brazil zAR Answering this question correctly corresponds to a difficulty of 406 score points on the PISA mathematics scale. Across countries, 80% of students answered correctly. To answer the question correctly students have to draw on skills from the reproduction competency cluster.

Figure 1 shows changing levels of Lake Chad, in Saharan North Africa. Lake Chad disappeared completely in about 20,000BC, during the last Ice Age. In about 11,000 BC it reappeared. Today, its level is about the same as it was in AD 1000.

Figure 2 aurochs giraffe buffalo Saharan rock art and changing patterns of wildlife rhinoceros hippopotamus elephant ostrich gazelle cattle dog horse camel 8000 BC 7000 BC 6000 BC 5000 BC 4000 BC 3000 BC 2000 BC 1000 BC 0 AD 1000 Question: Figure 2 is based on the assumption that: A. the animals in the rock art were present in the area at the time they were drawn. B. the artists who drew the animals were highly skilled. C. the artists who drew the animals were able to travel widely. D. there was no attempt to domesticate the animals which were depicted in the rock art. Figure 2 shows Saharan rock art (ancient drawings or paintings found on the walls of caves) and changing patterns of wildlife Answering this question correctly corresponds to a difficulty of 397 score points on the PISA reading scale. Across countries, 77% of students answered correctly. To do so, they interpreted the text correctly. % students by country who answered correctly Finland87 Hungary85 Korea85 Netherlands84 Austria83 Sweden82 Spain82 France82 Belgium81 Czech Republic80 Denmark80 Canada80 Germany80 Australia80 Liechtenstein79 Japan79 Italy79 Switzerland78 New Zealand78 Portugal78 OECD average 77 United Kingdom76 Poland73 Ireland72 Luxembourg72 Norway72 United States71 Iceland70 Greece68 Latvia68 Brazil63 Russian Federation59 Mexico49

The economic value of improvement Value of improvement in terms of current GDP over working life of today’s 15-year-olds The increase in GDP among high income countries would still exceed total current spending on schooling

Low mathematics performance * Substituted from TIMSS 3880% GDP 4,526 bn$ 751% GDP 23,841 bn$

High mathematics performance Low mathematics performance 86% GDP 402 bn$ 153% GDP 27,929 bn$

Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance Average performance of 15-year-olds in mathematics Strong socio-economic impact on student performance

2012 Socially equitable distribution of learning opportunities Strong socio-economic impact on student performance Massachusetts High mathematics performance Low mathematics performance

2012 Socially equitable distribution of learning opportunities Strong socio-economic impact on student performance High mathematics performance Low mathematics performance

Socially equitable distribution of learning opportunities Strong socio-economic impact on student performance High mathematics performance Low mathematics performance

Contribution of various factors to upper secondary teacher compensation costs, per student as a percentage of GDP per capita (2004) Percentage points

Mean mathematics performance, by school location, after acc ounting for socio-economic status Fig II Behavioural issues equate to lower job satisfaction, class size doesn’t Teachers' job satisfaction level following the number of students in the classroom in relation to the percentage of students with behavioural problems

High mathematics performance Low mathematics performance Socially equitable distribution of learning opportunities Strong socio-economic impact on student performance

Singapore Shanghai Singapore Germany, Turkey and Mexico improved both their mathematics performance and equity levels Brazil, Italy, Macao-China, Poland, Portugal, Russian Federation, Thailand and Tunisia improved their mathematics performance (no change in equity) Liechtenstein, Norway, the United States and Switzerland improved their equity levels (no change in performance) Chile 2003 Turkey 2003 High mathematics performance Low mathematics performance Socially equitable distribution of learning opportunities Strong socio-economic impact on student performance

23 Don’t close achievement gaps the wrong way Performance differences between top and bottom quarter of socio-economic distribution Quarter of most disadvantaged students PISA performance (mathematics) Quarter of most privileged students Source: PISA 2012

24 Why poverty need not be destiny It’s not just about poor kids in poor neighborhoods but about many kids in many neighborhoods The country where students go to class matters more than what social class students come from

25 PISA mathematics performance by decile of social background Source: PISA 2012

Social background – principal and students Size of bullet represents impact of social background on student performance

Percentage of top performers in mathematics 27 Tab I.2.1a Across OECD, 13% of students are top performers (Level 5 or 6). They can develop and work with models for complex situations, and work strategically with advanced thinking and reasoning skills Massachusetts Connecticut Florida

28 Why care about advanced skills? Evolution of employment in occupational groups defined by PIAAC problem-solving skills Employment of workers with advanced problem-solving skills Employment of workers with poor problem-solving skills Employment of workers with medium-low problem-solving skills (PIAAC) Source:PIAAC 2011

Math teaching ≠ math teaching PISA = reason mathematically and understand, formulate, employ and interpret mathematical concepts, facts and procedures 29

Focus on word problems Fig I.3.1a 30 Formal math situated in a word problem, where it is obvious to students what mathematical knowledge and skills are needed

Focus on conceptual understanding Fig I.3.1b 31

Lessons from high performers Catching up with the top-performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence r A commitment to education and the belief that competencies can be learned and therefore all children can achieve Universal educational standards and personalization as the approach to heterogeneity in the student body… …as opposed to a belief that students have different destinations to be met with different expectations, and selection/stratification as the approach to heterogeneity Clear articulation who is responsible for ensuring student success and to whom

Countries where students have stronger beliefs in their abilities perform better in mathematics 35 Fig III.4.5

Perceived self-responsibility for failure in mathematics Percentage of students who reported "agree" or "strongly agree" with the following statements: Fig III

Greater self-efficacy among girls could shrink the gender gap in mathematics performance, particularly among the highest-performing students 37 Fig III.7.12 Boys do better Girls do better

Percentage of girls and boys who intend to take additional mathematics, rather than language, courses after they leave school

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence r Clear ambitious goals that are shared across the system and aligned with high stakes gateways and instructional systems Well established delivery chain through which curricular goals translate into instructional systems, instructional practices and student learning (intended, implemented and achieved) High level of metacognitive content of instruction …

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence r Capacity at the point of delivery Attracting, developing and retaining high quality teachers and school leaders and a work organisation in which they can use their potential Instructional leadership and human resource management in schools Keeping teaching an attractive profession System-wide career development …

Developing Teaching as a profession Recruit top candidates into the profession Support teachers in continued development of practice Retain and recognise effective teachers – path for growth Improve the societal view of teaching as a profession Mean mathematics performance, by school location, after acc ounting for socio-economic status 41 Capacity at the point of delivery

Mean mathematics performance, by school location, after accounting for socio-economic status Fig II Teachers' perceptions of the value of teaching Percentage of lower secondary teachers who "agree" or "strongly agree" that teaching profession is a valued profession in society

Mean mathematics performance, by school location, after accounting for socio-economic status Fig II Countries where teachers believe their profession is valued show higher levels of student achievement Relationship between lower secondary teachers' views on the value of their profession in society and the country’s share of top mathematics performers in PISA 2012 R 2 = 0.24 r= 0.49

Teacher skills and graduate skills (numeracy) Middle half of the numeracy skill distribution of graduates (16-65 years) PIAAC test scores (numeracy)

Teacher skills and graduate skills (numeracy) Middle half of the numeracy skill distribution of graduates (16-65 years) Numeracy skills of teachers PIAAC test scores (numeracy)

Mean mathematics performance, by school location, after accounting for socio-economic status Fig II Teachers Self-Efficacy and Professional Collaboration

Percentage of lower secondary teachers who report doing the following activities at least once per month Teacher co-operation 47 Exchange and co-ordination

Mean mathematics performance, by school location, after acc ounting for socio-economic status Fig II Teachers' needs for professional development Percentage of lower secondary teachers indicating they have a high level of need for professional development in the following areas

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence r Incentives, accountability, knowledge management Aligned incentive structures For students How gateways affect the strength, direction, clarity and nature of the incentives operating on students at each stage of their education Degree to which students have incentives to take tough courses and study hard Opportunity costs for staying in school and performing well For teachers Make innovations in pedagogy and/or organisation Improve their own performance and the performance of their colleagues Pursue professional development opportunities that lead to stronger pedagogical practices A balance between vertical and lateral accountability Effective instruments to manage and share knowledge and spread innovation – communication within the system and with stakeholders around it A capable centre with authority and legitimacy to act

Lessons from high performers 50 Aligning autonomy with accountability

Lessons from high performers 51 Countries that grant schools autonomy over curricula and assessments tend to perform better in mathematics Source: PISA 2012

Schools with more autonomy perform better than schools with less autonomy in systems with standardised math policies Score points School autonomy for curriculum and assessment x system's extent of implementing a standardised math policy (e.g. curriculum and instructional materials) Fig IV.1.16

Schools with more autonomy perform better than schools with less autonomy in systems with more accountability arrangements Score points School autonomy for curriculum and assessment x system's level of posting achievement data publicly Fig IV.1.16

Schools with more autonomy perform better than schools with less autonomy in systems with more collaboration Score points School autonomy for resource allocation x System's level of teachers participating in school management Across all participating countries and economies Fig IV.1.17

Quality assurance and school improvement Fig IV

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence r Investing resources where they can make most of a difference Alignment of resources with key challenges (e.g. attracting the most talented teachers to the most challenging classrooms) Effective spending choices that prioritise high quality teachers over smaller classes

57 Align the resources with the challenges Greater equity Less equity Adjusted by per capita GDP Countries with better performance in mathematics tend to allocate educational resources more equitably Source: PISA 2012

58 Adequate resources to address disadvantage Disadvantaged schools reported more teacher shortage Advantaged schools reported more teacher shortage A shortage of qualified teachers is more of concern in disadvantaged schools

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence r Coherence of policies and practices Alignment of policies across all aspects of the system Coherence of policies over sustained periods of time Consistency of implementation Fidelity of implementation (without excessive control)

Lessons from high performers Low impact on outcomes High impact on outcomes Low feasibilityHigh feasibility Money pits Must haves Low hanging fruits Quick wins Commitment to universal achievement Gateways, instructional systems Capacity at point of delivery Incentive structures and accountability Resources where they yield most A learning system Coherence

Lessons from high performers Some students learn at high levels All students need to learn at high levels Student inclusion Routine cognitive skills Conceptual understanding, complex ways of thinking, ways of working Curriculum, instruction and assessment Standardisation and compliance High-level professional knowledge workers Teacher quality ‘Tayloristic’, hierarchical Flat, collegial Work organisation Primarily to authorities Primarily to peers and stakeholders Accountability What it all means The old bureaucratic systemThe modern enabling system

Lessons from high performers 62 Thank you Find out more about our work at –All publications –The complete micro-level database Twitter: SchleicherEDU and remember: Without data, you are just another person with an opinion