Local, national and international benchmarking for improvement

Local, national and international benchmarking for improvement
Programme for International Student Assessment Local, national and international benchmarking for improvement First results from the OECD test for schools based on PISA School Board Meeting Fairfax, VA 4 April 2013 Alejandro Gomez Palma Policy Analyst / PISA Instrument Development Directorate for Education and Skills

In the dark all schools and school systems look the same…
But with a little light….

In the dark all schools and school sytems look the same…
But with a little light…. …important differences become apparent….

A world of change in the global talent pool Approximated by percentage of persons with high school or equivalent qualfications in the age groups 55-64, 45-55, und years % 1 13 1 27 1. Excluding ISCED 3C short programmes 2. Year of reference 2004 3. Including some ISCED 3C short programmes 3. Year of reference 2003.

How the demand for skills has changed Economy-wide measures of routine and non-routine task input (US) Mean task input as percentiles of the 1960 task distribution It is important that we look at skill utilisation in a dynamic framework. The kind of skills that are needed for success are rapidly evolving. As an example, this chart shows how the composition of the US work force has changed between 1970 and Work involving routine manual input, the jobs of the typical factory worker, was down significantly, that is the result of automation and outsourcing. Non-routine manual work, things we do with our hands, but in ways that are not so easily put into formal algorithms, was down too, albeit with much less change over recent years – and that is easy to understand because you cannot easily computerise the bus driver or outsource your hairdresser. All that is not surprising, but here is where the interesting story begins: Among the skill categories that you see here, routine cognitive input, that is cognitive work that you can easily put into the form of a script saw the sharpest decline in demand over the last couple of decades. So schools are now challenged on where they have traditionally put much of their focus, and what we tend value in multiple choice tests. The point is, that the skills that are easiest to teach and test are also the skills that are easiest to digitise, automatise and offshore. Where are the winners in this process? These are those who engage in expert thinking, up 8% - and complex communication, up almost 14%. This chart is just translating into numbers what I have said before. The dilemma for education and training: The skills that are easiest to teach and test are also the ones that are easiest to digitise, automate and outsource (Levy and Murnane)

PISA countries in 2009 1998 2006 2003 2001 2000 Coverage of world economy 87% 86% 77% 81% 83% 85% We started to develop PISA in 1998 with 28 OECD countries, but since then country participation has grown and our latest PISA assessment covers 74 education systems that make up 86% of the world economy. Coverage in China and India is still patchy though, in China we have now covered 12 provinces and in India we are working in two states only. One aspect that makes PISA stand apart from traditional school tests is that PISA puts less emphasis on whether students can reproduce what they were taught, but focuses on their capacity to extrapolate from what they know and creatively apply what they know in novel situations. Some people complain that PISA is unfair, because it confronts students with tasks they have not dealt with before, but if you take that line, then you should consider life unfair, because in this fast-changing world, that is precisely what will expect students later in life. You will see that in the callout box. Students also provided data on their socio-economic context, their schools and their attitudes and engagement with school and learning. In addition, PISA collected data from parents, principals and system leaders to yield insights on school policies, practices, resources and institutional factors that help explain performance differences.

Mathematics in PISA The real world The mathematical World
Making the problem amenable to mathematical treatment A model of reality A mathematical model Understanding, structuring and simplifying the situation Using relevant mathematical tools to solve the problem A real situation Validating the results Mathematical results Real results Interpreting the mathematical results

Average performance of 15-year-olds in reading – extrapolate and apply
High reading performance Average performance of 15-year-olds in reading – extrapolate and apply Woodson High School (W.T.), serves middle-class students Langley High School, serves upper-class neighbourhoods Performance distribution in US 18% do not reach baseline Level 2 (16% when excluding immigrants) (Finland 6%, Canada 9%) Economic cost: 72 trillion $ 10% are top performers (Shanghai 20%) North Star Academy College Preparatory High School, serves students in disadvantaged neighbourhoods (social context like in Chile, Portugal or Shanghai) Suburban schools Northeast Midwest Not just about poor schools in poor neighbourhoods West Urban schools South This chart illustrates the reading literacy scale, from below the OECD average, marked in red, to around the OECD average, marked in yellow, to high performance, marked in green. You see that Shanghai, Korea, Hong-Kong, Singapore, New Zealand and Japan in Asia, Finland in Europe and Canada in North America do particularly well. Some will attribute the high performance of the East Asian countries to the Confucian culture, but be careful, for example, Chinese Taipei or Macao come from that same culture and don’t do particularly well. They tend to excel in rote learning, but don’t do well on the kind of creative skills that PISA values. … 17 countries perform below this line Low reading performance

Performance on PISA scale
High reading performance Average performance of 15-year-olds in reading – extrapolate and apply Woodson High School (W.T.), serves middle-class students Langley High School, serves upper-class neighbourhoods Reading Pilot Results PISA 2009 Results Schools in the United States Schools in the Shanghai-China Schools in Mexico Performance distribution in US 18% do not reach baseline Level 2 (16% when excluding immigrants) (Finland 6%, Canada 9%) Economic cost: 72 trillion $ 10% are top performers (Shanghai 20%) North Star Academy College Preparatory High School, serves students in disadvantaged neighbourhoods (social context like in Chile, Portugal or Shanghai) 10% above 25% above 50% above/below 25% below 10% below Suburban schools Northeast Midwest 10% above 25% above 50% above/below 25% below 10% below West Performance on PISA scale Urban schools 10% above 25% above 50% above/below 25% below 10% below South This chart illustrates the reading literacy scale, from below the OECD average, marked in red, to around the OECD average, marked in yellow, to high performance, marked in green. You see that Shanghai, Korea, Hong-Kong, Singapore, New Zealand and Japan in Asia, Finland in Europe and Canada in North America do particularly well. Some will attribute the high performance of the East Asian countries to the Confucian culture, but be careful, for example, Chinese Taipei or Macao come from that same culture and don’t do particularly well. They tend to excel in rote learning, but don’t do well on the kind of creative skills that PISA values. … 17 countries perform below this line Low reading performance

Average performance of 15-year-olds in reading – extrapolate and apply
High reading performance Average performance of 15-year-olds in reading – extrapolate and apply Woodson High School (W.T.), serves middle-class students Langley High School, serves upper-class neighbourhoods Performance distribution in US 18% do not reach baseline Level 2 (16% when excluding immigrants) (Finland 6%, Canada 9%) Economic cost: 72 trillion $ 10% are top performers (Shanghai 20%) Suburban schools Northeast Midwest West Urban schools South This chart illustrates the reading literacy scale, from below the OECD average, marked in red, to around the OECD average, marked in yellow, to high performance, marked in green. You see that Shanghai, Korea, Hong-Kong, Singapore, New Zealand and Japan in Asia, Finland in Europe and Canada in North America do particularly well. Some will attribute the high performance of the East Asian countries to the Confucian culture, but be careful, for example, Chinese Taipei or Macao come from that same culture and don’t do particularly well. They tend to excel in rote learning, but don’t do well on the kind of creative skills that PISA values. … 17 countries perform below this line Low reading performance

Average performance of 15-year-olds in science – extrapolate and apply
High reading performance Average performance of 15-year-olds in science – extrapolate and apply High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

Durchschnittliche Schülerleistungen im Bereich Mathematik
High reading performance 2009 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

High performing systems often prioritize the quality of teachers over the size of classes Contribution of various factors to upper secondary teacher compensation costs per student as a percentage of GDP per capita (2004) Percentage points The red dot indicates classroom spending per student, relative to the spending capacity of countries, the higher the dot, the more of its GDP a country invests. High salaries are an obvious cost driver. You see Korea paying their teachers very well, the green bar goes up a lot. Korea also has long school days, another cost driver, marked here by the white bar going up. Last but not least, Korea provides their teachers with lots of time for other things than teaching such as teacher collaboration and professional development, which costs money as well. So how does Korea finances all of this? They do this with large classes, the blue bar pulls costs down. If you go to the next country on the list, Luxembourg, you see that the red dot is about where it is for Korea, so Luxembourg spends roughly the same per student as Korea. But parents and teachers in Luxembourg mainly care about small classes, so policy makers have invested mainly into reducing class size, you see the blue bar as the main cost driver. But even Luxembourg can only spend its money once, and the result is that school days are short, teacher salaries are average at best and teachers have little time for anything else than teaching. Finland and the US are a similar contrast. Countries make quite different spending choices. But when you look at this these data long enough, you see that many of the high performing education systems tend to prioritise the quality of teachers over the size of classes.

High reading performance 2009 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

High reading performance 2000 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities For a moment, let us go back to the year Remember, that was the year before the iPod was invented. This is how the world looked then in terms of PISA literacy performance. The first thing you can see is that the bubbles were generally much smaller, we spent around a quarter less per student. So was the increase in spending matched by better outcomes? Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

High reading performance Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Not generally, but I want to highlight a few countries that have seen impressive improvements. Korea’s average performance was already high in 2000, yet Korean policy makers were concerned that only a small elite achieved levels of excellence in the PISA assessment of reading back then. Within less than a decade, Korea was able to double the share of students demonstrating excellence in reading literacy. If you raise performance at the top end of the scale only, that of course increases disparities in outcomes, and you see Korea moving slightly backwards on the equity dimension. But it is still a strong and equitable performer. At the other end of the spectrum we have seen impressive gains as well. Chile provides an example. In 2000, Chile performed so low that you would not even see it on this chart. 9 years later the performance of 15-year-olds was roughly a school year better. A major overhaul of Poland’s school system helped to dramatically reduce performance variability among schools, turn around the lowest performing schools and raise overall performance by the equivalent of more than half a school year. So is Portugal, which improved both overall performance and equity. And so did Hungary. Finally, Germany was jolted into action when PISA 2000 revealed below-average performance and large social disparities in their results, and has been able to make progress on both fronts. And this is not the complete list, also Peru, Indonesia, Latvia, Israel and Brazil raised their outcomes and in Mathematics impressive improvements have been realised by Mexico, Brazil, Turkey, Greece, Portugal, Italy and Germany. When it comes to Japan, the results have remained unchanged, and that is true for reading, mathematics and science. I know that some people have been very concerned in Japan about an apparent decline in student performance in 2003, but when you look at long-term trends properly, you will see that there has been no decline in Japan’s student performance. There has, however, been a widening gap in student and school performance in reading, essentially because the share of top-performers in reading has grown while similar improvements have not been witnessed at the bottom end of the performance distribution. Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

Changes in performance by type of task
Increase percentage correct Japan In my view, one of the most important improvements in Japan has been the significant rise in the performance of Japanese students on open-ended tasks, the kind of tasks that require students to create an answer, rather than to just reproduce an answer from a multiple-choice task. In other words, Japan is advancing fastest on the kind of ‘new skills’ that I spoke about at the beginning. Japan OECD OECD

School performance and socio-economic background United States
Private school Public school in rural area Public school in urban area School performance and schools’ socio-economic background Student performance and students’ socio-economic background within schools Student performance 700 There is hope for improvement not just internationally, but there is an amazing number of US schools that succeed and, and some of them succeed againsts all odds. Let me illustrate this. The horizontal axis on this chart represents social disadvantage, the vertical one the performance of schools. Each dot is one school. As you can see, there is a clear relationship between the social background of U.S. schools and how well their students do. But you find a surprising number of exceptions. If you look at schools in social disadvantage, some of them do very well, while others perform poorly. Similarly, if you look at schools with children mainly from privileged social backgrounds, you see that poor performance is not just a question of poor kids in poor neighbourhoods, but one of many kids in many neighbourhoods. If the US would achieve what Poland achieved, your economy would be 40 trillion dollars richer over the lifetime of today’s 15-year-olds. If you contrast that with Finland, you will find that only 5% of the performance variation lies between schools there, virtually every school succeeds irrespective of its social context. Now you might say Finland is a special case because it is not as diverse as the U.S, but then take Shanghai, a socio-economically very heterogeneous province and you see also here a fairly consistent high level of performance among schools. That has not come about by chance, but is the result of a concerted effort to turn “weaker schools” around into stronger schools. If you are a successful vice principal in a high performing school in Shanghai, and you want to make a career, you will need to take on a disadvantaged school and show that you can turn it around. And the system will not leave you alone with that challenge. Listen to how the Director of the Education Bureau in Pudong explains that success [Video]. Advantage PISA Index of socio-economic background Disadvantage

   Policies and practices Learning climate Discipline
Policy Policies and practices Learning climate Discipline Teacher behaviour Parental pressure Teacher-student relationships Dealing with heterogeneity Grade repetition Prevalence of tracking Expulsions Ability grouping (all subjects) Standards /accountability Nat. examination Standardised tests Posting results Governing schools School autonomy (content) Choice and competition Private schools Managing resources Prioritising pay Student-staff ratios Length of pre-school R System   School  E Equity Let me briefly summarise the influences that we have measured in PISA.

Learning from the World’s top performers

PISA analytic framework
Domain 1 Domain 2 Domain 3 Outputs and Outcomes impact of learning Policy Levers shape educational outcomes Antecedents contextualise or constrain ed policy Individual learner Level A Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour Socio-economic background of learners Instructional settings Level B Quality of instructional delivery Teaching, learning practices and classroom climate Student learning, teacher working conditions Schools, other institutions Level C Output and performance of institutions The learning environment at school Community and school characteristics Country or system Level D Social & economic outcomes of education Structures, resource alloc and policies National educ, social and economic context

Lessons from PISA on successful education systems
Commitment to universal achievement Goals, gateways, instructional systems Capacity at point of delivery Incentives and accountability Resources where they yield most A learning system Coherence A commitment to education and the belief that competencies can be learned and therefore all children can achieve Universal educational standards and personalisation … as opposed to a belief that students have different destinations to be met with different expectations, and selection/stratification as the approach to heterogeneity Clear articulation who is responsible for ensuring student success and to whom Lessons from PISA on successful education systems First, there is no question that most nations declare that education is important. But the test comes when these commitments are weighed against others. How do countries pay teachers, compared to other highly-skilled workers? How are education credentials weighed against other qualifications when people are being considered for jobs? Would you want your child to be a teacher? How much attention do the media pay to schools and schooling? What we have learned from PISA is that in high performing systems political and social leaders have persuaded citizens to make choices that show they value education more than other things. But placing a high value on education is only part of the equation. Another part is belief in the possibilities for all children to achieve success. In some countries, students are separated into different tracks at an early age, reflecting a notion shared by teachers, parents and citizens that only a subset of the nation’s children can or need to achieve world class standards. Our analysis shows that systems that track students in this way, based differing expectations for different destinations, tend to be fraught with large social disparities. By contrast, the best performing systems deliver strong and equitable learning outcomes across very different cultural and economic contexts. In Finland, Japan, Singapore, Shanghai-China and Hong Kong-China, parents, teachers and the public at large share the belief that all students are capable of achieving high standards and need to do so, and they provide great examples for how public policy can support the achievement of universal high standards.

Commitment to universal achievement Goals, gateways, instructional systems Capacity at point of delivery Incentives and accountability Resources where they yield most A learning system Coherence Clear ambitious goals that are shared across the system and aligned with high stakes gateways and instructional systems Well established delivery chain through which curricular goals translate into instructional systems, instructional practices and student learning (intended, implemented and achieved) High level of metacognitive content of instruction Lessons from PISA on successful education systems High-performing education systems also share clear and ambitious standards across the board. Everyone knows what is required to get a given qualification, both in terms of the content studied and the level of performance needed to earn it. Students cannot go on to the next stage—be it in work or in further education—unless they show that they are qualified to do so. They know what they have to do to realise their dream, and they put in the work that is needed to do it. As discussed in the 2009 edition of OECD’s Education at a Glance¸ over the past decade, assessments of student performance have become common in many OECD countries – and the results are often widely reported and used in both public and more specialised debate. However, the rationale for assessments and the nature of the instruments used vary greatly within and across countries. Methods employed in OECD countries include different forms of external assessment, external evaluation or inspection, and schools’ own quality assurance and self-evaluation efforts. One aspect relating to accountability systems concerns the existence of standards-based external examinations. These are examinations that focus on a specific school subject and assess a major portion of what students who are studying this subject are expected to know or be able to do (Bishop, 1998, 2001). Essentially, they define performance relative to an external standard, not relative to other students in the classroom or school. These examinations usually have a direct impact on students’ education – and even on their futures – and may thus motivate students to work harder. Other standardised tests, which may be voluntary and implemented by schools, often have only indirect consequences for students. For teachers, standardised assessments can provide information on students’ learning needs and can be used to tailor their instruction accordingly. In some countries, such as Brazil, Hungary, Italy, Malaysia, Mexico, Poland and the Slovak Republic, such tests are also used to determine teachers’ salaries or to guide professional development (for data, see the 2009 edition of Education at a Glance ). At the school level, information from standardised tests can be used to determine the allocation of additional resources, and what interventions are required to establish performance targets and monitor progress. Across OECD countries, students in school systems that require standards-based external examinations perform, on average, over 16 points higher than those in school systems that do not use such examinations (Figure IV.2.6a). Among OECD countries, there are standards-based external examinations for secondary school students in the Czech Republic, Denmark, Estonia, Finland, France, Hungary, Iceland, Ireland, Israel, Italy, Japan, Korea, Luxembourg, the Netherlands, New Zealand, Norway, Poland, the Slovak Republic, Slovenia, Turkey and the United Kingdom. In Australia, these examinations cover 81% of secondary students, in Canada 51% and in Germany 35%. In Austria, Belgium, Chile, Greece, Mexico, Portugal, Spain, Sweden and Switzerland, such examinations do not exist or only in some parts of the system (Table IV.3.11). In PISA 2009, school principals were asked to report on the types and frequency of assessment used: standardised tests, teacher-developed tests, teachers’ judgemental ratings, student portfolios or student assignments. Some 76% of students in OECD countries are enrolled in schools that use standardised tests. Standardised tests are relatively uncommon in Slovenia, Belgium, Spain, Austria and Germany, where less than half the 15-year-olds attend schools that assess students through standardised tests. In contrast, the use of standardised tests is practically universal in Luxembourg, Finland, Korea, the United States, Poland, Denmark, Sweden and Norway, where over 95% of students attend schools that use this assessment at least once a year (Table IV.3.10). In Japan, 65% of students are in schools that use standardised tests.

Commitment to universal achievement Goals, gateways, instructional systems Capacity at point of delivery Incentives and accountability Resources where they yield most A learning system Coherence Capacity at the point of delivery Attracting, developing and retaining high quality teachers and school leaders and a work organisation in which they can use their potential Instructional leadership and human resource management in schools and districts Keeping teaching an attractive profession System-wide career development Lessons from PISA on successful education systems Third, the quality of an education system cannot exceed the quality of its teachers and principals. Just like companies, high quality school systems pay attention to how they select and train their staff. They watch how they improve the performance of those who are struggling; how structure teachers’ pay packets; and how they reward their best teachers. They provide an environment in which teachers work together to frame good practice. That is where teachers conduct field-based research to confirm or disprove the approaches they develop, and they judge their colleagues by the degree to which they use these practices in their classrooms. Listen to what the Finnish Minister had to say about that. Sahl, SIN

Commitment to universal achievement Goals, gateways, instructional systems Capacity at point of delivery Incentives and accountability Resources where they yield most A learning system Coherence Incentives, accountability, knowledge management Aligned incentive structures For students How gateways affect the strength, direction, clarity and nature of the incentives operating on students at each stage of their education Degree to which students have incentives to take tough courses and study hard Opportunity costs for staying in school and performing well For teachers Make innovations in pedagogy and/or organisation Improve their own performance and the performance of their colleagues Pursue professional development opportunities that lead to stronger pedagogical practices A balance between vertical and lateral accountability Effective instruments to manage and share knowledge and spread innovation – communication within the system and with stakeholders around it A capable administration with authority and legitimacy to act Lessons from PISA on successful education systems Fourth, as you have seen, success has to do with incentives and accountability, and how these are aligned in the system. It has also to do with how vertical accountability to superiors is balanced with horizontal or professional accountability towards peers, how knowledge is shared and spread. For students this affects: How gateways affect the strength, direction, clarity and nature of the incentives operating on students at each stage of their education; as well as the degree to which students have incentives to take tough courses and study hard and the opportunity costs for staying in school and performing well. It also means providing incentives for teachers to make innovations in pedagogy and/or organisation, improve their own performance and the performance of their colleagues, and to pursue professional development opportunities that lead to stronger pedagogical practices. High performing systems tend to provide a balance between vertical and lateral accountability and have effective instruments to manage and share knowledge and spread innovation – and that means both communication within the system and with stakeholders around it.

School autonomy, accountability and student performance Impact of school autonomy on performance in systems with and without accountability arrangements PISA score in reading Posting achievement data publicly What is important is that autonomy and accountability need to be seen in conjunction. Data from PISA show that in school systems where most schools post achievement data publicly, schools with greater discretion in managing their resources tend to show higher levels of performance. In school systems where schools do not post achievement data publicly, a student who attends a school with greater autonomy in resource management than the average OECD school tends to perform 3.2 score points lower in reading than a student attending a school with an average level of autonomy. In contrast, in school systems where schools do post achievement data publicly, a student who attends a school with above-average autonomy scores 2.6 points higher in reading than a student attending a school with an average level of autonomy (Table IV.2.5).

Local responsibility and system-level prescription
Trend in OECD countries Finland and other improvers today Every school an effective school System-level prescription ‘Tayloristic’ work organisation Schools today The industrial model, detailed prescription of what schools do Schools tomorrow? Building capacity Schools leading reform Teachers as ‘knowledge workers’

Private schools perform better Public schools perform better
Public and private schools % Score point difference Private schools perform better Public schools perform better School education takes place mainly in public schools. Nevertheless, with an increasing variety of educational opportunities, programmes and providers, governments are forging new partnerships to mobilise resources for education and to design new policies that allow all stakeholders to participate more fully and share costs and benefits more equitably. Privately funded education is not only a way of mobilising resources from a wider range of funding sources, it is sometimes also considered a way of making education more cost-effective. Publicly financed schools are not necessarily also publicly managed. Instead, governments can transfer funds to public and private educational institutions according to various allocation mechanisms. Indeed, publicly funded private schools are the most common model of private education in OECD countries (see section on school choice, above). Across OECD countries, 15% of students are enrolled in privately managed schools that are either privately or government funded, although in many countries government authorities retain significant control over these schools, including the power to shut down non-performing schools. Enrolment in privately managed schools exceeds 50% of 15-year-old students in the Netherlands, Ireland and Chile, and in Australia and Korea between 35% and 40% of students are enrolled in such schools. In Japan, 29 % of students attend schools that are privately managed and 71% attend schools that are publicly managed. In contrast, in Turkey, Iceland and Norway, more than 98% of students attend schools that are publicly managed (Table IV.3.9). On average across OECD countries, privately managed schools display a performance advantage of 30 score points on the PISA reading scale (Table IV.3.9). However, once the socio-economic backgrounds of students and schools is accounted for, public schools come out with a slight advantage of seven score points, on average across OECD countries. In Japan, public and privately managed schools do not show a performance difference before accounting for the socio-economic background, and public schools outperform private schools after accounting for students’ and schools’ socio-economic backgrounds.

Commitment to universal achievement Goals, gateways, instructional systems Capacity at point of delivery Incentives and accountability Resources where they yield most A learning system Coherence Lessons from PISA on successful education systems Investing resources where they can make most of a difference Alignment of resources with key challenges (e.g. attracting the most talented teachers to the most challenging classrooms) Effective spending choices that prioritise high quality teachers over smaller classes The most impressive outcome of world class education systems is perhaps that they deliver high quality learning consistently across the entire education system so that every student benefits from excellent learning opportunity. To achieve this, they invest educational resources where they can make most of a difference, they attract the most talented teachers into the most challenging classroom, and they establish effective spending choices that prioritise the quality of teachers. Let me come back to the example of Shanghai once more here. Let us have a look at the struggling schools six years later. Research usually shows a weak relationship between educational resources and student performance, with more variation explained by the quality of human resources (i.e. teachers and school principals) than by material and financial resources, particularly among industrialised nations. The generally weak relationship between resources and performance observed in past research is also seen in PISA. At the level of the education system, and net of the level of national income, the only type of resource that PISA shows to be correlated with student performance is the level of teachers’ salaries relative to national income (Figure IV.2.8). Teachers’ salaries are related to class size in that if spending levels are similar, school systems often make trade-offs between smaller classes and higher salaries for teachers. The findings from PISA suggest that systems prioritising higher teachers’ salaries over smaller classes, such as those in Japan and Korea, tend to perform better. The lack of correlation between the level of resources and performance among school systems does not mean that resource levels do not affect performance at all. Rather, it implies that, given the variation in resources observed in PISA, they are unrelated to performance or equity. A school system that lacks teachers, infrastructure and textbooks will almost certainly perform at lower levels; but given that most school systems in PISA appear to satisfy the minimum resource requirements for teaching and learning, the lack of a relationship between many of the resource aspects and both equity and performance may result simply from a lack of sufficient variation among OECD countries. CHN

Commitment to universal achievement Goals, gateways, instructional systems Capacity at point of delivery Incentives and accountability Resources where they yield most A learning system Coherence A learning system An outward orientation to keep the system learning, technology, international benchmarks as the ‘eyes’ and ‘ears’ of the system Recognising challenges and potential future threats to current success, learning from them, designing responses and implementing these Lessons from PISA on successful education systems Some of the most successful systems are also actively looking outward, realising that the benchmark for success is no longer simply improvement by national standards, but the best performing systems internationally. Whether Singapore is interested in designing a better sewer system, retirement system or school system, it sends key people in the relevant sector to visit those countries that are the world’s best performers in those areas with instructions to find out how they do it, and to put together a design for Singapore that is superior to anything that they have seen anywhere. SIN

Commitment to universal achievement Goals, gateways, instructional systems Capacity at point of delivery Incentives and accountability Resources where they yield most A learning system Coherence Coherence of policies and practices Alignment of policies across all aspects of the system Coherence of policies over sustained periods of time Consistency of implementation Fidelity of implementation (without excessive control) Lessons from PISA on successful education systems Last but not least, in high performing systems these policies and practices are aligned across all aspects of the system, they are coherent over sustained periods of time, and they are consistently implemented. And PISA shows, success is within the reach for nations that have the capacity to creating and executing policies with maximum coherence in the system. Of course, the path to reform is not easy and it can be fraught with political controversy. Moving away from administrative and bureaucratic control toward professional norms of control can be counterproductive if a nation does not yet have teachers and schools with the capacity to implement these policies and practices. Pushing authority down to lower levels can be as problematic if there is not agreement on what the students need to know and should be able to do. Recruiting high quality teachers is not of much use if those who are recruited are so frustrated by what they perceive to be a mindless system of initial teacher education that they will not participate in it and turn to another profession. Or if they become school teachers, but are so turned off by the bureaucratic forms of work organisation they find there that they leave teaching for some other occupation. So this is all about alignment. CAN

A tool for greater resolution for schools and local educators PISA for Schools In the U.S.: OECD Test for Schools (based on PISA)

OECD Test for Schools (based on PISA) Uses of the assessment tool
Provides information on the competencies, knowledge, skills and engagement of students, and the learning environment at the school (teacher-student relations, disciplinary climate, student confidence and attitudes towards mathematics and science)  comparable to global PISA scales Tool in support of peer-learning and improvement Can be used by schools, networks of schools and districts It is important to consider that both the cognitive test items (questions) and the student and school questionnaires administered are integral parts of the assessment.

OECD Test for Schools (based on PISA) Uses of the assessment tool
The test is not… A mandated standardised test Intended to influence – in of itself – everyday teaching practices An accountability tool or a tool for “rankings” or “league tables”

OECD Test for Schools (based on PISA) What does the actual assessment look like?
Experience for students similar to that of the main PISA tests: ~ 3.25 hours (with breaks and student questionnaire) Currently focussed on three areas (reading, math and science) equally represented (over 90 minutes of assessment items, with a mix of open-ended and multiple-choice items) Student sample size per school (target): 75 (some schools tested over 100 students) Contextual information  questionnaires for students and school authorities Paper and pencil for first phase… The field trialling of test items and the equating study were conducted before the pilot trial that is described in the presentation.

OECD Test for Schools (based on PISA) Overview of results from the pilot
126 schools in Canada, U.K. and U.S. 8 000 students tested between May and October 2012 Each participating school (and district) decided whether to make their results public – (examples of practices…) Convenience samples in all three countries – variety of school types but not meant to be statistically representative!

Performance on PISA scale Socioeconomic status (ESCS)
OECD Test for Schools (based on PISA) Overview of results from the pilot Schools in the U.S. that participated in PISA 2009 Reading Pilot Schools in the U.S. Performance on PISA scale Key Messages: Reading Distribution of schools in the US in PISA as we would expect – Correlation of advantage with performance However, large variation of performance across the socio-economic spectrum and also, among schools with the same “average” results – they are doing so with students from very different backgrounds. Explanation: The assessment provides school-level results in terms of performance in reading, mathematics and science on the PISA scales. On this chart, the vertical axis (y-axis) shows increasing levels of performance as one goes from bottom to top. On the bottom, the x-axis, a measure of socio-economic advantage and disadvantage is shown – as one moves from left to right, the relative advantage of students increases. The bubbles shown are the schools that participated in PISA 2009 in the United States and their results relative to both cognitive outcomes (in this case reading, vertical scale) and socio-economic status (horizontally along the bottom). The size of the bubbles represent the relative size of the students enrolled in the school. The dark blue bubbles, therefore, represent the relative position of the schools that participated in the United States in PISA 2009. Now, we’ll map the schools in the United States that participated in the pilot… these are the bright orange bubbles. As you can see, the general tendency in terms of the relationship between socio-economic status of students in the schools and the average performance is correlated – the more advantaged students tends to result in higher average performance of schools. Note: Size of bubbles is proportional to the number of students enrolled at the school. Socioeconomic status (ESCS)

Performance on PISA scale Socioeconomic status (ESCS)
OECD Test for Schools (based on PISA) Overview of results from the pilot Schools in the U.S. that participated in PISA 2009 Mathematics Pilot Schools in the U.S. Performance on PISA scale U.S. average PISA 2009 U.S. average PISA 2009 Key Messages: Mathematics Note: Size of bubbles is proportional to the number of students enrolled at the school. Socioeconomic status (ESCS)

Schools with students from similar socioeconomic backgrounds
OECD Test for Schools (based on PISA) Overview of results from the pilot Schools in the U.S. that participated in PISA 2009 Science Pilot Schools in the U.S. U.S. average PISA 2009 Performance on PISA scale Key Messages: SCIENCE results to use for ESCS discussion We see large variation in performance among schools with students from similar socio-economic backgrounds, with average students (starting point) and then looking at students from advantaged backgrounds (animation) as well as schools with disadvantaged students. This is one learning opportunity – what has occurred or is occurring in these schools to make the difference? But just as importantly, we see that some schools that have similar average performance may be doing so with students from very diverse backgrounds (animation – horizontal zoom bar). Here we see schools that have similar performance in science – these may be considered “under-performing schools” as they have average performance below the United States average in science in PISA 2009, but the important point is that these schools are doing so with students from very diverse backgrounds (almost 2 standard deviations). With the new school-level assessment, we are able to look closer at what is occurring in these schools but also to look at differences within and between schools. Let’s take a closer look at this group of schools…. (next slides show results for this selection of “under-performing schools”) U.S. average PISA 2009 Note: Size of bubbles is proportional to the number of students enrolled at the school. Socioeconomic status (ESCS)

Performance on PISA scale
OECD Test for Schools (based on PISA) Overview of results from the pilot Reading Pilot Results PISA 2009 Results Schools in the United States Schools in the Shanghai-China Schools in Mexico 10% above 25% above 50% above/below 25% below 10% below 10% above 25% above 50% above/below 25% below 10% below Performance on PISA scale 10% above 25% above 50% above/below 25% below 10% below

Reading What does the same mean mean? 543 and 543 Key Messages: SCIENCE results to use for ESCS discussion We see large variation in performance among schools with students from similar socio-economic backgrounds, with average students (starting point) and then looking at students from advantaged backgrounds (animation) as well as schools with disadvantaged students. This is one learning opportunity – what has occurred or is occurring in these schools to make the difference? But just as importantly, we see that some schools that have similar average performance may be doing so with students from very diverse backgrounds (animation – horizontal zoom bar). Here we see schools that have similar performance in science – these may be considered “under-performing schools” as they have average performance below the United States average in science in PISA 2009, but the important point is that these schools are doing so with students from very diverse backgrounds (almost 2 standard deviations). With the new school-level assessment, we are able to look closer at what is occurring in these schools but also to look at differences within and between schools. Let’s take a closer look at this group of schools…. (next slides show results for this selection of “under-performing schools”) Level 1 and be`low Level Level Level Level Level 6

Content of reports I. Introduction: Understanding your school’s results What students in your school know and can do in Reading, Mathematics and Science Student engagement and the learning environment at your school (teacher-student relations, disciplinary climate, student confidence and attitudes towards mathematics and science) Your School Compared with similar schools in Your Country Your School’s Results in an international context School name School District State United States How your school compares internationally OECD Test for Schools Pilot Trial 2012 The schools that successfully participated and tested in the pilot received an electronic school report. Each school report contains a summary and the five sections described here. - Only the school authorities and the district-level contacts, if appropriate, received access to the school reports. The OECD did not publicly release the reports or the results for schools without prior and written consent from the participating schools. The information presented in the school reports on students’ achievement, their engagement, and the teaching and learning environment should stimulate further reflection and discussion among school staff and local educational authorities. Active hyperlinks to OECD research, reports and video resources were included in the reports. Example school reports available at the OECD Website.

OECD Test for Schools (based on PISA) What next
OECD Test for Schools (based on PISA) What next? Opportunities for development… Receiving feedback… District-level information based on school-level results Reporting on country-specific variables Examples of policies, practices and insights based on feedback from pilot participants (and results from PISA 2012) Peer-to-peer exchanges… examples of practices and network-specific follow-up

OECD Test for Schools (based on PISA) What now? Next Steps
Availability of the assessment in the United States Starting in September 2013 (for testing) Interested schools and districts can begin planning now To sign up: OECD will accredit a service provider

Thank you ! Find out more about PISA at… OECD www.pisa.oecd.org
All national and international publications The complete micro-level database … and remember: Without data, you are just another person with an opinion Thank you !

1 in 3 of the pilot schools are Title I
OECD Test for Schools (based on PISA) Overview of results from the pilot 105 Schools in the U.S.: 48 school districts in 22 states 1 in 3 of the pilot schools are Title I 6 magnets, 6 charters and 1 private school 3 of 4 schools with 20% or more of students from diverse ethnic backgrounds Several schools from the same districts and networks (EdLeader21, Fairfax,…) 24 schools from the network EdLeader21 (Ken Kay and Valerie Greenhill’s group) 18 schools that are part of Project Lead the Way (Kern Family Foundation) 10 schools from Fairfax Public Schools participated.

Local, national and international benchmarking for improvement

Similar presentations

Presentation on theme: "Local, national and international benchmarking for improvement"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Local, national and international benchmarking for improvement

Similar presentations

Presentation on theme: "Local, national and international benchmarking for improvement"— Presentation transcript:

Similar presentations

About project

Feedback