Quality and equity in educational outcomes Seeing school systems through the prism of PISA Organisation for Economic Cooperation and Development (OECD) Campbell What Works seminar 9 November 2006 Dr. Karin Zimmer OECD / Directorate for Education

In the dark… …all students, schools and education systems look the same… But with a little light….

But with a little light…. …important differences become apparent…. In the dark… …all students, schools and education systems look the same…

Baseline qualifications A world of change Approx. by % of persons with upper secondary qualfications in age groups 55-64, 45-55, und years

Overview

The PISA approach Measuring the quality of learning outcomes

OECD countries participating from PISA 2000 OECD countries participating from PISA from 2003 OECD partner countries participating from PISA 2000 OECD partner countries participating from PISA 2003 OECD partner countries participating from PISA 2006 PISA country participation Key features of PISA 2003 r Information collected volume of the tests –3½ hours of mathematics assessment, less than half in multiple-choice format –1 hour for each of reading, science and problem solving each student –2 hours on paper-and-pencil tasks (subset of all questions) –½ hour for questionnaire on background, learning environment, engagement and motivation school principals –questionnaire (school demography, learning environment quality ) r Coverage PISA covers roughly nine tens of the world economy Representative samples of between 3,500 and 50,000 students

Deciding whom to assess... grade-based sample OR age-based sample For PISA, the OECD countries chose the latter, selecting 15-year-olds in school as the population.

Deciding what to assess... looking back at what students were expected to have learned …or… looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

Three broad categories of key competencies Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Learning strategies Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively PISA concept of literacy Accessing, managing, integrating and evaluating written information in order to develop ones knowledge and potential, and to participate in, and contribute to, society

Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Reading literacy Using, interpreting and reflecting on written material

Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Scientific literacy Using scientific knowledge, identifying scientific questions, and drawing evidence-based conclusions to understand and make decisions about the natural world

Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Mathematical literacy Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways

Where we are - and where we can be What PISA shows students can do Examples of the best performing countries

Average performance of 15-year-olds in mathematics High mathematics performance Low mathematics performance

Mathematical literacy in PISA The real world The mathematical World A real situation A model of reality A mathematical model Mathematical results Real results Understanding, structuring and simplifying the situation Making the problem amenable to mathematical treatment Interpreting the mathematical results Using relevant mathematical tools to solve the problem Validating the results

Average performance of 15-year-olds in mathematics Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance

Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance

Student performance School performance and schools’ socio- economic background - Germany Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES within schools School performance and school SES

Student performance School performance and schools’ socio- economic background - Denmark Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES OECD

Student performance School performance and schools’ socio- economic background - Finland Advantage PISA Index of social background Disadvantage Figure 4.13 Student performance and student SES Student performance and student SES within schools School performance and school SES School proportional to size

Is it all innate ability? Variation in student performance OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p

Variation of performance between schools Variation of performance within schools Is it all innate ability? Variation in student performance in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383. In some countries, parents can rely on high and consistent standards across schools In Canada, Denmark, Finland, Iceland and Sweden average student performance is high… …and largely unrelated to the individual schools in which students are enrolled. In other countries, large performance differences among schools persist In Austria, Belgium, Germany, Hungary, Italy, Japan, the Netherlands and Turkey, most of the performance variation among schools lies between schools… …and in some of these countries, most notably those that are highly stratified, a large part of that variation is explained by socio-economic inequalities in learning opportunities

How can we get there? Levers for policy that emerge from international comparisons… …and what countries have done with the findings

Money matters but other things do too Mexico Greece Portugal Italy Spain Germany Austria Ireland United States Norway Korea Czech republic Slovak republic Poland Hungary Finland Netherlands Canada Switzerland Iceland Denmark France Sweden Belgium Australia Japan R 2 = 0.28 Cumulative expenditure (US$) Performance in mathematics r Spending per student is positively associated with average student performance… …but not a guarantee for high outcomes Australia, Belgium, Canada, the Czech Republic, Finland, Japan, Korea and the Netherlands do well in terms of “value for money”… …while some of the big spenders perform below-average

High ambitions and clear standards Access to best practice and quality professional development r Sympathy doesn’t raise standards – aspiration does PISA suggests that students and schools perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations –Among these aspects, students’ perception of teacher-student relations and classroom disciplinary climate display the strongest relationships

Challenge and support Weak support Strong support Low challenge High challenge Strong performance Systemic improvement Poor performance Improvements idiosyncratic Conflict Demoralisation Poor performance Stagnation

Governance of the school system r In many of the best performing countries School-based decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities The provision of standards and curricula at national/subnational levels is combined with advanced evaluation and support systems –That are implemented by professional agencies Process-oriented assessments and/or centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress r Standard setting and equity-related goals Key objectives: –Raise educational aspirations, establish transparency over educational objectives, reference framework for teachers Approaches range from definition of broad educational goals up to formulation of concise performance expectations Some countries go beyond establishing educational standards as mere yardsticks and use performance benchmarks that students at particular age or grade levels should reach Instruments –Minimum standards, targets defining excellence, normative performance benchmarks r Monitoring and equity-related goals Diverging views how evaluation and assessment can and should be used –Some see them primarily as tools to reveal best practices and identify shared problems in order to encourage teachers and schools to improve and develop more supportive and productive learning environments –Others extend their purpose to support contestability of public services or market-mechanisms in the allocation of resources –e.g. by making comparative results of schools publicly available to facilitate parental choice or by having funds following students Differences in type of performance benchmarks being used and reported for the various stakeholders involved, including parents, teachers and schools

High ambitions Access to best practice and quality professional development Accountability and intervention in inverse proportion to success Devolved responsibility, the school as the centre of action

Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance

Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance School with responsibility for deciding which courses are offered High degree of autonomy Low degree of autonomy

Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance Early selection and institutional differentiation High degree of stratification Low degree of stratification

Strong ambitions Access to best practice and quality professional development Accountability Devolved responsibility, the school as the centre of action Integrated educational opportunities Individualised learning

High ambitions Access to best practice and quality professional development Accountability and intervention in inverse proportion to success Individualised learning Devolved responsibility, the school as the centre of action Integrated educational opportunities

 

Creating a knowledge-rich profession in which schools and teachers have the authority to act, the necessary knowledge to do so wisely, and access to effective support systems The tradition of education systems has been “knowledge poor” The future of education systems needs to be “knowledge rich” National prescription Professional judgement Informed professional judgement, the teacher as a “knowledge worker” Informed prescription Uninformed professional judgement, teachers working in isolation Uninformed prescription, teachers implement curricula

–All national and international publications –The complete micro-level database