1. Seeing schools through the prism of PISA An international comparative perspective on quality and equity in education systems Organisation for Economic Cooperation and Development (OECD) Tokyo 24 June 2005 Andreas Schleicher Head, Indicators and Analysis Division OECD Directorate for Education

3. In the dark, all education systems look the same… But with a little light…. 暗がりのなかでは、どの学校も教育システムも同じように見える … だが、少し光を当てると ….

4. But with a little light…. …important differences become apparent… … 重要な違いが明らかになってくる …. だが、少し光を当てると ….

5. Average performance of 15-year-olds in mathematics High mathematics performance Low mathematics performance

6. Overview

7. The PISA approach Measuring the quality of learning outcomes

8. Three broad categories of key competencies Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Learning strategies Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively PISA concept of literacy Accessing, managing, integrating and evaluating written information in order to develop ones knowledge and potential, and to participate in, and contribute to, society

9. Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Reading literacy Using, interpreting and reflecting on written material

10. Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Scientific literacy Using scientific knowledge, identifying scientific questions, and drawing evidence-based conclusions to understand and make decisions about the natural world

11. Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Mathematical literacy Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways

12. Mathematical literacy in PISA The real world The mathematical World A real situation A model of reality A mathematical model Mathematical results Real results Understanding, structuring and simplifying the situation Making the problem amenable to mathematical treatment Interpreting the mathematical results Using relevant mathematical tools to solve the problem Validating the results

13. Deciding what to assess... looking back at what students were expected to have learned …or… looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

14. Development of assessments r Frameworks by international experts r Assessment materials submitted by countries developed by research consortium screened for cultural bias –by countries –by expert, international panel –items with prima facie cultural bias removed at this stage internationally validated translations trialled to check items working consistently in all countries r Final tests items shown in trial to be culturally biased removed best items chosen for final tests –balanced to reflect framework –range of difficulties –range of item types (constructed response, multiple choice)

15. Key features of PISA 2003 r Information collected volume of questions –3½ hours of mathematics assessment –1 hour for each of reading, science and problem solving each student –2 hours on paper-and-pencil tasks (subset of all questions) –½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals –questionnaire (school demography, learning environment quality ) r Coverage PISA covers roughly nine tens of the world economy

16. Where we are - and where we can be What PISA shows students can do Examples of the best performing countries

17. PISA provides five key benchmarks for the quality of education systems

18. Average performance of 15-year-olds in mathematics Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance Top-performers Finland remained first in reading and since 2000 moved further in math and science… …and is now on a par with the East Asian countries that were previously unmatched in math and science Also the Netherlands is among the top-performers in math …though not in reading and science. As is the Flemish Community of Belgium Progress Other countries with improvements in at least two assessment areas were Belgium, the Czech Republic and Germany …In Belgium and Germany it was the top performers who drove improvements. Progress Poland raised it’s overall performance in all four assessment areas …thanks to big improvements among lower-performing students in the wake of a major reform in 1999. A widening gap More improvement at the top of the scale has widened the gap between the top and bottom performers in the OECD.

19. Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance Differences in socio-economic background pose major challenges for education systems Students whose parents have better-paid jobs, are better educated or have more “cultural” possessions in their homes tend to perform better… …But the performance advantage varies –Australia, Canada, Finland, Iceland and Japan provide examples showing that it is possible to combine quality and equity –In contrast, results for Belgium, Germany, Hungary and the Slovak Republic reveal large socio-economic inequalities in the distribution of learning opportunities.

20. Is it all innate ability? Variation in student performance OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383. 20

21. Variation of performance between schools Variation of performance within schools Is it all innate ability? Variation in student performance in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383. In some countries, parents can rely on high and consistent standards across schools In Canada, Denmark, Finland, Iceland and Sweden average student performance is high… …and largely unrelated to the individual schools in which students are enrolled. 111 14 12 5

22. Student performance School performance and schools’ socio- economic background - Germany Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES

23. 生徒の成績 学校の成績と学校の社会経済的背景（日本） 有利 社会的背景に関する PISA 指数 不利 生徒の成績と生徒の社会経済的地位（ SES ） 生徒の成績と学校内における生徒の SES 学校の成績と学校の SES 学校の規模

24. Student performance School performance and schools’ socio- economic background - Norway Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES OECD

25. Student performance School performance and schools’ socio- economic background - Finland Advantage PISA Index of social background Disadvantage Figure 4.13 Student performance and student SES Student performance and student SES within schools School performance and school SES School proportional to size

26. Number of school types orprogrammesProportion of 15-year-oldsin programmes leading tovocational education/workFirst age of selection in theeducation systemRepeaters in primaryeducationProportion of repeaters inlower secondary educationProportion of repeaters inupper secondary educationPerformance on themathematics scale – meanscorePerformance in mathematics– standard deviationBetween school varianceRelationship between sesand performance Number of school types or programmes1 Proportion of 15-year-olds in programmes leading to vocational education/work0.501 First age of selection in the education system-0.76-0.521 Repeaters in primary education0.390.27-0.231 Proportion of repeaters in lower secondary education0.22-0.02-0.110.561 Proportion of repeaters in upper secondary education0.450.22-0.530.230.271 Performance on the mathematics scale – mean score-0.090.260.23-0.21-0.17-0.401 Performance in mathematics – standard deviation0.250.19-0.29-0.05-0.060.580.081 Between school variance0.620.63-0.700.150.160.65-0.140.621 Relationship between ses and performance0.510.24-0.530.290.170.43-0.190.480.571 Cross-country correlations with structural features

27. Student performance School performance and schools’ socio- economic background Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Universal policies Increasing educational performance of all children through reforms applied equally across the school system, e.g. –Altering content or pace of curriculum –Improving instructional techniques –Changing the learning environment in schools and classrooms –Standards and accountability –Teacher professional development

28. Student performance School performance and schools’ socio- economic background Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Socio-economically targeted policies Providing a specialised curriculum or additional educational resources to students from disadvantaged backgrounds –Students are often also identified through other risk factors, e.g. immigration, ethnicity, low- income community

29. Student performance School performance and schools’ socio- economic background Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Compensatory policies Providing additional economic resources to students from disadvantaged backgrounds –Different to socio-economically targeted policies, efforts are directed to ameliorating economic circumstances, rather than providing specialised curriculum or additional educational resources

30. Student performance School performance and schools’ socio- economic background Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Performance targeted policies Providing additional economic resources to students based on their academic performance –Early intervention programmes –Remedial and recovery programmes –Performance-based tracking or streaming Countries with flat gradients In combination with SES-targeted policies for countries with steeper gradients

31. Gender differences r In reading, girls are far ahead l In all countries, girls significantly outperform boys in reading r In mathematics, boys tend to be somewhat ahead l In most countries, boys outperform girls …but mostly by modest amounts… …and mainly because boys are overrepresented among top- performers while boys and girls tend to be equally represented in the “at risk” group –Within classrooms and schools, the gender gap is often larger l Strong problem-solving performance for girls suggests… …that it is not the cognitive processes underlying mathematics that give boys an advantage… …but the context in which mathematics appears in school l Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap –Girls report much lower intrinsic (though not instrumental) motivation in mathematics, more negative attitudes and much greater anxiety with mathematics… …and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects

32. Interest in and enjoyment of mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.

33. Instrumental motivation in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.2a, p.360 and Figure 3.3a, p.122.

34. Anxiety in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.8, p.374 and Figure 3.8, p.139.

35. Attitudes towards school OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.

36. Low Performance High Mathematics performance Low performance Low social equity High performance Low social equity Low performance High social equity High performance High social equity Strong impact of social background on performance Moderate impact of social background on performance                 Student anxiety in mathematics High degree of anxiety Low degree of anxiety  

37. How can we get there? Levers for policy that emerge from international comparisons

38. OECD framework National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1

39. Money matters but other things do too Mexico Greece Portugal Italy Spain Germany Austria Ireland United States Norway Korea Czech republic Slovak republic Poland Hungary Finland Netherlands Canada Switzerland Iceland Denmark France Sweden Belgium Australia Japan R 2 = 0.28 Cumulative expenditure (US$) Performance in mathematics r Spending per student is positively associated with average student performance… …but not a guarantee for high outcomes Australia, Belgium, Canada, the Czech Republic, Finland, Japan, Korea and the Netherlands do well in terms of “value for money”… …while some of the big spenders perform below-average

40. Sympathy doesn’t raise standards – aspiration does r In the focus countries National research teams report a strong “culture of performance” –Which drives students, parents, teachers and the educational administration to high performance standards r PISA suggests… …that students and schools perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations –Among these aspects, students’ perception of teacher-student relations and classroom disciplinary climate display the strongest relationships

41. Governance of the school system r In the focus countries Decentralised decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities The provision of standards and curricula at national/subnational levels is combined with advanced evaluation systems –That are implemented by professional agencies Process-oriented assessments and/or centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress r Standard setting and equity-related goals Key objectives: –Raise educational aspirations, establish transparency over educational objectives, reference framework for teachers Approaches range from definition of broad educational goals up to formulation of concise performance expectations Some countries go beyond establishing educational standards as mere yardsticks and use performance benchmarks that students at particular age or grade levels should reach Instruments –Minimum standards, targets defining excellence, normative performance benchmarks r Monitoring and equity-related goals Diverging views how evaluation and assessment can and should be used –Some see them primarily as tools to reveal best practices and identify shared problems in order to encourage teachers and schools to improve and develop more supportive and productive learning environments –Others extend their purpose to support contestability of public services or market-mechanisms in the allocation of resources –e.g. by making comparative results of schools publicly available to facilitate parental choice or by having funds following students Differences in type of performance benchmarks being used and reported for the various stakeholders involved, including parents, teachers and schools

42. Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance

43. Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance School with responsibility for deciding which courses are offered High degree of autonomy Low degree of autonomy

44. Public and private schools Private schools perform better Public schools perform better

45. Organisation of instruction r In the focus countries Schools and teachers have explicit strategies and approaches for teaching heterogeneous groups of learners –A high degree of individualised learning processes –Disparities related to socio-economic factors and migration are recognised as major challenges Students are offered a variety of extra-curricular activities Schools offer differentiated support structures for students –E.g. school psychologists or career counsellors Institutional differentiation is introduced, if at all, at later stages –Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes

46. Low Performance High Mathematics performance Low performance Low social equity High performance Low social equity Low performance High social equity High performance High social equity Strong impact of social background on performance Moderate impact of social background on performance Early selection and stratification High degree of stratification Low degree of stratification

47. Support systems and professional teacher development r In the focus countries Effective support systems are located at individual school level or in specialised support institutions Teacher training schemes are selective The training of pre-school personnel is closely integrated with the professional development of teachers Continuing professional development is a constitutive part of the system Special attention is paid to the professional development of school management personnel

48. Teacher support in mathematics Students’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.1a, p.403 and Figure 5.1, p.213.

49. Student-related factors affecting school climate Principals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.2a, p.406 and Figure 5.2, p.216.

50. Teacher-related factors affecting school climate Principals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.4a, p.410 and Figure 5.4, p.220.

51. Pre-school attendance and performance Percentage of students who attended pre-school Difference in performance between those who attended pre-school for more than one year and those with no pre-school 38 score points is the average performance difference associated with one school year

52. Creating a knowledge-rich profession in which schools and teachers have the authority to act, the necessary knowledge to do so wisely, and access to effective support systems The tradition of education systems has been “knowledge poor” The future of education systems needs to be “knowledge rich” National prescription Professional judgement Informed professional judgement, the teacher as a “knowledge worker” Informed prescription Uninformed professional judgement Uninformed prescription, teachers implement curricula

53. OECD countries participating from PISA 2000 OECD countries participating from PISA from 2003 OECD partner countries participating from PISA 2000 OECD partner countries participating from PISA 2003 OECD partner countries participating from PISA 2006 PISA – Participating Countries

54. Further information www.pisa.oecd.org –All national and international publications –The complete micro-level database email: pisa@oecd.org Andreas.Schleicher@OECD.org …and remember: Without data, you are just another person with an opinion

55. Space & shape item Answers: Yes, No, Yes, Yes Process skill: connections Context: educational quasi-realistic problem typical in maths classes not genuine occupational problem Form: complex multiple-choice Source: OECD (2004) Learning for tomorrow’s world: First results from PISA 2003, Figure 2.4a, p.52.

56. Change and relationships item Scores: 1 for n = 140x0.8 = 112 but no further work shown 2 for correct steps/min but not m/min; correct m/min but not km/hr; correct method but error of calculation; correct km/hr but not giving m/sec 3 for correct m/min (89.6) and m/hr (5.4), rounding acceptable. Process skill: score 1=connections score 2=connections score 3=reflection Context: personal Form: open-constructed

57. Quantity item Form: open constructed response Answer: Yes, with adequate explanation Process skill: reflection Context: public Form: short constructed response Answer: 12 600 ZAR (unit not required) Process skill: reproduction Context: public Form: short constructed response Answer: 975 SGD (unit not required) Process skill: reproduction Context: public

58. Uncertainty item Scores: 1 for “No, not reasonable” but explanation lacking detail (e.g. focusing on exact increase in number of robberies without comparison with total) 2 for “No, not reasonable” with argument focusing on only small part of graph shown, ratio or percentage increase, or need for trend data. Process skill: connections Context: personal Form: open- constructed