Presentation on theme: "Potential impact of PISA"— Presentation transcript:
1 Potential impact of PISA Organisation for Economic Cooperation and Development (OECD)Potential impact of PISAMichael DavidsonOECD/Directorate for Education
2 A three-yearly assessment that… What is PISA?A three-yearly assessment that…… examines the performance of 15-year-olds in key subject areas as well as a wider range of educational outcomesIncluding students attitudes to learning, their beliefs about themselves, and their learning strategies… collects contextual data from students schools, parents and systems to identify policy leversContextual data explain 71% of the performance variation among schools internationallyCoverageRepresentative samples of between 3,500 and 50, year-old students drawn in each countryMost federal countries also draw regional samples
3 Results from PISA 2006Scientific knowledge and use of that knowledge in…… identifying scientific issues,… explaining scientific phenomena,… using scientific evidence
4 PISA provides key benchmarks for the quality of education systems 1. Overall performance of education systems2. Equity in the distribution of learning opportunitiesMeasured by the impact students’ and schools’ socio-economic background has on performance…… not merely by the distribution of learning outcomes3. Consistency of performance standards across schools4. Gender differences
5 Mean science scores The centre line is the mean The boundaries indicate the 95% confidence limitsThere is only a significant difference if there is no overlap of scoresOECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Table 2.1c
6 Is it all innate ability? Variation in student performance Performance variation across schools20Consistent standards across schools is a gaol for many education systems. PISA allows for an analysis of the variation of student performance across schools.OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 4.1a
7 Variation in student performance The variation in country performance between the high performers and the low performers can be seen clearly in this chart. For example, the United States, New Zealand and the United Kingdom have a much higher degree of variation than Turkey or Mexico. However, what is even more interesting is if we take the same chart and separate the variation into two different types – that which occurs within schools and that which occurs between schools …PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.
8 Variation in student performance Variation of performance within schoolsVariation of performance between schoolsBetween-school variation is an indication of how much a student’s performance depends on the school that he or she attends. It can be seen in the diagram that in countries such as Finland, Iceland and Norway that the between-school variation is relatively small, meaning that standards across schools in these countries are very consistent, whereas in the countries at the other end of the chart there are large differences between schools. This can be a result of the design of the education system where there may be different types of schools that students attend (as in Germany).PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.
9 High science performance Average performance of 15-year-olds in science – extrapolate and applyEquityIn PISA 2006 we also looked at students’ attitudes towards science. Why is this important? Competing successfully in a globalised world increasingly depends on countries’ ability to innovate. This in turn will require major investments in scientific infrastructure and the ability to attract qualified individuals into science-related professions. Governments have to secure broad public support for scientific endeavour. Science and technology have enabled remarkable achievement over the past 100 years, but addressing these challenges successfully will require countries to make major investments in scientific infrastructure and to attract qualified individuals into science-related professions, as well as to secure broad public support for scientific endeavour and the capacity of all citizens to use science in relation to their lives. Peoples’ attitudes to science thus play a key role. In general, 15-year-olds report a fairly strong level of appreciation of science, and they reported an above average level of awareness of environmental issues.… 18 countries perform below this lineLow science performance
11 Impact of selected student and school factors on school performance (after accounting for all other factors in the model)Schools with greater autonomy (resources)School activities to promote science learningOne additional hour of self-study or homeworkOne additional hour of science learning at schoolSchool results posted publiclyAcademically selective schools but no system-wide effectSchools practising ability grouping for all subjects20The combined effect of a number of different factors including autonomy are summarised in this chart. Values which have a negative association with student performance are displayed below the zero line and those that have a positive association with performance are displayed above the line. PISA tells us that the most important factors for success they are not the ones most closely associated with finite material resources, such as the distribution of good teachers. Rather, what matters is how schools and the school system are run – for example, the amount of time that students spend in class and the extent to which schools are accountable for their results. Delivering such advantages to one student is not obviously at the expense of another. This, in itself, is an important conclusion from PISA. It underlines once more that quality, equity and coherence in educational standards are indeed achievable policy goals.USE TEXT ON CHARTEffect after accounting for the socio-economic background of students, schools and countriesOECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a
13 External evaluation of the policy impact of PISA ObjectiveTo what degree does PISA impact policies, meet expectations and priorities of the participants?Evaluation criteriaRelevanceEffectivenessSustainabilityUnexpected / Unplanned impacts
14 External evaluation of the policy impact of PISA Team of three external evaluatorsDavid Hopkins, Dianne Pennock, Jo RitzenTwo-step procedureQuestionnaire to stakeholders in the countryIn-depth evaluation of 5 case studiesStakeholder groupsPolicy makers at the national/federal and local level as well as members of the PISA Governing BoardSchool principals, representatives of teacher and parent organisationsAcademics and researchersRepresentatives of the business community and the mediaEvaluation reportConclusions and recommendations based on the empirical results obtained
15 Strategy for an evaluation of the policy impact of PISA Effectiveness – Specific questionsTo what extent are the participating countries and other stakeholders aware of the output results of PISA?To what extent are these output results being used as input into the policy-making processes of clients and other stakeholders?How significant is the influence of the PISA output results used in the policy-making processes?What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data?To what extent has PISA provided added value to the national assessment strategies?What impact has PISA had on policy coherence in participating countries?Effectiveness in terms of visibility, use and impact – both with regards to PISA itself, and in its relation to relevant national assessments:To what extent are the participating countries and other stakeholders aware of the output results of PISA? This will include an analysis of the dissemination strategy for PISA 2006 and of the PISA reports both at international and national levels.To what extent are these output results being used as input into the policy-making processes of clients and other stakeholders? This will include a review of the involvement of various national stakeholders in PISA, as well as a more general analysis of the reasons for countries to participate in PISA.How significant is the influence of the PISA output results used in the policy-making processes? This will include a review of the delivery chain at national and international levels from the PISA data through their utilisation for policy-making at national levels up to the impact of policy decisions.Specifically: To what extent has PISA impacted on school level practices and student learning?What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data?To what extent has PISA provided added value to the national assessment strategies?What impact has PISA had on policy coherence in participating countries?
16 Outcomes – Relevance of PISA Policy makers are considered the most significant stakeholder group in relation to PISA and its results.Policy makers are also considered as the major group responsible for implementing policies in light of PISA.The various stakeholder groups assume relatively low levels of responsibility for the PISA results in their countries.
17 Outcomes – Relevance of PISA Overall, PISA is seen as a useful tool for identifying and addressing nationally relevant themes and problemsIn many countries PISA is used to monitor and evaluate the quality and equity of the education systemImpact of PISA varies from country to country:In countries with unsatisfactory outcomes often direct policy impactIn high-performing systems: PISA as an evaluation mechanism
19 Outcomes – Effectiveness and sustainability of PISA In all countries, PISA is seen as an important instrument for policy making with a high level of credibilityImpact of PISA on policy formation increasing from PISA 2000 to PISA 2006Influence of PISA greater at the national / federal level than at the local level or in school practices and classroom instructionPolicy makers, academics and researchers and the media most aware of PISA
23 Outcomes – Effectiveness and sustainability of PISA According to respondents, the influence of PISA could be increased by:a better coordinated and strategic approach for the dissemination of PISA results;further support for various stakeholder groups in interpreting PISA results and in designing strategies for improvement in light of PISA;In some countries, clearer linkage of PISA results to national or federal assessment strategies; and,greater utilisation of the PISA results by participating countries.
24 Unexpected / unplanned outcomes Great public interest and debatePISA as an instrument to justify reform“Culture of blame”Increased interest in empirical educational research