1. Potential impact of PISA
Organisation for Economic Cooperation and Development (OECD)
Potential impact of PISA
Michael Davidson
OECD/Directorate for Education

2. A three-yearly assessment that…
What is PISA?
A three-yearly assessment that…
… examines the performance of 15-year-olds in key subject areas as well as a wider range of educational outcomes
Including students attitudes to learning, their beliefs about themselves, and their learning strategies
… collects contextual data from students schools, parents and systems to identify policy levers
Contextual data explain 71% of the performance variation among schools internationally
Coverage
Representative samples of between 3,500 and 50, year-old students drawn in each country
Most federal countries also draw regional samples

3. Results from PISA 2006
Scientific knowledge and use of that knowledge in…
… identifying scientific issues,
… explaining scientific phenomena,
… using scientific evidence

4. PISA provides key benchmarks for the quality of education systems
1. Overall performance of education systems
2. Equity in the distribution of learning opportunities
Measured by the impact students’ and schools’ socio-economic background has on performance…
… not merely by the distribution of learning outcomes
3. Consistency of performance standards across schools
4. Gender differences

5. Mean science scores The centre line is the mean
The boundaries indicate the 95% confidence limits
There is only a significant difference if there is no overlap of scores
OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Table 2.1c

6. Is it all innate ability? Variation in student performance
Performance variation across schools 20
Consistent standards across schools is a gaol for many education systems. PISA allows for an analysis of the variation of student performance across schools.
OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 4.1a

7. Variation in student performance
The variation in country performance between the high performers and the low performers can be seen clearly in this chart. For example, the United States, New Zealand and the United Kingdom have a much higher degree of variation than Turkey or Mexico. However, what is even more interesting is if we take the same chart and separate the variation into two different types – that which occurs within schools and that which occurs between schools …
PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.

8. Variation in student performance
Variation of performance within schools
Variation of performance between schools
Between-school variation is an indication of how much a student’s performance depends on the school that he or she attends. It can be seen in the diagram that in countries such as Finland, Iceland and Norway that the between-school variation is relatively small, meaning that standards across schools in these countries are very consistent, whereas in the countries at the other end of the chart there are large differences between schools. This can be a result of the design of the education system where there may be different types of schools that students attend (as in Germany).
PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.

9. High science performance
Average performance of 15-year-olds in science – extrapolate and apply
Equity
In PISA 2006 we also looked at students’ attitudes towards science. Why is this important? Competing successfully in a globalised world increasingly depends on countries’ ability to innovate. This in turn will require major investments in scientific infrastructure and the ability to attract qualified individuals into science-related professions. Governments have to secure broad public support for scientific endeavour. Science and technology have enabled remarkable achievement over the past 100 years, but addressing these challenges successfully will require countries to make major investments in scientific infrastructure and to attract qualified individuals into science-related professions, as well as to secure broad public support for scientific endeavour and the capacity of all citizens to use science in relation to their lives. Peoples’ attitudes to science thus play a key role. In general, 15-year-olds report a fairly strong level of appreciation of science, and they reported an above average level of awareness of environmental issues.
… 18 countries perform below this line
Low science performance

11. Impact of selected student and school factors on school performance (after accounting for all other factors in the model)
Schools with greater autonomy (resources)
School activities to promote science learning
One additional hour of self-study or homework
One additional hour of science learning at school
School results posted publicly
Academically selective schools but no system-wide effect
Schools practising ability grouping for all subjects 20
The combined effect of a number of different factors including autonomy are summarised in this chart. Values which have a negative association with student performance are displayed below the zero line and those that have a positive association with performance are displayed above the line. PISA tells us that the most important factors for success they are not the ones most closely associated with finite material resources, such as the distribution of good teachers. Rather, what matters is how schools and the school system are run – for example, the amount of time that students spend in class and the extent to which schools are accountable for their results. Delivering such advantages to one student is not obviously at the expense of another. This, in itself, is an important conclusion from PISA. It underlines once more that quality, equity and coherence in educational standards are indeed achievable policy goals.
USE TEXT ON CHART
Effect after accounting for the socio-economic background of students, schools and countries
OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a

12. Evaluating the policy impact of PISA

13. External evaluation of the policy impact of PISA
Objective
To what degree does PISA impact policies, meet expectations and priorities of the participants?
Evaluation criteria
Relevance
Effectiveness
Sustainability
Unexpected / Unplanned impacts

14. External evaluation of the policy impact of PISA
Team of three external evaluators
David Hopkins, Dianne Pennock, Jo Ritzen
Two-step procedure
Questionnaire to stakeholders in the country
In-depth evaluation of 5 case studies
Stakeholder groups
Policy makers at the national/federal and local level as well as members of the PISA Governing Board
School principals, representatives of teacher and parent organisations
Academics and researchers
Representatives of the business community and the media
Evaluation report
Conclusions and recommendations based on the empirical results obtained

15. Strategy for an evaluation of the policy impact of PISA
Effectiveness – Specific questions
To what extent are the participating countries and other stakeholders aware of the output results of PISA?
To what extent are these output results being used as input into the policy-making processes of clients and other stakeholders?
How significant is the influence of the PISA output results used in the policy-making processes?
What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data?
To what extent has PISA provided added value to the national assessment strategies?
What impact has PISA had on policy coherence in participating countries?
Effectiveness in terms of visibility, use and impact – both with regards to PISA itself, and in its relation to relevant national assessments:
To what extent are the participating countries and other stakeholders aware of the output results of PISA? This will include an analysis of the dissemination strategy for PISA 2006 and of the PISA reports both at international and national levels.
To what extent are these output results being used as input into the policy-making processes of clients and other stakeholders? This will include a review of the involvement of various national stakeholders in PISA, as well as a more general analysis of the reasons for countries to participate in PISA.
How significant is the influence of the PISA output results used in the policy-making processes? This will include a review of the delivery chain at national and international levels from the PISA data through their utilisation for policy-making at national levels up to the impact of policy decisions.
Specifically: To what extent has PISA impacted on school level practices and student learning?
What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data?
To what extent has PISA provided added value to the national assessment strategies?
What impact has PISA had on policy coherence in participating countries?

16. Outcomes – Relevance of PISA
Policy makers are considered the most significant stakeholder group in relation to PISA and its results.
Policy makers are also considered as the major group responsible for implementing policies in light of PISA.
The various stakeholder groups assume relatively low levels of responsibility for the PISA results in their countries.

17. Outcomes – Relevance of PISA
Overall, PISA is seen as a useful tool for identifying and addressing nationally relevant themes and problems
In many countries PISA is used to monitor and evaluate the quality and equity of the education system
Impact of PISA varies from country to country:
In countries with unsatisfactory outcomes often direct policy impact
In high-performing systems: PISA as an evaluation mechanism

18. Important aspects of PISA

19. Outcomes – Effectiveness and sustainability of PISA
In all countries, PISA is seen as an important instrument for policy making with a high level of credibility
Impact of PISA on policy formation increasing from PISA 2000 to PISA 2006
Influence of PISA greater at the national / federal level than at the local level or in school practices and classroom instruction
Policy makers, academics and researchers and the media most aware of PISA

20. Addressing policy needs

21. Contributing to change

23. Outcomes – Effectiveness and sustainability of PISA
According to respondents, the influence of PISA could be increased by:
a better coordinated and strategic approach for the dissemination of PISA results;
further support for various stakeholder groups in interpreting PISA results and in designing strategies for improvement in light of PISA;
In some countries, clearer linkage of PISA results to national or federal assessment strategies; and,
greater utilisation of the PISA results by participating countries.

24. Unexpected / unplanned outcomes
Great public interest and debate
PISA as an instrument to justify reform
“Culture of blame”
Increased interest in empirical educational research

25. Next step

26. Thank you very much for your attention. http://www. pisa. oecd
Thank you very much for your attention!