1. 1 Perspectives on the Achievements of Irish 15-Year-Olds in the OECD PISA Assessment gerry.shiel@erc.ie www.erc.ie/pisa

2. 2 Overview Purposes of PISA Components of PISA 2003 Performance in four achievement domains (mathematics, reading, science, cross- curricular problem solving) School and student variables associated with performance Looking forward

3. 3 Questions that PISA Attempts to Answer Are students well prepared to meet the challenges of the future? Are they able to analyse, reason and communicate their ideas effectively? Do they have the capacity to continue learning throughout life?

4. 4 PISA’s Focus on Literacy PISA adopts a ‘literacy-based’ approach to student assessment. Emphasis is on applying skills and knowledge to solve authentic, real-life problems. PISA does not provide a direct assessment of ‘traditional, school-based learning’ Competencies, such as ICT literacy, or interpersonal relationships, are not assessed, though there is an interest in looking at these.

5. 5 Other Characteristics of PISA Age-based sample (15-year olds) Cross-sectional design (not longitudinal) Cyclical (currently every three years) Monitors trends over time Major and minor assessment domains An emphasis on informing policy development

6. 6 PISA 2003 Participating Countries

7. 7 PISA 2003 Assessment Components Mathematics (major domain), reading, science, cross-curricular problem solving (minor domains) Mathematics subscales: Space & Shape, Change & Relationships, Quantity, Uncertainty Questionnaires for Principals, Teachers and Students

8. 8 How Achievement Outcomes Are Reported Country rankings by mean score Scores at key percentile ranks Percentages of students achieving proficiency levels (maths, reading, problem solving) Between-school / cluster variation in achievement Note: Measurement error, alternative approaches, ‘comparison’ countries

9. 9 Key Outcomes – Mathematics in PISA 2003 Ireland's mean score (503) is not significantly different from the OECD average (500) Significantly lower than Finland (544) and Belgium (529) Not significantly different from Denmark (514) Ireland’s rank in mathematics: 17th of 29th OECD countries (95%CI = 14th to 19th); and 20th of 40 participating countries (95%CI = 16th to 21st)

10. 10

11. 11 Key Outcomes – Mathematics in PISA 2003 In Ireland there are fewer lower achievers.... 16.8% of Irish students scored at or below proficiency Level 1, compared to an OECD country average of 21.4% In Ireland, there are also fewer high achievers... 11.3% of Irish students scored at proficiency Levels 5 and 6 (the two highest Levels), compared to the OECD average of 14.6%, 21.5% in Belgium, and 23.4% in Finland

12. 12 Performance by Mathematics Proficiency Level

13. 13 Key Outcomes – Mathematics Subscales in PISA 2003 Irish students performed best on Uncertainty, with a mean score (517) that is significantly above the OECD average (502) Students in Ireland also scored significantly above the OECD average on Change & Relationships (506 compared to 499)

14. 14 Key Outcomes – Mathematics Subscales in PISA 2003 Performance on the Quantity subscale (502 in Ireland compared to an OECD average of 501) mirrored overall performance in maths Performance on the Shape & Space subscale was weakest (476 in Ireland compared to an OECD average of 496)

15. 15 Key Outcomes – Reading in PISA 2003 Minor domain in 2003 Ireland (516) significantly higher than the OECD country average (494) and Denmark (492) Significantly lower than Finland (544), Korea (534) and Canada (528) Not significantly different from New Zealand (522) or Belgium (507)

16. 16 PISA 2003 Reading Literacy Ireland ranked 6th of 29 OECD countries and 7th of 40 participating countries Fewer students in Ireland were at or below proficiency Level 1 (11.0% compared to an OECD average of 19.1%). More students in Ireland were at Levels 4 and 5 (35.5% compared to 29.6%)

17. 17 PISA 2003 Reading Literacy But... Scores of Irish students at the 90 th and 95 th percentile ranks (622 and 647) were not significantly different from the corresponding OECD country average scores (619 and 646 respectively)

18. 18 Reading: Performance at Key Markers

19. 19 Key Outcomes – Science in PISA 2003 Minor domain in 2003 The mean for Ireland (505) is significantly above the OECD country average (500) Mean scores for Finland (548) and New Zealand (521) significantly higher Mean score for Denmark (475) significantly lower

20. 20 Key Outcomes – Science in PISA 2003 Ireland ranked 13th of 29 OECD countries and 16th of 40 participating countries Performance among lower achievers (those at the 5 th, 10 th and 25 th percentiles) in Ireland is higher than corresponding OECD average scores; Performance among high achievers in Ireland (those at the 90 th and 95 th percentiles) is lower than OECD average scores at these makers;

21. 21 Key Outcomes – Cross- Curricular Problem Solving Minor domain Performance in Ireland does not differ from the OECD country average (499 compared to 500) Ireland ranked 18th of 29 OECD countries and 21st of 40 participating countries Slightly more Irish students scored at or below proficiency Level 1 (49.4%) compared to the OECD average (47.7%) Fewer Irish students (12.3%) scored at Level 3 (the highest proficiency level) compared to the OECD average (18.2%)

22. 22 Compariing Achievement in 2000 and 2003 Relatively short period – full implementation of revised curricula may take longer than 3 years An increase or decline cannot be interpreted as a trend (three years may be too short) Technical issues around linking scales over time (e.g., number of items used for linking; placement of link items in test booklets)

23. 23 Comparing Performance in Mathematics in PISA 2000 and 2003 Comparison for two subscales – Shape and Space or Change and Relationships Irish context: Revised JC mathematics syllabus introduced in 2000, for first examination in 2003 No significant difference in Ireland on either scale

24. 24 Comparing Performance in Mathematics in PISA 2000 and 2003 No significant differences at of 6 key benchmarks (5 th, 10 th, 25 th, 75 th, 90 th and 95 th pecentile ranks) on either scale Increase in mean performance on Shape and Space in 8 countries, and a decline in 2 Increase in mean performance in Change and Relationships in 13 countries, and a decline in one

25. 25 Comparing Performance in Reading in PISA 2000 and 2003 Mean reading achievement in Ireland decreased by 11 points, which is statistically significant Achievement decreased significantly at the upper end of the achievement distribution (scores at 75th, 90th and 95th percentile ranks) No changes in achievement scores were found at the lower end of the distribution Increase in mean reading scores in 3 countries, and a decline in 9

26. 26 Comparing Performance in Science in PISA 2000 and 2003 No difference in mean performance in Ireland, or in performance at any percentile rank Increase in mean performance in 12 countries, including Finland and Belgium Decline in mean performance in 5 countries, including Korea

27. 27 School Variables Associated with Achievement in Ireland School-level socioeconomic status (SES) (a) school socio-economic, social and cultural status; (b) school disadvantaged status; (c) percent in school with JC fee waiver; Disciplinary climate in mathematics classes (a student-level variable aggregated to school level)

28. 28 School Variables Associated with Achievement in Ireland School gender composition School type (secondary, community /comprehensive, vocational) Ratio of computers to students (negative relationship) Instructional time (minutes of instruction per week)

29. 29 Student Variables Associated with Achievement in Ireland Socioeconomic status Home educational resources Gender Grade level Frequency of absence from school Lone-parent status Number of siblings Number of books in the home Access to home educational resources Study of science

30. 30 Additional Student Variables for Mathematics Self-efficacy in mathematics Anxiety about mathematics

31. 31 Looking Ahead Relevance of PISA to curriculum and assessment in Ireland (particularly the Junior Certificate syllabus/examination). Use of modelling to identify school- and student-level variables that explain variation in achievement, while controlling for other variables

32. 32 Looking Ahead An in-depth analysis of Reading, Mathematics and Science, where frameworks, stimulus texts, underlying items and performance scales can be examined in more detail A consideration of broad policy issues (e.g., gender, performance of high and low achievers) as well as issues specific to each domain