1. Education for Life: PISA 2003 Symposium PISA's Approach to Assessing Knowledge and Skills in the Context of Curriculum and Assessment in Ireland Jude Cosgrove, Educational Research Centre Regency Hotel, April 6, 2005

2. Content of Presentation Brief overview of PISA 2003 frameworks The PISA test-curriculum rating project Purposes Design Results Associations between performance on PISA and on the Junior Certificate Performance standards for PISA reading and mathematics by Junior Certificate syllabus level Summing up: Key issues raised

3. The PISA 2003 Assessment Earlier studies, e.g., TIMSS, took a ‘common curriculum’ approach PISA in contrast focuses on knowledge and skills deemed necessary for future participation in society Position justified on the basis of broader domain coverage authenticity relevance (but note inherent limitations of assessment mode)

4. The PISA 2003 Assessment Assessment framework underpins the three domains Developed by an international panel of subject matter experts in consultation with PISA countries Each framework contains definition (and elaboration) of the subject domain description of key content and processes with illustrative examples description of other characteristics problem-solving framework not discussed here

5. Definition of PISA 2003 Mathematics “…an individual’s capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgements and to engage with mathematics in ways that meet the needs of that individual’s life as a constructive, concerned and reflective citizen.” (OECD, 2003, p. 24) Reporting scales by content area; scales for mathematical processes may be developed in the future (Reproduction, Connections, Reflection)

6. Overarching ideas in PISA 2003 Mathematics Shape & Space: geometry and spatial reasoning Change & Relationships: functions, inequalities Quantity: number sense, arithmetic and measurement Uncertainty: statistics and probability Not the same as, but linked to, curriculum areas Approximately evenly represented (85 items)

7. Definition of PISA 2003 Reading Literacy “…understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society.” (OECD, 2003, p. 15) Retrieving information – locating one or more pieces of information in a text Developing interpretation – constructing meaning and drawing inferences using information from the text Reflecting on content and form – relating a text to one’s knowledge and ideas to provide a critical evaluation. Reported as three separate subscales in 2000, only one combined reading scale in 2003. Subscales by text format were also developed in 2000.

8. Definition of PISA 2003 Science “…the capacity to use scientific knowledge, to identify questions and to draw evidence-based conclusions, in order to understand and help make decisions about the natural world and the changes made to it through human activity.” (OECD, 2003, p. 133) Science in life and health Science in Earth and in the environment Science in technology about one-third of items in each topic area low emphasis on physical sciences influenced by STS movement no science subscales in 2000 or 2003

9. Issues in Interpretation While giving breadth and relevance to the assessment, the PISA approach can make results difficult to interpret vis a vis national curricula (focus, priorities, development) Purpose of Irish test-curriculum rating project is to provide a broad description of overlap and divergence between PISA and the Junior Certificate Junior Certificate rather than Leaving Certificate selected since 95% of Irish students have completed/almost completed JCE, while only 15% or so have just one year of LCE completed Sample design not conducive to exact linkages between curriculum/assessment and achievements on PISA

10. PISA Test-Curriculum Rating Projects (2000 and 2003) Limitations Is a measure of intended curriculum Provides only a broad measure Is a one-way mapping – classifies PISA items in terms of Junior Certificate content but not vice versa Would be usefully supplemented with qualitative comparison of PISA framework/items with Junior Certificate syllabus/guidelines/examination papers Other countries have expressed interest (Belgium, Denmark, Germany, Italy, Poland, Portugal, Switzerland, UK, Uruguay, USA) but no international comparative framework

11. PISA Test-Curriculum Rating Projects (2000 and 2003) Framework Ratings attempt to measure the extent and type of similarities and differences between PISA test items in reading, mathematics and science, and the types of questions students in Third year of the Junior Cycle are exposed to, based on an examination of the intended curriculum at each level of the syllabus Intended curriculum = instruction and learning goals in mathematics as defined at the system level (cf. TIMSS)

12. PISA Test-Curriculum Rating Projects (2000 and 2003) Framework Each test item is rated with regard to each syllabus level A 3-point rating scale is used (not/somewhat/very familiar) Ratings for mathematics: concept, context of application, item format Ratings for reading: process, context of application, item format Ratings for science: concept, process, context of application, item format Each item thus receives 9 or 12 ratings These can be examined at both the item level and the student level

13. PISA Test-Curriculum Rating Projects (2000 and 2003) Method Three individuals with extensive experience in their subject area in Ireland (ranging from curriculum development and examination setting to teaching) carried out the curriculum rating exercise for each domain Initially each item was rated independently This was followed by a meeting of the raters to obtain consensus on items on which there was ‘disagreement’ 1, 1, 1; 2, 2, 3 : OK – choose modal rating 1, 2, 3; 1, 3, 3: Disagreement – needs discussion

14. PISA Test-Curriculum Rating Projects (2000 and 2003) Results Reading and Science: from 2000 few or no new items no change in curriculum minor domains science to be examined in detail in 2006 Mathematics: from 2003 in addition to familiarity ratings, mathematics items were classified by Junior Certificate syllabus topic area

15. READING: PISA 2000

16. SCIENCE: PISA 2000

17. MATHEMATICS: PISA 2003

18. Mathematics: JC Curriculum Area

19. Associations Between Curriculum Familiarity and Performance Student ‘familiarity’ scores were computed for each scale depending on Syllabus level taken The combination of items attempted NB interdependence of scales In the case of reading in particular, possible confounding with difficulty level of text In the case of science and reading in particular, students may learn concepts/processes in other contexts

20. Associations Between Curriculum Familiarity and Performance Reading (2000): Process =.55 Context =.54 Format =.46 Science (2000): Concept =.19 Process =.05 Context = -.01 Format =.06 Mathematics (2003): Concept =.37 Context =.21 Format =.28

21. Associations Between Performance on the Junior Cert. and on PISA 2003 Students taking the JCE in 2002 or 2003 had their results matched to the PISA 2003 dataset (about 94% of the cohort) Students’ Junior Certificate achievements were mapped onto a 12-point JCPS (9-point in the case of science) NB the scale is not of equal intervals – in particular, there is evidence of clustering at the lower end

22. JC Performance Scale

23. Associations Between Performance on the Junior Cert. and on PISA 2003 Results PISA maths/JCE maths =.75 PISA reading/JCE English =.67 PISA science/JCE science =.67 Indicates an appreciable degree of overlap in achievements, despite disparities in test content and format BUT correlations between achievements on the PISA domains themselves are higher (.80 to.85)

24. Performance Standards by Syllabus Level According to the OECD (2004), students scoring at or below Level 1 on the mathematics and reading proficiency scales have only very basic, or inadequate, literacy skills and may be at risk in their future occupational, educational and personal lives. NB PISA is not designed to measure the literacy skills of students below Level 1 with any confidence.

25. Performance Standards by Syllabus Level: Reading

26. Performance Standards by Syllabus Level: Mathematics

27. Summing Up PISA’s real-life literacy approach raises issues for interpreting achievements, particularly relating results to curriculum and instruction. The test-curriculum rating project, unique to Ireland, was developed to facilitate interpretation and stimulate discussion around such issues.

28. Summing Up Ratings for reading and science were reviewed for 2000, and for 2003, mathematics ratings were reviewed; an in-depth analysis for science in 2006 is planned. In interpreting outcomes, it might be borne in mind that students may have learned skills/concepts outside the classroom or in other subject areas (cf. the Uncertainty conundrum). Other limitations… In all three domains, divergence between PISA and the Junior Certificate syllabus/examinations was noted; areas of overlap are also in evidence. It appears that PISA reading taps many of the processes associated with Junior Certificate English.

29. Summing Up The particularly strong performance of Irish students on the Reflect subscale in PISA 2000, supports this observation (critical evaluation and audience awareness are often required in both JCE and Reflect tasks). The contexts in which science and mathematics concepts are assessed in PISA are largely not familiar. The low level of familiarity with context in mathematics and science suggests that a more in- depth examination of how PISA embeds problems in context is merited.

30. Summing Up Support for the applied nature of PISA mathematics comes from the finding that concepts appear to be located mainly in Junior Certificate mathematics topic areas of applied arithmetic & measure, and to a lesser degree, in statistics and number systems. Up to half of the concepts underlying PISA mathematics and science items are not taught for JCE. The relevance of such concepts for students in Ireland should be considered. Analysis of content assessed for JCE but not on PISA also merit consideration with the issue of relevance in mind…

31. Summing Up The lack of PISA items located in JC maths topic areas of geometry, trigonometry, functions, and algebra is noteworthy and suggests that items on the Space & Shape, and Change & Relationships scales in particular that are not taught for the Junior Certificate merit further examination. Relating expected familiarity of various aspects of the items to student performance, the relationship is appreciable in the case of PISA reading (although possible confounding with text was noted). For mathematics and science, concept familiarity is most strongly predictive of achievement (although the association is weak in the case of science). Refinements to the methodology of the test- curriculum rating project may be required for 2006.

32. Summing Up The divergence between what PISA and the Junior Certificate assess is further supported by the observation that, although associations between PISA and Junior Certificate are substantial, achievements amongst the PISA domains are more closely associated with one another. A cross-tabulation of PISA proficiency levels and Junior Certificate syllabus levels in the areas of English/reading and mathematics suggest a further area of divergence between the two assessments (although the purposes of the two assessments of course are very different; moreover, the Junior Certificate is not intended to be a terminal examination).

33. Summing Up Nonetheless, the relatively high proportions of students at Ordinary and Foundation levels at or below the lowest proficiency levels in both English/reading and mathematics suggests that many students at the end of lower secondary education lack basic literacy skills in reading and mathematics. Further, the decline in the percentages of Higher level students achieving Level 5 on the PISA reading scale from 2000 to 2003 suggests that performance standards at Higher level in particular merit careful monitoring.