1. 1 PISA 2006 Main achievement outcomes and factors associated with performance on science Eemer Eivers, Gerry Shiel & Rachel Cunningham Educational Research Centre

2. 2 Main Topics Background Background Scientific literacy as defined by PISA Scientific literacy as defined by PISA Performance on science (major domain) Performance on science (major domain) Performance on maths and reading (minor domains) Performance on maths and reading (minor domains) Trends in IRL and OECD Trends in IRL and OECD Factors related to performance Factors related to performance Attitudes to and engagement in science Attitudes to and engagement in science Explaining performance Explaining performance

3. 3 Background: What is PISA? An OECD project An OECD project Takes place in 3-year cycles (2000 – 2003 – 2006 – …..) Takes place in 3-year cycles (2000 – 2003 – 2006 – 2009 – 2012 – 2015 …..) Assesses 1 major and 2 minor ‘domains’ Assesses 1 major and 2 minor ‘domains’ –Science, reading & maths Examines ‘literacy’ - applying knowledge, not just reproducing facts studied in a curriculum Examines ‘literacy’ - applying knowledge, not just reproducing facts studied in a curriculum

4. 4 Background: What is PISA? Information collected Information collected –2-hour test –Questionnaires: students, principals & science teachers Who participated in 2006? Who participated in 2006? –All OECD countries, & 27 ‘partner’ countries –Target group: 15-year-olds –400,000 students (probably the world’s largest survey) Ireland: 165 schools and 4,585 students

5. 5

6. 6 Framework Overall Scale (mean 500, SD 100) Items & Topics SubscalesSubscalesSubscales

7. 7 Context Situations involving science / technology Competencies  Identify scientific issues  Explain phenomena scientifically  Use scientific evidence Knowledge of  living systems  physical systems  Earth and space systems  technology systems Knowledge about science Attitudes  Interest in science  Support for scientific enquiry  Responsibility towards resources Science Framework

8. 8 Item 1…. ItemItemItemItemItem…..item 103 Competency Identify Scientific Issues Explain Phenomena Scientifically Identify Scientific Issues Knowledge of / about … of physical systems … about science …of living systems ContextSocialPersonalGlobal

9. 9 Overall Science Performance: Reading the table 1. Countries vs. OECD average Green = sig. above OECD Green = sig. above OECD Amber = not sig. different Red = sig. below OECD 2. Countries vs. Ireland ▲ = sig. above IRL O = not sig. different ▼ = sig. below IRL **only top 40 countries shown

11. 11 Proficiency Levels Convert continuous scale into skill levels Convert continuous scale into skill levels 6 Levels for science 6 Levels for science –PL 6 is the highest –Need to reach PL 2 to show ‘baseline’ scientific competency

12. 12 Science proficiency levels

13. 13 What can they do at PL 6 or PL 1? Level 1 Level 1 –often confuse key features of an investigation, apply incorrect scientific information, and mix personal beliefs with scientific facts in support of a decision. Level 6 Level 6 –consistently identify, explain and apply scientific knowledge of and about science in a variety of situations. –link different information sources and explanations and use evidence from those sources to justify decisions.

14. 14 What are the items like? PISA uses test units PISA uses test units –A group of items around a common ‘stimulus’ (text and/or graphic) Allows more real-world context Allows more real-world context Sample item handout in delegate pack Sample item handout in delegate pack –3 science units, 1 reading and 1 maths –Attitudes items shown for science

15. 15 Example: Identifying Scientific Issues

16. 16 Example: IRL above average

17. 17 Examples: open-ended

18. 18 Coding open-ended responses

19. 19 So far …. How does PISA define science? How does PISA define science? How did we do? How did we do? What does the test look like? What does the test look like?Next How does it compare to JC science? How does it compare to JC science?

20. 20 Linking PISA & Curriculum Test-curriculum comparison a feature of Irish PISA analyses Test-curriculum comparison a feature of Irish PISA analyses –(see: Shiel et al., 2001; Cosgrove et al., 2005) Reasonable correlations between rated ‘familiarity’ of concepts and performance on reading and maths Reasonable correlations between rated ‘familiarity’ of concepts and performance on reading and maths –r ranges from.3 to.5 For science, familiarity:performance link is weaker For science, familiarity:performance link is weaker –r is around.1

21. 21 Matching PISA items to the JC Science Syllabus Location in rJCSS % of PISA items Not on syllabus 15.5* Biology29.1 Chemistry14.6 Physics22.3 General Skills 18.4 * Versus 43% of items in PISA 2000

22. 22 Knowledge of Science How successful is the revised syllabus in providing coverage of the three core areas (Biology, physics and chemistry)? How successful is the revised syllabus in providing coverage of the three core areas (Biology, physics and chemistry)? What are the JC ‘equivalents’ of PISA knowledge areas? What are the JC ‘equivalents’ of PISA knowledge areas? –Living systems / biology –Physical systems / physics & chemistry –Earth & space / ?? parts of geography?

23. 23 Scores on ‘knowledge’ scales Red = significantly higher than OECD

24. 24 Science Competencies Red = significantly higher than OECD

25. 25 Gender differences in IRL

26. 26 Science Performance: Overview IRL just above the OECD average IRL just above the OECD average Strength: identifying scientific issues Strength: identifying scientific issues Gender differences on some subscales Gender differences on some subscales Fewer low-achieving students, and about average % of high achievers Fewer low-achieving students, and about average % of high achievers No ‘weak’ areas nationally, but females weak on physical systems No ‘weak’ areas nationally, but females weak on physical systems The link between familiarity and performance is weaker for science than for reading and maths The link between familiarity and performance is weaker for science than for reading and maths

27. 27 ?