1. 22 April 2009 Benefits and limitations of international comparative research in science and mathematics education AMSTEL – Seminar Pauline Vos Universiteit van Amsterdam AMSTEL Instituut

2. 2 International comparison Use identical instruments in different countries and look at differences between countries. My own international experience: Went to school in NL, D and F worked in NL, Zimbabwe, Mozambique & South Africa Worked at the Dutch research centre for TIMSS- 1999 (University of Twente) PhD on mathematics curriculum in NL using TIMSS-instruments

3. 3 Many variations Bilateral studies Affect in science: ROSE (Relevance of Science Education), 40 countries Teacher competencies: TEDS(M), 17 countries

4. 4 Two abbreviations… PISA = Project for International Student Assessment organised by OECD (Organisation for Economic Cooperation and Development) TIMSS = Trends in International Maths and Science Study previously: Third Internat. Maths & Science Study organised by IEA (Internat. Assoc. for the Evaluation of Educational Achievement)

5. 5 Translate and define Mathematics Basic calculations and further Science physics, chemistry, biology, physical geography, astronomy, ecology, etc Students all students (not just pre-university)

6. 6 NLUSAFlaJapMarocRuss. 384938471637

7. 7 NLUSAFlaJapMarocRuss. 454446311638

8. 8 NLUSAFlaJapMarocRuss. 8276 773374

9. 9 NLUSAFlaJapMarocRuss. 443666803155

10. 10 NL36 USA19 Fla21 Jap44 Maroc 2 Russ. 9

11. 11 NLUSAFlaJapMarocRuss. 817471784576

12. 12 Some history… 1964 A comparison of 12 countries, Husén & Bloom 1971 FISS 1980-82 SIMS 1984 SISS 1995 TIMSS (later known as TIMSS-1995 ) 1999 TIMSS-Repeat (later known as TIMSS-1999) 2003 TIMSS-2003 2007TIMSS-2007 2008 TIMSS Advanced TIMSS-2011 2000PISA-2000 2003PISA-2003 2006PISA-2006 PISA-2009

13. 22 April 2009 To what extent do countries differ in the education of science and mathematics? How can you do research on the question:

14. 14 Hours of instruction for science (grd 4)

15. 15 Teacher characteristics math (grd 8)

16. 22 April 2009 Both studies focus on a test

17. 17 Complex studies.. Assessment framework Large number of items Long design process for instruments Large random sample Quality control trend items (retained) coding and re-coding excluding countries when criteria were not met

18. 18 Pisa framework for science

19. 19 Pisa framework for science competencies

20. 20 Pisa framework for science

21. 21 Pisa framework for science

22. 22 Pisa science item Acid Rain

23. 23 Pisa science item Acid Rain

24. 24 Pisa science item Acid Rain

25. 25 Complex studies… (1) Not just measuring country scores in one discipline, but also: Sub-disciplines (algebra, ecology,..) Sub-competencies (reproduce, explain, communicate,..) Background variables Not just measuring students scores, but also trying to explain: Differences between countries Differences between schools Differences between students

26. 26 Complex studies… (2) PISA: 85 exercises distributed over 13 test booklets PISA-2000 Main topic: Literacy (reading) Additional: science, maths PISA-2003: Main topic: Mathematical Literacy Additional: literacy, science, problem solving PISA-2006 Main topic: Science Additional: literacy, maths

27. 27 Complex studies… (3) TIMSS: 300 exercises distributed over 12 booklets TIMSS-95: grades 3, 4, grades 7, 8 Performance Assessment (= test in a lab-like environment) grade 12: math/science literacy; advanced maths/science video study (USA, Jap, Germ.) TIMSS-99: grade 8 TIMSS-R Video study (USA, Jap, NL, Cz, Switz, Austr, HongK.) TIMSS-2003 grade 4, grade 8 TIMSS-2007 grade 4, grade 8 TIMSS-2008 grade 12

28. 28 Percentage Distribution Math Items PISA - TIMSS 15-year-olds PISA Grade 8 TIMSS Number 22%33% Measurement 18%14% Geometry 12%18% Data 40%11% Algebra 11%23% Classified to multiple strands 2%0%

29. 29 Math Item Formats Multiple Choice Short Constructed Response Extended Constructed Response TIMSS71%16%13% PISA*33%42%25% * PISA definitions: Multiple Choice includes regular and complex multiple-choice items. Short Constructed Response includes short answer and closed response items. Extended Constructed Response includes open response items.

30. 30 Mathematical Complexity LowModerateHigh TIMSS51%46%3% PISA29%64%7%

31. 31 PISA-2006 science country ranking

32. 32 PISA-2006 science country ranking

33. 33 TIMSS- 2007 country ranking Maths grade 4

34. 34 TIMSS-2007 multi-country comparison

35. 35 Horse race - thinking both in PISA and TIMSS Ranking is simplistic Sample: intrinsically inaccurate Scores of some countries are very close. With s.e.>3 are small differences not significant NL position in 2003: math: on shared positions 2-7 in PISA and shared positions 6-9 in TIMSS science: on shared positions 4-8 in PISA and shared positions 7-11 in TIMSS

36. 36 Differences PISA and TIMSS Population: PISA: age cohort: 15-year olds (indep. of grade) TIMSS: grade (indep. of age) Countries: PISA: economically developed (OECD) TIMSS: whole world (incl. Saudi Arabia, Botswana) Cyclus: PISA: 3-yearly with different thematic foci TIMSS: 4-yearly with different populations Content: PISA: applicable knowledge (situations): reading, math. literacy, scientific literacy TIMSS: school knowledge (US curriculum): mathematics, science Justification of the study: PISA: knowledge has economical importance TIMSS: knowledge about education is needed

37. 37 Country score and GDP

38. 38

39. 39 TIMSS-1999 Video Study Grade 8 seven countries: Australia, Hong Kong, Japan, NL, Czech Rep, USA, Switzerland Aim: describe patterns in lessons question: what is typical for the math lessons in a certain country? in each country 85 random lesson filmed, transcribed & coded Points of analysis: Structure of the lesson: e.g. who speaks? How many exercises? Mathematical content: complexity, procedures Char of exercise: symbols, contexts

40. 40 TIMSS-Video study: use of materials: Percentage lessons in which a calculator is used

41. 41 TIMSS-Video study: structure of lessons Percentage lessons in which a goal is stated

42. 42 TIMSS-Video study: structure of lessons Percentage lessons, in which a summary statement is made

43. 43 TIMSS-Video study: exercises Percentage exercises per lesson with a context of symbols only

44. 44 TIMSS-1999 Video Study USA video USA Grade 8 (14 yrs) Heterogeneous non-problem segment: the teacher shows the class examples of angles formed by secants and tangents intersecting inside, on, or outside a circle.

45. 45 TIMSS-1999 Video Study Netherlands video NL Grade 8 (14 yrs) Homogeneous (pre-university) non-problem segment: the teacher briefly presents historical background related to the Pythagorean theorem (i.e., Pythagoras was a Greek gentleman). She also connects ideas learned previously (i.e., the definitions of various types of triangles) to the present lesson.

46. 46 TIMSS-1999 Video Study Hong Kong: grd 8, heterogeneous Explanation of a new concept. video HongKong Pre-knowledge: students can solve linear equations What preceded the scene: Solving 2x+10=2(x+5) 2x+10=2x+10 0 = 0 Then different values were insterted for x. left hand side right hand side expanded form factorized form

47. 47 Next in this theatre… TEDS(M) report in December 2009 TIMSS-2008 Advanced report in December 2009 Pisa 2009 report in 2010

48. 48 References for PISA and TIMSS www.pisa.oecd.org http://timss.bc.edu

49. 49 remains the question…. what are the benefits and limitations?