Presentation on theme: "22 April 2009 Benefits and limitations of international comparative research in science and mathematics education AMSTEL – Seminar Pauline Vos Universiteit."— Presentation transcript:
22 April 2009 Benefits and limitations of international comparative research in science and mathematics education AMSTEL – Seminar Pauline Vos Universiteit van Amsterdam AMSTEL Instituut
2 International comparison Use identical instruments in different countries and look at differences between countries. My own international experience: Went to school in NL, D and F worked in NL, Zimbabwe, Mozambique & South Africa Worked at the Dutch research centre for TIMSS- 1999 (University of Twente) PhD on mathematics curriculum in NL using TIMSS-instruments
3 Many variations Bilateral studies Affect in science: ROSE (Relevance of Science Education), 40 countries Teacher competencies: TEDS(M), 17 countries
4 Two abbreviations… PISA = Project for International Student Assessment organised by OECD (Organisation for Economic Cooperation and Development) TIMSS = Trends in International Maths and Science Study previously: Third Internat. Maths & Science Study organised by IEA (Internat. Assoc. for the Evaluation of Educational Achievement)
5 Translate and define Mathematics Basic calculations and further Science physics, chemistry, biology, physical geography, astronomy, ecology, etc Students all students (not just pre-university)
17 Complex studies.. Assessment framework Large number of items Long design process for instruments Large random sample Quality control trend items (retained) coding and re-coding excluding countries when criteria were not met
25 Complex studies… (1) Not just measuring country scores in one discipline, but also: Sub-disciplines (algebra, ecology,..) Sub-competencies (reproduce, explain, communicate,..) Background variables Not just measuring students scores, but also trying to explain: Differences between countries Differences between schools Differences between students
26 Complex studies… (2) PISA: 85 exercises distributed over 13 test booklets PISA-2000 Main topic: Literacy (reading) Additional: science, maths PISA-2003: Main topic: Mathematical Literacy Additional: literacy, science, problem solving PISA-2006 Main topic: Science Additional: literacy, maths
27 Complex studies… (3) TIMSS: 300 exercises distributed over 12 booklets TIMSS-95: grades 3, 4, grades 7, 8 Performance Assessment (= test in a lab-like environment) grade 12: math/science literacy; advanced maths/science video study (USA, Jap, Germ.) TIMSS-99: grade 8 TIMSS-R Video study (USA, Jap, NL, Cz, Switz, Austr, HongK.) TIMSS-2003 grade 4, grade 8 TIMSS-2007 grade 4, grade 8 TIMSS-2008 grade 12
28 Percentage Distribution Math Items PISA - TIMSS 15-year-olds PISA Grade 8 TIMSS Number 22%33% Measurement 18%14% Geometry 12%18% Data 40%11% Algebra 11%23% Classified to multiple strands 2%0%
29 Math Item Formats Multiple Choice Short Constructed Response Extended Constructed Response TIMSS71%16%13% PISA*33%42%25% * PISA definitions: Multiple Choice includes regular and complex multiple-choice items. Short Constructed Response includes short answer and closed response items. Extended Constructed Response includes open response items.
35 Horse race - thinking both in PISA and TIMSS Ranking is simplistic Sample: intrinsically inaccurate Scores of some countries are very close. With s.e.>3 are small differences not significant NL position in 2003: math: on shared positions 2-7 in PISA and shared positions 6-9 in TIMSS science: on shared positions 4-8 in PISA and shared positions 7-11 in TIMSS
36 Differences PISA and TIMSS Population: PISA: age cohort: 15-year olds (indep. of grade) TIMSS: grade (indep. of age) Countries: PISA: economically developed (OECD) TIMSS: whole world (incl. Saudi Arabia, Botswana) Cyclus: PISA: 3-yearly with different thematic foci TIMSS: 4-yearly with different populations Content: PISA: applicable knowledge (situations): reading, math. literacy, scientific literacy TIMSS: school knowledge (US curriculum): mathematics, science Justification of the study: PISA: knowledge has economical importance TIMSS: knowledge about education is needed
39 TIMSS-1999 Video Study Grade 8 seven countries: Australia, Hong Kong, Japan, NL, Czech Rep, USA, Switzerland Aim: describe patterns in lessons question: what is typical for the math lessons in a certain country? in each country 85 random lesson filmed, transcribed & coded Points of analysis: Structure of the lesson: e.g. who speaks? How many exercises? Mathematical content: complexity, procedures Char of exercise: symbols, contexts
40 TIMSS-Video study: use of materials: Percentage lessons in which a calculator is used
41 TIMSS-Video study: structure of lessons Percentage lessons in which a goal is stated
42 TIMSS-Video study: structure of lessons Percentage lessons, in which a summary statement is made
43 TIMSS-Video study: exercises Percentage exercises per lesson with a context of symbols only
44 TIMSS-1999 Video Study USA video USA Grade 8 (14 yrs) Heterogeneous non-problem segment: the teacher shows the class examples of angles formed by secants and tangents intersecting inside, on, or outside a circle.
45 TIMSS-1999 Video Study Netherlands video NL Grade 8 (14 yrs) Homogeneous (pre-university) non-problem segment: the teacher briefly presents historical background related to the Pythagorean theorem (i.e., Pythagoras was a Greek gentleman). She also connects ideas learned previously (i.e., the definitions of various types of triangles) to the present lesson.
46 TIMSS-1999 Video Study Hong Kong: grd 8, heterogeneous Explanation of a new concept. video HongKong Pre-knowledge: students can solve linear equations What preceded the scene: Solving 2x+10=2(x+5) 2x+10=2x+10 0 = 0 Then different values were insterted for x. left hand side right hand side expanded form factorized form
47 Next in this theatre… TEDS(M) report in December 2009 TIMSS-2008 Advanced report in December 2009 Pisa 2009 report in 2010
48 References for PISA and TIMSS www.pisa.oecd.org http://timss.bc.edu
49 remains the question…. what are the benefits and limitations?