Presentation on theme: "PISA Mathematics Assessment APEC Tokyo Feb 2010 Kaye Stacey University of Melbourne, Australia Data and images in this presentation are from OECD websites."— Presentation transcript:
PISA Mathematics Assessment APEC Tokyo Feb 2010 Kaye Stacey University of Melbourne, Australia Data and images in this presentation are from OECD websites and official publications and from ACER publications on PISA in Australia. The views expressed here are those of the author and do not represent the OECD or associates.
Outline What is PISA and what does it test? What is mathematical literacy? A small sample of results –Country comparisons –Levels of proficiency –Performance of subgroups –Social gradient Possibilities for Computer-based Assessment of Mathematics
PISA: Programme for International Student Assessment Test years 2000, 2003, 2006, 2009, 2012,.. 15 year olds assesses the knowledge and skills that students have acquired at school and their ability to use them in everyday tasks and challenges –reading literacy –scientific literacy –mathematical literacy statistically rigorous, to ensure that the results are as meaningful as possible, measuring –student performance –data on the student, family, school and system factors
Key features of PISA (from OECD) policy orientation –major aim is informing educational policy and practice –aim to significantly improve understanding of the outcomes of education concept of literacy (discussed later) relevance to lifelong learning –motivation to learn, –attitudes towards learning –learning strategies; surveys to explore features associated with educational success –characteristics of students and schools –trends monitored every 3 years; breadth –by 2006, around 90% of the world economy –nearly students Recent studies tracking young people in the years after age 15 show PISA measures knowledge and skills relevant to a life success
Participating countries PISA 2000: 43 PISA 2003: 41 PISA 2006: 57 –nearly students PISA 2009: 66 –PISA Plus : +9 PISA 2012: about 90?
Asia-Pacific 2009 China –Hong Kong –Macao –Shanghai Indonesia New Zealand Thailand Japan Korea Australia Singapore Chinese Taipei Chile Peru Panama Argentina Mexico USA Canada +9 PISA Plus –Malaysia, …
Survey methods Schools randomly selected by PISA (usually 150+) Random sample of 35+ students per school –between age 15 yrs 3 mths & 16 yrs 2 mths Strict sampling criteria to be included in reports –e.g. Netherlands in 2000 below required so not in trend data Some countries oversample for their own purposes Each student does 2 hour test and 30 min questionnaire Items are in rotating booklets (about 13) –results of individual students not available/meaningful
TIMSS: Trends in International Mathematics and Science Study Independent body, not OECD Tests every 4 years, since 1994/5 Grade based sample (years 4, 8, 12) Tests randomly sampled intact classes –hence teacher survey makes sense Aims to test achievement of curriculum goals –Careful and extensive curriculum comparisons More Asian countries have participated in TIMSS –Singapores high results very famous
Schedule of performance measures Additional cognitive assessments: –2003: problem solving –2006: computer based assessment of science –2009: electronic reading –2012: problem solving –2012: computer based assessment of mathematics Now preparing
Questionnaire components School context and attitudes –themselves and their homes –attitudes to learning School questionnaire, optional teacher questionnaires TRENDS since 2000 (nearly)
Country rankings are always of interest – statistics make comparison complicated
Statistically better than Australia - Maths PISA 2003 –Hong-Kong-China, Finland, Korea, Netherlands, Liechtenstein, Japan, Canada PISA 2006 –Chinese Taipei, Finland, Hong-Kong-China, Korea, Netherlands, Switzerland, Canada, Macao-China Movements: 5 stay above Australia, 2 drop to Australias group, 1 rises from Australias group, 2 new entrants
What percent of students in top performance bands?
OECD: PISA Proficiency Level 2 a baseline level of proficiency at which students begin to demonstrate skills that enable them to actively use mathematics What percent of students in lowest performance bands?
What is mathematical literacy?
PISA assesses the knowledge and skills that students have acquired at school and their ability to use them in everyday tasks and challenges Reflect recognition that globalisation and computerisation are changing labour markets and societies, and that a different set of skills is needed US evidence: –greatest decline in jobs over the past decade has not been in manual labour, but in routine cognitive tasks – those that can easily be done at less cost by computer (Levy & Murnane, 2006).
Mathematical literacy 2003/2006 "an individuals capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgments, and to engage in mathematics in ways that meet the needs of that individuals current and future life as a constructive, concerned and reflective citizen." Strong links to other concepts –Mathematical modelling (in PISA framework) –Numeracy (but certainly not just basic skills) –Quantitative literacy
Sample domain items PISA: Take the test –Reading – page 13 –Science – page 187 –Mathematics – page 97 Questionnaires –Also download from MyPISA
% teachers unaware
Reading Written text followed by questions This one: answer as graph
A test across countries needs… cultural breadth and balance in tests –bullying, Ministry of Education, …. –not a question of intersection of school curricula around the world (TIMSS) –school curricula (e.g. reading graphs) influence success rate and hence usefulness of items for constructing a measure high quality in translations –two masters for each item (English and French) –translations from both masters compared –back translation etc –some informal language will not translate corner vs vertex English: corner or vertex French: vertex
Growing Up 6.1 A height of female in 1980 (given increase since then) 6.2 Explain how graph shows growth rate of girls slows down after 12 yrs of age 6.3 When are females taller than males of same age?
Growing Up (6.2 – growth rate girls) Classification –Scientific; Change and Relationships; Connections –Difficulty 574 PISA score points. The question requires students to: –Analyse different growth curves –Evaluate characteristics of data set, represented by graph. –Note and interpret different slopes along graphs. –Reason and communicate the results of this process, within explicit models of growth. OECD average 45% Most successful countries: Netherlands (77%), Finland (68%), Belgium (64%), Canada (64%) Large omission rates: Austria (44%) and Greece (43%).
Scoring for 6.2 (growth rate for girls) Score (Code) 1 : Response refers to change of gradient of female graph, explicitly or implicitly. Code 11:Refers to reduced steepness, using daily-life language. –It does no longer go up, it straightens out; The curve levels off; It is more flat after 12; The line of the girls starts to even out and the boys line just gets bigger; It straightens out and the boys graph keeps rising. Code 12: Refers to reduced steepness,using mathematical language. –You can see the gradient is less; The rate of change of the graph decreases from 12 years on; (uses words like gradient, slope, or rate of change) Code 13: Compares actual growth (comparison can be implicit). –From 10 to 12 the growth is about 15 cm, but from 12 to 20 the growth is only about 17 cm; The average growth rate from 10 to 12 is about 7.5 cm per year, but about 2 cm per year from 12 to 20 years. Score (Code) 0 Code 01: Student indicates that female height drops below male height, but does NOT mention the steepness of the female graph or a comparison of the female growth rate before and after 12 years. –The female line drops below the male line. Code 02: Other incorrect responses. For example, the response does not refer to the characteristics of the graph, as the question clearly asks about how the GRAPH shows…. –Girls mature early; Girls dont grow much after 12.
Growing Up (6.2 – growth rate girls) Answer type: –Daily life language: over 70% of correct answers in 24 countries –Mathematical language : 56% of correct answers in Korea –Comparing actual growth: common in Austria (34%); Mexico (26%), Greece (23%), France and Turkey (19%). Common errors –Most common error: not referring to graph e.g. girls dont grow much after 12. –Around 40% of incorrect answers in France, Korea and Poland refer to graph, only to show the female height drops below the male height. (concept of gradient??)
Heatbeat (M537) - graph Heartrate = 200 – age Heartrate = 208 – 0.7*age Newspaper statement in text alerts student to phenomenon Question 46.1: from which age does the recommendation increase? Question 46.2: write formula for most effective training heartrate (80% of max) Possible solution to Q46.1
Meaning of 2 out of 3 in next 20 years
How additional data affects average – complex multiple choice item
Interpreting an unusual representation
Bookshelves Classification –Quantity; Occupational; Connections –Difficulty rating: 499 PISA score points (Mean is set to 500) The question requires students to: –Develop a strategy to connect two pieces of information for each component: how many available, how many needed per set –Use logical reasoning to link that analysis across the components to produce the required solution. –Communicate the mathematical answer as a real-world solution (not 5.5 bookshelves) Most successful: –Finland and Hong Kong-China (74%), –Korea, the Czech Republic, Belgium and Denmark (72%). OECD average: 61% correct, 29% of students attempted & incorrect and 10% did not attempt.
2012 CBAM: computer based test of mathematics New opportunities for presentation of items to measure same material better –More attractive presentation –Better presentation (e.g. animation) –Better response formats (e.g. move an animation?) Able to test some aspects of doing mathematics by computer and so extend notion of literacy to better match world –What are these aspects?
Recommendation 1. Raising Visibility and Awareness of the Importance of Mathematical Literacy in the Workplace The focus should be: –The nature of mathematical literacy: that it is anchored in real data, in the context of a particular workplace. –That maths used in the workplace has economic benefits in the market- place. –That mathematics may be present quite implicitly in jobs and tasks, which are not obviously mathematical. –Many employees, regardless of their level of employment, are required to use mathematical literacy. –That IT and mathematical skills are interdependent. Mathematical Skills in the Workplace Final Report to the Science, Technology and Mathematics Council, UK, 2002 C. Hoyles, A. Wolf, S. Molyneux-Hodgson & P. Kent
What IT-maths skills in ML?
Use 3-D views See GoogleSketchUp demo
What IT-maths skills in ML? Plot graph
What IT-maths skills in ML? Enter data? Write formula? What tools? (graphics) calculator? computer – what software? mobile phone?
How well can you….. use a spreadsheet to plot a graph? Do well by myself –OECD total: 42.09% –Macao: 28.49% –Thailand 25.75% –Australia 58.42% MyPISA query
What IT-maths skills in ML? Explore maths Open Geogebra demo
Studying subgroups of students Gender Ethnic and home background –Migration –Language background –Indigenous students Social gradient
Mathematical literacy by socio-economic background (Australia) Graphic shows: mean and confidence interval (white) 5 th, 10 th, 25 th, 75 th, 90 th, 95 th percentiles
Performance against social index (Science 2006) (Note wide spread)
Social gradient (Science 2006) (Sci-literacy score against social index) From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up
Position of line; strength of relationship; gradient of line; curvature; length of line From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up
Interpreting the social gradient Strength of association (variance explained) –Hong Kong 6.9% < Australia 11.3% < OECD 14.5% –Asian countries tend to be low Gradient –Australia 43 > OECD 40 > Hong Kong 26 Length –Australia has less variation in social index than OECD –US has higher top level than OECD; same bottom Position –Australia does better than the OECD average Curvature –low and high groups do not differ on relationship (cf NZ, Canada) Australia – maths slightly less affected by social index than reading and science – relatively more affected by school NOTE: For almost all countries, the effect of school average ESCS outweighs effect of students own ESCS.
Equally prepared for life? OECD Reading PISA2000 –females significantly outscored males in all countries Mathematics PISA2003 –males often outscored females Science PISA2006 –no significant difference between males and females overall, but some difference in patterns of strengths –no significant differences in attitudes to school science –marked differences in expectations of science career
Gender differences in science in schools with different social levels
Australias indigenous girls and boys Graphic shows: mean and confidence interval (white) 5 th, 10 th, 25 th, 75 th, 90 th, 95 th percentiles
Information about PISA https://mypisa.acer.edu.au/ PISA: Take-the-test MyPISA: public database and analysis
MyPISA data request
Thoughts and discussion points Immense amount of information and excellent reports –Available to you through public databases (except secure trend items) –But no study answers all questions in-depth understanding of thinking e.g. of algebra What has caused the results (e.g. Finlands success; Asias success) An international study operates under severe constraints –Create items that work internationally to measure target construct validly –Anomalies can reveal differences e.g. Korea-Australia average, 10 3 vs 10 4 Concepts of mathematical and scientific literacy –Major contribution to educational aspiration in many countries –Still developing How does computer-based mathematics affect definition of math literacy? Has mathematical literacy changed from 2003 to 2012? Aim is to find the school systems, schools, teaching and societies that best prepare all future citizens for living productive and satisfying lives
Thank you https://mypisa.acer.edu.au/ Stacey, K. & Stephens, M. (2008). Performance of Australian School Students in International Studies in Mathematics. Schooling Issues Digest 2008/1. Canberra schooling_issues_digest/. schooling_issues_digest/