1. Assessing assessment: the role of student effort in comparative studies Ray Adams Jayne Butler

2. 2 Participation 2000 –OECD 28, non-OECD 4 –PISA+, 11 non-OECD 2003 –OECD 30, non-OECD 12 2006 –OECD 30, non-OECD 28

3. 3 Content Coverage of domains Reading literacy Mathematical literacy Scientific literacy

4. 4 Methods Pencil and paper test Multiple choice questions Constructed responses Student background questionnaire School Principal survey

5. 5 CYCLE OF MAJOR DOMAINS 2000 2003 Reading literacy Mathematical literacy Scientific literacy 2006

6. 6 PISA outcomes Profiles of knowledge and skills Contextual indicators Trends and league tables

7. 7 Instructions to students before the test You have been chosen to take part in an important international education study. This study is called the Programme for International Student Assessment, ‘PISA’ for short. Its goal is to find out what students your age all around the world know about reading, mathematics and science. There are about students representing. Around the world there are about 200,000 students involved, from more than 7000 schools in 40 countries. The results of the study will help countries determine what students are learning. Because the study may affect students all over the world in the future, we ask that you do the very best that you can.

8. 8 Instructions to students at the end of the cognitive session Please stop. Now turn to the last page or so in your booklet, where there is a question about calculator use and a question about effort. Please answer these now, and then close your booklet.

9. 9 The Effort Thermometer How much effort did you invest?

10. Please try to imagine an actual situation (at school or in some other context) that is highly important to you personally, so that you would try your very best and put in as much effort as you could to do well. In this situation you would mark the highest value on the “effort thermometer,“ as shown below: Compared to the situation you have just imagined, how much effort did you put into doing this test? How much effort would you have invested if your marks from the test were going to be counted in your school marks?  10    9  9  9  8  8  8  7  7  7  6  6  6  5  5  5  4  4  4  3  3  3  2  2  2  1  1  1

11. 11 Combined sample sizes for Country X and Y YearNo of students 20009597 200315997 Total25594

12. 12 Percentage of invalid cases

13. 13 New variable Relative Effort = School mark effort – PISA effort Idealists: negative relative effort Supporters: zero relative effort Realists: positive relative effort Cynics: extreme relative effort

14. 14 Relative Effort Distribution

15. 15 Relative Effort Distribution

16. 16 Relative Effort Distribution

17. 17 Relative Effort Distribution

18. 18 Means for Relative Effort Country XCountry Y 20001.742.14 20031.901.85 Note: higher scores mean lower effort Significant decline in effort Significant increase in effort

19. 19 Means for Relative Effort by gender Note: higher scores mean lower effort 23 40 16 23

20. Country X 2003 idealists supporters diligent realists realists cynics Relative effort 2000 Plausible value in Reading idealists supporters diligent realists realists cynics

21. idealists supporters diligent realists realists cynics Country Y 2003 Relative effort 2000 Plausible value in Reading idealists supporters diligent realists realists cynics

22. Relative effort by gender for reading idealists supporters diligent realists realists cynics Plausible value in reading Female Male Relative effort

23. 23 Transformed mean scores for reading

24. 24 Does effort count?

25. 25 Males Consistent reduction in performance after adjustment for effort Decrease rather than increase in performance after adjustment for effort

26. 26 Females Relatively consistent performances, both with and without adjustment Improvement (some, but not all) accounted for by effort

27. 27 Country X Decline in boys over time leading to increased gender differences. Effort not affecting the pattern

28. 28 Country Y Effort explains some of an increase in female performance Effort disguises a decline in male performance Adjusted results show a widening of gender differences that is more consistent with Country X

29. 29 Summary Effort is related to achievement but the relationship is non-linear Effort is less in Country Y than Country X but Country Y improved its effort investment from 2000 to 2003. Effort is less for boys than girls Effort explains some but not all of the differences between the achievement of boys and girls Effort explains some but not all of the improved performance of Country Y

30. 30 Future Direction Compare students matched by effort Compare effort distributions across countries Examine variance explained by effort Explore group differences after controlling for effort

31. 31