1. University of York May 2008 Using assessment to support student learning Graham Gibbs

2. Personal Background Open University 1975-80 and 1997-2003 –Top in National Student Survey, especially assessment and feedback ratings Oxford Brookes University 1980-1997 –Most ‘coursework’ assessment –Systematic course design University of Oxford 2004- –Least ‘coursework’ assessment –Top in National Student Survey, especially assessment and feedback ratings –‘No course design’

3. Personal Background Practical books and articles about assessment ‘53 Interesting ways to assess your students’ ‘Assessing Student Centred Courses’ ‘Assessing Large Classes’ Consultancy to universities on strategic decisions about assessment policy Research into the impact of assessment on student learning

4. Student experience of assessment

5. “I just don’t bother doing the homework now. I approach the courses so I can get an ‘A’ in the easiest manner, and its amazing how little work you have to do if you really don’t like the course.”

6. “I am positive there is an examination game. You don’t learn certain facts, for instance, you don’t take the whole course, you go and look at the examination papers and you say ‘looks as though there have been four questions on a certain theme this year, last year the professor said that the examination would be much the same as before’, so you excise a good bit of the course immediately…”

7. “The feedback on my assignments comes back so slowly that we are already on the topic after next and I’ve already submitted the next assignment. I just look at the mark and throw it in the bin”

8. “ The tutor likes to see the right answer circled in red at the bottom of the problem sheet. He likes to think you’ve got it right first time. You don’t include any workings or corrections – you make it look perfect. The trouble is when you go back to it later you can’t work out how you did it and you make the same mistakes all over again”

9. “One course I tried to understand the material and failed the exam. When I took the resit I just concentrated on passing and got 98%. My tutor couldn’t understand how I failed the first time. I still don’t understand the subject so it defeated the object, in a way”

10. “I do not like the on-line assessment method…it was too easy to only study to answer the questions and still get a good mark … the wrong reasoning can still result in the right answer so the student can be misled into thinking she understands something … I think there should have been a tutor-marked assessment part way through the course so someone could comment on methods of working, layout etc.”

11. “We were told this course was going to be an opportunity to be creative, to take risks. Then in week five we were hit with a multiple choice question test and we realised what it was really all about.”

12. Summative assessment that is redundant Most students can, in their first year, predict their final results with some accuracy As few as 5% of assessments are necessary to produce the same overall grades

13. Assessment that improves learning The case of the Engineer The case of the Manager The case of the Pharmacist The case of the Psychologist The case of the Accountant

14. The case of the engineer Weekly lectures, problem sheets and classes Marking impossible Problem classes large enough to hide in Students didn’t tackle the problems Exam marks: 45%

15. The case of the engineer Course requirement to complete 50 problems Peer assessed in six ‘lecture’ slots Marks do not count Lectures, problems, classes, exams unchanged

16. The case of the engineer Course requirement to complete 50 problems Peer assessed in six ‘lecture’ slots Marks do not count Lectures, problems, classes, exams unchanged Exam marks increased from 45% to 85% Why did it work?

17. The case of the engineer time on task social learning and peer pressure timely and influential feedback learning by assessing –error spotting –developing judgement (internalisation of standards) –self-supervision (meta-cognitive awareness)

18. Assessment that improves learning The case of the Engineer The case of the Manager The case of the Pharmacist The case of the Psychologist The case of the Accountant

19. “Conditions under which assessment supports student learning”

20. Quantity and distribution of student effort 1 Assessed tasks capture sufficient student time and effort 2These tasks distribute student effort evenly across topics and weeks

21. Quality and level of student effort 3These tasks engage students in productive learning activity 4Assessment communicates clear and high expectations to students

22. Quantity and timing of feedback 5Sufficient feedback is provided, both often enough and in enough detail 6The feedback is provided quickly enough to be useful to students

23. Quality of feedback 7Feedback focuses on learning rather than on marks or students themselves 8Feedback is understandable to students, given their sophistication

24. Student response to feedback 9 Feedback is received by students and attended to, and is acted upon by students to improve their work or their learning

25. Effective assessment tactics Bioscience: poster reports Engineering: sampling lab reports + cheap feedback Law: essay requirements + sampling + models Estates management: project exams French Literature: critiquing texts under examination conditions

26. Assessment Experience Questionnaire Measures extent to which the ‘conditions’ are perceived to be met –Quantity and distribution of effort –Quality, quantity and timeliness of feedback –Use of feedback –Impact of exams on quality of learning –Deep approach –Surface approach –Clarity of goals and standards –Appropriateness of assessment

29. ‘University A’ is the Open University 8 assignments per course Detailed written feedback on every assignment Quality assurance of feedback Less funding per student than any other university, mainly spent on feedback Best student ratings nationally Much pedagogic research Formative-only early assignments improve retention Computer-based assignments reduce retention and performance

30. …then I went to Oxford Oxford responds in a limited way to most national quality assurance guidelines –learning outcomes –assessment criteria –alignment of assessment with aims Oxford has not ‘modernised’ its assessment –reliance on examinations, little assessed coursework, little summative assessment of any kind, no modularisation

31. …then I went to Oxford Oxford responds in a limited way to most national quality assurance guidelines –learning outcomes –assessment criteria –alignment of assessment with aims Oxford has not ‘modernised’ its assessment –reliance on examinations, little assessed coursework, little assessment of any kind, no modularisation Outstanding quality of student experience at Oxford –student retention of 98% (1 st in UK) –Oxford ranked 1 st for teaching in UK (Times, Guardian) –better CEQ scores than elsewhere in world –better NSS ratings than the Open University

32. Research questions What are the characteristics of programme level assessment environments that are associated with positive student learning responses? Are the characteristics of programme level assessment environments that are most closely associated with positive student learning responses those that quality assurance regulations emphasise?

33. Research design Three contrasting universities (Oxford, pre-1992, post 1992) Three contrasting programmes in each (Humanities, Science, Applied Social Science) Characterise assessment environments –Read documentation (all modules) –Interview programme leader, lecturers and students Administer AEQ Explore relationships between characteristics of programme level assessment design and qualities of student learning -with Harriet Dunbar-Goddet, Chris Rust and Sue Law -funded by the Higher Education Academy

34. Coding characteristics of programme level assessment environments % marks from examinations Volume of summative assessment Volume of formative only assessment Volume of (formal) oral feedback Volume of written feedback Timeliness: days after submission before feedback provided Explicitness of criteria and standards Alignment of goals and assessment

35. Coding characteristics of programme level assessment environments % marks from examinations –High: more than 70% –Med: between 40% and 70% –Low: less than 40%

36. Coding characteristics of programme level assessment environments Explicitness of criteria and standards –High: clear criteria for most assignments & exams; link to grades; effort made to enable students to internalise criteria & standards –Low: explicit criteria and standards rare and/or nebulously formulated; marks/grades arrived at through global judgment in tacit way; no effort to enable students to internalise criteria and standards

37. Range of characteristics of programme level assessment environments

38. % marks from exams: 17% - 100%

39. Range of characteristics of programme level assessment environments % marks from exams: 17% - 100% number of times work marked: 11 - 95

40. Range of characteristics of programme level assessment environments % marks from exams: 17% - 100% number of times work marked: 11 - 95 number of times formative-only assessment: 2 - 134

41. Range of characteristics of programme level assessment environments % marks from exams: 17% - 100% number of times work marked: 11 - 95 number of times formative-only assessment: 2 - 134 number of hours of oral feedback: 3 - 68

42. Institutional assessment environments Aspect of assessment OxbridgePost- 1992 Pre-1992 Volume/frequency of formative only assessment HighLowMed % marks from examsHighLowMed % marks from coursework LowHighMed Alignment of learning activity with assessment LowHighMed Explicitness of goals/outcomes and criteria LowHighLow

43. Patterns of assessment features within programmes every programme that is high on the volume of formative assessment is low on the volume of summative assessment no examples of high volume of summative assessment and high volume of feedback

44. Patterns of assessment features within programmes every programme that is low on the volume of summative assessment is high on the volume of formative assessment no examples of high volume of summative assessment and high volume of feedback there may be enough resources to mark student work many times, or to give feedback many times, but not enough resources to do both

45. Relationships between assessment characteristics and student learning

46. Assessment characteristics and student learning response: 1 When the level of explicitness of criteria and standards is high, students’ experience is characterised by: Less coverage of the syllabus Less and poorer quality feedback Less use of feedback Less learning from the examination Less deep approach

47. Assessment characteristics and student learning response: 2 When the level of alignment of goals and standards is high, students’ experience is characterised by: Less coverage of the syllabus Less and poorer quality feedback Less use of feedback Less appropriate assessment Less clear goals and standards Less learning from the examination Less deep approach

48. Assessment characteristics and student learning response: 3 When the level variety of assessment methods is high, students’ experience is characterised by: Less and poorer quality feedback Less use of feedback Less appropriate assessment Less clear goals and standards Less learning from the examination Less deep approach More surface approach Less overall satisfaction

49. Assessment characteristics and student learning response: 4 When the volume of formative-only assessment is high, students’ experience is characterised by: More coverage of the syllabus More and better quality feedback More use of feedback More appropriate assessment More clear goals and standards More learning from the examination More deep approach More overall satisfaction

50. Assessment characteristics and student learning response: 5 When the volume of oral feedback is high, students’ experience is characterised by: More coverage of the syllabus More and better quality feedback More use of feedback More appropriate assessment More clear goals and standards More learning from the examination More deep approach More overall satisfaction

51. Assessment characteristics and student learning response: 6 When the timeliness of feedback is high, students’ experience is characterised by: More effort More coverage of the syllabus More and better quality feedback More use of feedback More appropriate assessment More clear goals and standards More learning from the examination

52. Summary Explicitness of criteria and standards, alignment of goals and assessment and variety of assessment are all associated with a negative learning experience …they are also associated with more summative and less formative-only assessment, less oral feedback and less prompt feedback

53. Summary Explicitness of criteria and standards, alignment of goals and assessment and variety of assessment are all associated with a negative learning experience …they are also associated with more summative and less formative-only assessment, less oral feedback and less prompt feedback Formative only assessment, oral feedback and prompt feedback are all associated with positive learning experience …even when they are also associated with lack of explicitness of criteria and standards, lack of alignment of goals and assessment and a narrow range of assessment.

54. Why? being explicit does not result in students being clear about what they are supposed to be doing or what counts as high quality ‘legitimate peripheral engagement in a community of practice’ (Lave and Wenger; Price et al)

55. Why? Students experience very varied forms of assessment as confusing: ambiguity and anxiety are associated with a surface approach Feedback improves learning most when there are no marks Possible to turn feedback round quickly when there are no QA worries about marks

56. …alternative explanation A The features of assessment environments identified here that appear to have negative consequences for student learning are also the features that are associated with modular courses in which each separate module has to have ‘self-contained’ assessment within a short time frame. Conclusion It may be modularity, rather than QA regimes, that have caused some of the problems. Oxbridge is not modular (the Open University is…but has huge and long modules that are usually studied one at a time)

57. …alternative explanation B High volumes of assessed ‘coursework’ have been introduced in part to increase student engagement Student engagement improves learning outcomes High % of marks from coursework is associated with higher marks and better degrees (at post ’92 universities) However… The effect of innovations that enhance engagement on learning outcomes only holds for low ability students (high ability students engage themselves) (Carini et al, in press) The present study did not control for student ability

58. Conclusions Assessment has more impact on how students go about studying, on the quantity and quality of their effort, and on their performance – than does teaching It is relatively easy (and often cheap) to change student learning by changing assessment, provided the ‘conditions’ are met effectively Whole universities have implicit conventions about what is ‘acceptable’ in terms of assessment practice Some of these conventions are ill-informed and damaging … and are built in to QA systems Local contexts are likely to require different assessment strategies to engage their students.

59. References Carini, R.M., Kuh, G.D. & Klein, S.P. (in press) Student engagement and student learning: testing the outcomes. Research in Higher Education Dunbar-Goddet, H. & Gibbs, G. (under review) A methodology for evaluating the effects of programme assessment environments on student learning. European Association for Research into Learning and Instruction, Assessment Conference, Northumbria. Gibbs, G. (2002) Evaluation of the impact of formative assessment on student learning behaviour. European Association for Research into Learning and Instruction. Newcastle: Northumbria University. August 2002. Gibbs, G. & Simpson, C. (2003) Measuring the response of students to assessment: the Assessment Experience Questionnaire. 11 th International Improving Student Learning Symposium, Hinckley. Gibbs, G. & Simpson, C. (2004) Conditions under which assessment supports student learning. Learning and Teaching in Higher Education 1, pp3-31. Gibbs, G., Simpson, C. & Macdonald, R. (2003) Improving student learning through changing assessment – a conceptual and practical framework. European Association for Research into Learning and Instruction Conference, Padova, Italy.