1. The Quality of Peer-Feedback in the Computerised Peer-Assessment of Essays? The case for awarding ‘marks for marking’ based upon peer feedback not marks. Phil Davies School of Computing OR Lecturer getting out of doing marking

2. Need for assessment? As tutors we are trying to “separate” the sheep from the goats via the assessment process. As tutors we are trying to “separate” the sheep from the goats via the assessment process. This can often be difficult with the time constraints imposed on tutors, in what Chanock describes as “more goat-friendly times” (Chanock, 2000). This can often be difficult with the time constraints imposed on tutors, in what Chanock describes as “more goat-friendly times” (Chanock, 2000). Feedback against time! Feedback against time!

3. Defining Peer-Assessment In describing the teacher.. In describing the teacher.. A tall b******, so he was. A tall thin, mean b******, with a baldy head like a lightbulb. He’d make us mark each other’s work, then for every wrong mark we got, we’d get a thump. That way – he paused – ‘we were implicated in each other’s pain’ McCarthy’s Bar (Pete McCarthy, 2000,page 68)

4. Why Peer-Assessment Working together to learn (and be assessed) Working together to learn (and be assessed) Perceptions Perceptions –Staff > Lecturer getting easy life not marking –Student> Lecturer getting easy life not marking Student Awareness of Benefits …. IMPORTANT TANGIBILITY Student Awareness of Benefits …. IMPORTANT TANGIBILITY Success dependant upon scalability (computerisation).. “Student Numbers have risen dramatically since 1991 without a concomitant increase in resources” (Pond et al, 1995) Success dependant upon scalability (computerisation).. “Student Numbers have risen dramatically since 1991 without a concomitant increase in resources” (Pond et al, 1995)

5. Computerised Peer-Assessment CAP System CAP System Permits students to mark & comment the work of other students. (normally 6-8) Permits students to mark & comment the work of other students. (normally 6-8) Also initial self-assess stage (reflection) … used as a standard of expectation Also initial self-assess stage (reflection) … used as a standard of expectation Internet, not Web-based system (developed in Visual Basic / Access) Internet, not Web-based system (developed in Visual Basic / Access)

7. Having done the marking, what next? Students should receive feedback Students should receive feedback What feedback? What feedback? –Marks –Comments Which is most important? Which is most important? –To students or staff

9. AUTOMATICALLY EMAIL THE MARKER.. ANONYMOUS

10. What should the marker do? Reflect Look at essay again Look at essay again Take into account the essay owner’s comments Take into account the essay owner’s comments Further clarification (if it is needed, then is this a ‘black mark’ against the marker?) Further clarification (if it is needed, then is this a ‘black mark’ against the marker?) Try to ‘appease’ the essay owner? Try to ‘appease’ the essay owner? Modify mark based upon reflection? Modify mark based upon reflection? Give more feedback Give more feedback

12. Exercise One In the top box … what are the inherent advantages and disadvantages of peer- assessment In the top box … what are the inherent advantages and disadvantages of peer- assessment Middle Box … Mark (/10) and comment to support Middle Box … Mark (/10) and comment to support Bottom Box … Mark the marker /10 and comment to support Bottom Box … Mark the marker /10 and comment to support Now give the work back! Now give the work back!

13. Must be rewarded for doing the ‘mark for marking’ process.. Based on quality How to judge? How to judge? Standard of expectation (self-assessment) Standard of expectation (self-assessment) Marking consistency Marking consistency Commenting, quality, measure against mark Commenting, quality, measure against mark Discussion Element Discussion Element Need for additional comments – black mark? Need for additional comments – black mark? Reaction to requests / further clarification Reaction to requests / further clarification

15. How easy to get an automated mark for marking? Statistically fairly easy to create a mark for marking based upon marks Statistically fairly easy to create a mark for marking based upon marks Take into account high and low markers Take into account high and low markers Standard of expectation Standard of expectation Consistency … judge against final mark awarded for an essay (compensated median) Consistency … judge against final mark awarded for an essay (compensated median) What about the comments? What about the comments?

16. Feedback Index Produce an index that reflects the quality of commenting Produce an index that reflects the quality of commenting Produce an average feedback index for an essay (also compensated?) Produce an average feedback index for an essay (also compensated?) Compare against marker in a similar manner to marks analysis Compare against marker in a similar manner to marks analysis Where does this feedback index come from and is it valid? Where does this feedback index come from and is it valid?

17. The way to get the feedback index? Develop an application?? Develop an application?? –C-Rater? Spelling Mistakes Spelling Mistakes Similar Meanings? Similar Meanings? –That was cool –Really Choc –Really Good Essay Manually Manually

19. Commonality!! In the 67 essays that were marked In the 67 essays that were marked –Only 96 comments –44% positive and 56% negative –Highly critical if something not explained properly (21% of total comments (of which 73% were negative) Comments grouped into 10 categories Comments grouped into 10 categories Need to QUANTIFY these comments.. Feedback index Need to QUANTIFY these comments.. Feedback index Create a database holding positive & negative (by category) Create a database holding positive & negative (by category)

26. -7-6-5-4-3-2-0+0+1+2+3+4 34 51565256616370646870 39526146575862726570 444552546659626968 3435576665 6669 565968646371 615966 5548606968 6157586573 596160 63 62 68 3437.845.859.655.658.862.962.867.568 70

27. General points of note Main criteria used for peer-marking Main criteria used for peer-marking –Explanations, conclusions, references, examples Better students were critical (upper quartile) Better students were critical (upper quartile) –Better understanding permitting criticism? –Confidence? –Hostility? Weaker student in ‘cloud cuckoo’ land … consistently Weaker student in ‘cloud cuckoo’ land … consistently Consistency: 30-34, 35-39, 55-59; 65-69; 70-74 Consistency: 30-34, 35-39, 55-59; 65-69; 70-74 Note 70-74 OVER-Comment & Mark Note 70-74 OVER-Comment & Mark

28. Time Consuming? Can we formulate the marking process Can we formulate the marking process Take away need for quantification process of analyzing comments Take away need for quantification process of analyzing comments Is it still peer-assessment if the students are told what to say? Is it still peer-assessment if the students are told what to say?

30. Exercise Two 1. I think you’ve missed out a big area of the research 2. You’ve included a ‘big chunk’ that you haven’t cited 3. There aren’t any examples given to help me understand 4. Grammatically it is not what it should be like 5. Your spelling is atroceious 6. You haven’t explained anything to me 7. You’ve directly copied my notes as your answer to the question 8. Most of what you’ve said is wrong

32. STUDENT FRED REFERENCES: Positive ……… Negative ……. Personal Valuation 5, 3, 2, 1 3, 1, 2

33. Is it my job to teach students how to write essays, etc? Assessment MUST be directed Assessment MUST be directed Why bother writing essays, doing exam questions, etc. … doesn’t relate to needs or learning outcomes of subject Why bother writing essays, doing exam questions, etc. … doesn’t relate to needs or learning outcomes of subject Post HND … N-tier … Assess the essays of the final year (last year) Post HND … N-tier … Assess the essays of the final year (last year) Preparation/Research: Judge knowledge against last year’s results.. Both marks & comments Preparation/Research: Judge knowledge against last year’s results.. Both marks & comments Mistake!! Mistake!!

34. Can peer-assessment benefit other subject areas? Java Programming with Coursemarker Java Programming with Coursemarker Stuart Lewis’ idea Stuart Lewis’ idea Students create a solution to a programming assignment Students create a solution to a programming assignment Submission(s) Submission(s) Peer-Evaluate other solutions Peer-Evaluate other solutions Comments … Marks for Marking (weightings) Comments … Marks for Marking (weightings)

35. CourseMarker Core CM File Storage System Marking System Evaluation System Exercise Developm. SystemStudent Exercise Environment assignments exercises notes questions test methods solution template marking scheme exercise setup submission edit compile link run feedback and mark final mark position in class course statistics flagging-up of ‘problem cases’ immediate support TEACHER STUDENT re-usability automated marking - fair - frees time plagiarism check steep learning curve difficult setup (but it’s getting easier) immediate feedback fast support additional overheads TEACHER STUDENTS UNIX (Linux), Windows, Mac, based all platforms Assessment of text I/O assignments only no marking of graphical output remote student / teacher access distance learning, open all hours Advantages / Disadvantages FEATURES Computer Assisted Teaching and Assessment Modula-2 Java C comments / questions

36. PeerMarker Screen

37. Student while marking Exposure to different solutions Exposure to different solutions Development of critical evaluative skills Development of critical evaluative skills Useful experience of reading code for future employment situations Useful experience of reading code for future employment situations Plagiarism? … Good solution / No understanding Plagiarism? … Good solution / No understanding

38. Student while reviewing feedback from peers Range of subjective marking Range of subjective marking Confirmation of objective automated marking Confirmation of objective automated marking Anonymous discussion between marker and marked Anonymous discussion between marker and marked

39. Current position Test system working Test system working Changes following beta test in progress Changes following beta test in progress Plans to try sample study again (at a more convenient time, and with added rewards!) Plans to try sample study again (at a more convenient time, and with added rewards!) Employ 2 nd Placement Student Employ 2 nd Placement Student Graphical Interface Graphical Interface

40. Some Points Outstanding or Outstanding Points What should students do if they identify plagiarism? What should students do if they identify plagiarism? Is it ethical to get students to mark the work of their peers? Is it ethical to get students to mark the work of their peers? Is a computerised solution valid for all? Is a computerised solution valid for all? At what age / level can we trust the use of peer assessment? At what age / level can we trust the use of peer assessment? How do we assess the time required to perform the marking task? How do we assess the time required to perform the marking task? What split of the marks between creation & marking What split of the marks between creation & marking