1. Review in Computerized Peer- Assessment Dr Phil Davies Department of Computing Division of Computing & Mathematical Sciences FAT University of Glamorgan

2. AUTOMATICALLY CREATE A MARK THAT REFLECTS THE QUALITY OF AN ESSAY/PRODUCT VIA PEER MARKING, What do we need to provide to have fully Automated Peer –Assessment System? AUTOMATICALLY A MARK THAT REFLECTS THE QUALITY OF THE PEER MARKING PROCESS i.e. A FAIR/REFLECTIVE MARK FOR MARKING AND COMMENTING SLIGHT GAP

3. THE FIRST CAP MARKING INTERFACE

5. Typical Assignment Process Students register to use system - CAP Students register to use system - CAP Create an essay in an area associated with the module using an RTF template of headings Create an essay in an area associated with the module using an RTF template of headings Submit via Bboard Digital Drop-Box Submit via Bboard Digital Drop-Box Anonymous code given to essay automatically by system Anonymous code given to essay automatically by system Use CAP system to mark Use CAP system to mark

6. Self/Peer Assessment Often Self-Assessment stage used Often Self-Assessment stage used Set Personal Criteria Set Personal Criteria Opportunity to identify errors Opportunity to identify errors Get used to system Get used to system Normally peer-mark about 5/6 Normally peer-mark about 5/6 Raw peer MEDIAN mark produced Raw peer MEDIAN mark produced Need for student to receive Comments + Marks Need for student to receive Comments + Marks

7. Compensation High and Low Markers Need to take this into account Need to take this into account Each essay has a ‘raw’ peer generated mark - MEDIAN Each essay has a ‘raw’ peer generated mark - MEDIAN Look at each student’s marking and ascertain if ‘on average’ they are an under or over marker Look at each student’s marking and ascertain if ‘on average’ they are an under or over marker Offset mark given by this value Offset mark given by this value Create a COMPENSATED PEER MARK Create a COMPENSATED PEER MARK

8. Below are comments given to students. Select the 3 most Important to YOU 1. I think you’ve missed out a big area of the research 2. You’ve included a ‘big chunk’ - word for word that you haven’t cited properly 3. There aren’t any examples given to help me understand 4. Grammatically it is not what it should be like 5. Your spelling is atroceious 6. You haven’t explained your acronyms to me 7. You’ve directly copied my notes as your answer to the question 8. 50% of what you’ve said isn’t about the question 9. Your answer is not aimed at the correct level of audience 10. All the points you make in the essay lack any references for support

9. Each Student is using a different set of weighted comments Comments databases sent to tutor

10. First Stage => Self Assess own Work Second Stage (button on server) => Peer Assess 6 Essays Comments – Both Positive and Negative in the various categories. Provides a Subjective Framework for Commenting & Marking

11. Feedback Index Produced an index that reflects the quality of commenting Produced an index that reflects the quality of commenting Produced a Weighted Feedback Index Produced a Weighted Feedback Index Compare how a marker has performed against these averages per essay for both Marking + Commenting – Looking for consistency Compare how a marker has performed against these averages per essay for both Marking + Commenting – Looking for consistency

12. -5-4-3-2-0+0123456789 294441494653644953606269686982 38484751455458536262646573 4951506057576766 5158535059 5763 5965 64 04.25.01.43.54.06.84.83.63.94.72.53.12.80 294145484949565256585967647182

13. The Review Element Originally in Communications within CAP marking process, it requires the owner of the file to ‘ask’ questions of the marker Originally in Communications within CAP marking process, it requires the owner of the file to ‘ask’ questions of the marker Emphasis ‘should’ be on the marker Emphasis ‘should’ be on the marker Marker does NOT see comments of other markers who’ve marked the essays that they have marked Marker does NOT see comments of other markers who’ve marked the essays that they have marked Marker does not really get to reflect on their own marking – get a reflective 2 nd chance Marker does not really get to reflect on their own marking – get a reflective 2 nd chance I’ve avoided this in past -> get it right first time I’ve avoided this in past -> get it right first time

14. Click on button to get an essay previously marked + comments and marks

15. Click on button to get to view comments of another marker

16. Can change any marks and/or comments they feel appropriate and submit by clicking button

17. Trialled with Post-Graduate Group 13 students 13 students 76 markings 76 markings Average time per marking = 42 minutes (range 3-72) Average time per marking = 42 minutes (range 3-72) Average number of menu comments/marking = 15.7 Average number of menu comments/marking = 15.7 Peer Avge. mark = 59.69% (before review 60.15%) Peer Avge. mark = 59.69% (before review 60.15%) Number of students who did replacements = 10 (out of 13) Number of students who did replacements = 10 (out of 13) 41 ‘Replaced’ markings (54%) 41 ‘Replaced’ markings (54%) Out of 41 Markings ‘Replaced’ –> 26 changed mark 26/76 (34%) Out of 41 Markings ‘Replaced’ –> 26 changed mark 26/76 (34%) Only 33 out of 41 REALLY CHANGED ANYTHING Only 33 out of 41 REALLY CHANGED ANYTHING 2 students ‘Replaced’ ALL his/her markings 2 students ‘Replaced’ ALL his/her markings -8, -7, -7, -7, -6, -6,, -6, -5, -5, -5, -4, -4, -3, -3, -2, -2, -2, -2, -1, -1 -8, -7, -7, -7, -6, -6,, -6, -5, -5, -5, -4, -4, -3, -3, -2, -2, -2, -2, -1, -1 +1, +2, +6,+7, +8, +9 +1, +2, +6,+7, +8, +9

18. +7+6+5+4+3+2+1+0-0-2-3-4 816862725361524343 736651 51 Mapping of Feedback Indexes to Compensated Peer Essay Marks

19. #S/A Ref S/A Raw Peer Comp Peer Raw P/Rev Comp Peer P/Rev 1838370736772 2606069696968 3636254535453 4636247514751 545434543 6706849554652 7575764646162 8716086818681 97056515651 10696561626161 11797466716873 12665548444843 13696570656866 Initial Average Self Assessment 68.33% Reflective Average Self Assessment 64.63% -2 -9 -4 -5 -4

20. #S/A Ref S/A Raw Peer Comp Peer Raw P/Rev Comp Peer P/Rev 1838370736772 2606069696968 3636254535453 4636247514751 545434543 6706849554652 7575764646162 8716086818681 97056515651 10696561626161 11797466716873 12665548444843 13696570656866 Raw Peer Generated Mark Pre- Review 60.38% Compensated Peer Generated Mark Pre- Review 60.15% +3 +4 -2 +6 -5 +1 +5 -4 -5

21. #S/A Ref S/A Raw Peer Comp Peer Raw P/Rev Comp Peer P/Rev 1838370736772 2606069696968 3636254535453 4636247514751 545434543 6706849554652 7575764646162 8716086818681 97056515651 10696561626161 11797466716873 12665548444843 13696570656866 Raw Peer Generated Mark Post- Review 59.69% Compensated Peer Generated Mark Post- Review 59.69% +5 +4 -2 +6 +1 -5 0 +5 -5 -2

22. NEGATIVEPOSITIVE 87654321123456789 13232242110002111 AVERAGE MARK CHANGE = -1.69 Student Mark Changes During Review Stage 9 30-39 8 58-66 7 79-86 6 14-20; 40-46 8 67–59 7 73-67; 67-60; 60-53 6 73-67; 52-46

23. How to work out Mark (& Comment) Consistency Marker on average OVER marks by 10% Marker on average OVER marks by 10% Essay worth 60% Essay worth 60% Marker gave it 75% Marker gave it 75% Marker is 15% over Marker is 15% over Would expect 10% over, therefore Actual Consistency index (Difference) = 5 Would expect 10% over, therefore Actual Consistency index (Difference) = 5 If the marker on average had UNDER marked by 10% - Difference would have been 25 If the marker on average had UNDER marked by 10% - Difference would have been 25 Summing and Averaging these differences produces a Marking Consistency Index (low is good – high is poor) Summing and Averaging these differences produces a Marking Consistency Index (low is good – high is poor) This can be done for all marks and comments This can be done for all marks and comments

24. Pre-ReviewPost-Review Student Number Average Mark Difference Mark Consistency Average Mark Difference Mark Consistency 1-5.176.85-6.05.59 2-1.145.490.144.55 30.832.550.52.31 4125.5411.144.17 5 66.677.4677.59 7-19.2511.19-18.2510.78 8-11.678.58-9.834.54 9 107.836.1485.99 114.674.162.173.31 12-4.838.41-3.676.67 132.950.833.73 Would hope for MARK CONSISTENCY to DECREASE following review 6.30 -> 5.38 -1.26 -0.94 -0.24 -1.37 +0.13 -0.41 -4.04 -0.15 -0.85 -1.74 +0.78 Would hope for AVERAGE MARK DIFFERENCE to DECREASE following review -1.00 -> -0.72

25. Automatically Generate Mark for Marking Linear scale 0 -100 mapped directly to consistency … the way in HE? Linear scale 0 -100 mapped directly to consistency … the way in HE? Expectation of Normalised Results within a particular cohort / subject / institution? Map to Essay Grade Scale achieved (better reflecting ability of group)? Map to Essay Grade Scale achieved (better reflecting ability of group)?

26. Current ‘Simple’ method Average mark for essay e.g. 55% Average mark for essay e.g. 55% Ranges Highest – Lowest marks achieved for essay e.g. 45% 70% Ranges Highest – Lowest marks achieved for essay e.g. 45% 70% Average Marking Consistency e.g. 5.0 Average Marking Consistency e.g. 5.0 Ranges Highest – Lowest consistency indexes achieved e.g. 2.5 8.0 Ranges Highest – Lowest consistency indexes achieved e.g. 2.5 8.0 Essay 45 55% 70 Mark 8.0 5.0 2.5 Essay/Mark 3.33 / 1 6.0 / 1 e.g. Mark Cons = 6.0 > 1 below average Mark for marking = 55% - (1*3.33)= 51.66%

27. Student Number Mark Consistency Consistency Difference from Average 5.37 % Mark for Marking based on range diff. (Avge 59.69%) Mark they received for essay 15.59 + 0.22 59%72% 24.55 - 0.82 65%68% 32.31 - 3.06 81%53% 44.17 - 1.20 68%51% 5 67.59 + 2.22 53%52% 710.78 + 5.41 43%62% 84.54 - 0.83 65%81% 9 105.99 + 0.62 58%61% 113.31 - 2.06 74%73% 126.67 + 1.30 56%43% 133.73 - 1.64 71%66%

28. What about the commenting? Does not take into account the Quality of the Commenting Does not take into account the Quality of the Commenting Should look at the Average Feedback Differences per marker to get a Commenting Consistency Grade Should look at the Average Feedback Differences per marker to get a Commenting Consistency Grade Same as creating Mark Consistency Same as creating Mark Consistency Create a Commenting Consistency Index Create a Commenting Consistency Index

29. Student Number Average Feedback Difference Feedback Consistency Avge 2.80 Consistency Difference from Average % Mark for Comments based on range diff. (Avge 59.69%) Mark they received for essay 1-2.103.05+0.2559%72% 20.752.53-0.2763%68% 34.432.94+0.1459%53% 4-1.302.21-0.5965%51% 5 64.381.59-1.2175%52% 7-6.007.23+4.4343%62% 8-2.292.61-0.1962%81% 9 101.893.47+0.6757%61% 11-0.621.07-1.7381%73% 120.442.41-0.3964%43% 13-2.051.72-1.0873%66%

30. Student Number MARK FOR MARKING MARK FOR COMMENTING MARK FOR ESSAY FINAL MARK 60 / 20 / 20 159%59%72%67% 265%63%68%66% 381%59%53%60% 468%65%51%57% 5 653%75%52%57% 743%43%62%54% 865%62%81%74% 9 1058%57%61%60% 1174%81%73%75% 1256%64%43%50% 1371%73%66%68% Correlation between Marking & Commenting Consistency 0.49

31. Student Number MARK FOR MARKING MARK FOR COMMENTING MARK FOR ESSAY FINAL MARK 60 / 20 / 20 159%59%72%67% 265%63%68%66% 381%59%53%60% 468%65%51%57% 5 653%75%52%57% 743%43%62%54% 865%62%81%74% 9 1058%57%61%60% 1174%81%73%75% 1256%64%43%50% 1371%73%66%68% Correlation between Marking Consistency and Essay Mark 0.17

32. Student Number MARK FOR MARKING MARK FOR COMMENTING MARK FOR ESSAY FINAL MARK 60 / 20 / 20 159%59%72%67% 265%63%68%66% 381%59%53%60% 468%65%51%57% 5 653%75%52%57% 743%43%62%54% 865%62%81%74% 9 1058%57%61%60% 1174%81%73%75% 1256%64%43%50% 1371%73%66%68% Correlation between Commenting Consistency and Essay Mark 0.05 Final Grade for Coursework takes into account Essay Grade, Mark for Marking and Mark for CommentingP ercentages

33. Split in Marks 60 / 20 / 20 Is it reasonable? Is it reasonable? Higher Order skills of Marking – worth more? Higher Order skills of Marking – worth more? If we’re judging marking process on consistency – then should be rewarded for showing consistency within marking AND commenting If we’re judging marking process on consistency – then should be rewarded for showing consistency within marking AND commenting Revised split 60 / 15 / 15 / 10 Revised split 60 / 15 / 15 / 10

34. Student # MARK FOR MARKING MARK FOR COMMENTING MARK FOR BEING CONSISTENT IN MARKING & COMMENTING MARK FOR ESSAY FINAL MARK 15 / 15 / 10 / 60 159%59%80%72%69% 265%63%72%68%67% 381%59%43%53%57% 468%65%68%51%57% 5 653%75%43%52%55% 743%43%81%62%58% 865%62%68%81%75% 9 1058%57%79%61%62% 1174%81%58%73%73% 1256%64%56%43%49% 1371%73%74%66%69% Correlation between Final ESSAY grade & Mark for Marking, Commenting & Consistency is 0.54 Correlation between FINAL MARK & ESSAY GRADE is 0.85

35. Student # MARK FOR MARKING MARK FOR COMMENTING MARK FOR BEING CONSISTENT IN MARKING & COMMENTING MARK FOR ESSAY FINAL MARK 15 / 15 / 10 / 60 159%59%80%72%69% 265%63%72%68%67% 381%59%43%53%57% 468%65%68%51%57% 5 653%75%43%52%55% 743%43%81%62%58% 865%62%68%81%75% 9 1058%57%79%61%62% 1174%81%58%73%73% 1256%64%56%43%49% 1371%73%74%66%69% IS IT WORTH THE HASSLE??

36. Student Comments Have you used Peer-Assessment in Past? Have you used Peer-Assessment in Past? Have you used Peer-Assessment in Past? Have you used Peer-Assessment in Past? How did you find self-assessment? How did you find self-assessment? How did you find self-assessment? How did you find self-assessment? Creating the Comments Database? Creating the Comments Database? Creating the Comments Database? Creating the Comments Database? How did they find using the CAP system and peer-assessment? How did they find using the CAP system and peer-assessment? How did they find using the CAP system and peer-assessment? How did they find using the CAP system and peer-assessment? Thoughts on new Review Stage? Thoughts on new Review Stage? Thoughts on new Review Stage? Thoughts on new Review Stage? Thoughts on Mark for Marking? Thoughts on Mark for Marking? Thoughts on Mark for Marking? Thoughts on Mark for Marking?

37. Two Main Points to Consider How do we assess the time required to perform the marking task? How do we assess the time required to perform the marking task? What split of the marks between creation & marking What split of the marks between creation & marking Definition Definition Definition Student or Lecturer Comments Student or Lecturer Comments Student or Lecturer Comments Student or Lecturer Comments

38. Contact Information Email: pdavies@glam.ac.uk Phone: 01443 - 482247 Dr Phil Davies J317 Department of Computing & Mathematical Sciences Faculty of Advanced Technology University of Glamorgan

40. THE END

41. General View of Peer Assessment Lecturer or Student? Lectures getting out of doing their jobs i.e. marking Good for developing student skills & employability How can all students be expected to mark as ‘good’ as ‘experts’ Why should I mark ‘properly’ and waste my time - I get a fixed mark for doing it The feedback given by students is not of the same standard that I give. The best thing I’ve ever done to make me reflect salary WANT BENEFITS NOW!! VERY PERSONAL

42. Defining Peer-Assessment In describing the teacher.. In describing the teacher.. A tall b******, so he was. A tall thin, mean b******, with a baldy head like a light bulb. He’d make us mark each other’s work, then for every wrong mark we got, we’d get a thump. That way – he paused – ‘we were implicated in each other’s pain’ McCarthy’s Bar (Pete McCarthy, 2000, page 68)

43. Student Comments Used Peer-Assessment in past? Used Peer-Assessment in past? None to any real degree None to any real degree A couple for staff development type activities A couple for staff development type activities

44. Student Comments How did you feel about performing Self- assessment? Very Difficult. Very Difficult. Helped to promote critical thinking ready for peer-assessment stage. Helped to promote critical thinking ready for peer-assessment stage. Made me think about how I was going to assess others Made me think about how I was going to assess others

45. Student Comments Creating Comments Database? Creating Comments Database? Very difficult – not knowing what comments they’d need Very difficult – not knowing what comments they’d need Weighting really helped me create criteria ready for marking Weighting really helped me create criteria ready for marking Could have helped to do dummy marking Could have helped to do dummy marking

46. Student Comments How did they find using the CAP system and peer-assessment? How did they find using the CAP system and peer-assessment? Very positive & interesting Very positive & interesting Very time consuming Very time consuming Would do it better next time Would do it better next time Important to maintain anonymity Important to maintain anonymity Interesting & complex – thought more about the assessment process Interesting & complex – thought more about the assessment process Really helped student development Really helped student development

47. Student Comments Thoughts on new Review Stage Thoughts on new Review Stage Liked 2 nd chance to review own marks Liked 2 nd chance to review own marks Gained experience going through process Gained experience going through process Didn’t really take much note of peers’ comments Didn’t really take much note of peers’ comments Liked to see that others felt the same about an essay Liked to see that others felt the same about an essay

48. Student Comments Thoughts on Mark for Marking Thoughts on Mark for Marking Good rewarded appropriately Good rewarded appropriately Difficult to fully understand Difficult to fully understand Let owner of essay provide mark for marking Let owner of essay provide mark for marking