Peer feedback on a draft TMA answer: Is it usable? Is it used? Mirabelle Walker Department of Communication and Systems

Outline of talk The module context Introduction to the research Is the feedback from peers usable? Is the feedback from peers used?

The module context T215 Communication and information technologies Block 3 Communicating and collaborating Small group project – groups of up to 8 students Each group uses a wiki to create a shared document Overall topic –Online communication and collaboration

Group work using the wiki Choice of sub-topics from the block –Online communities; web 2.0; virtual worlds; etc. Each student –drafts a wiki page on their sub-topic –gives feedback on two members’ drafts –improves their draft page based on the peer feedback

The mark scheme Students are marked individually on –their final wiki page (30 marks) –the peer feedback they provided (10 marks) –using the peer feedback they received (5 marks) The group as a whole is marked on –how well they collaborated (5 marks)

The criteria students learn to use – and tutors use for TMA feedback Does the document meet the brief – exactly? Is it factually accurate? Is the structure appropriate for the audience, purpose and medium? Is the style appropriate for the audience, purpose and medium? Is the technical level appropriate for the audience, purpose and medium? Is the English correct?

Introduction to the research

The research questions What is the peer feedback like: how does it compare with feedback from tutors? is it usable? Did students pay attention to the feedback they received from peers?

Some relevant numbers 27 sample TMA 03s (one from each tutor) Mean number of comments per tutor on the final wiki page: examples of peer comments – limited to 250 words and assessed Mean number of comments per peer on the draft wiki page: 12.3

Some general findings Peers all used their word allocation – but some were more discursive than others Surprisingly little overlap between what the two peers said (only 16 out of 255 (6.3%) comments that pointed out problems and 54 out of 313 (17.3%) positive comments) – it appeared as if the second peer to post deliberately tried to find different things to say

Is the feedback from peers usable?

Criteria percentages

Criteria percentages (without ‘Other’)

Categories of comment from previous research Content Skills development Motivating Demotivating Reference to resource(s) Reference to future work

Categories of comment for this research Content Skills development Motivating Confirming Demotivating Reference to resource(s) Reference to future work

What is a ‘comment’? On the whole you’ve done a good job of meeting the brief, but you haven’t quite got the balance of topics right. You’ve spent too long on xxxx at the expense of yyyy.

What is a ‘comment’? On the whole you’ve done a good job of meeting the brief, motivating comment but you haven’t quite got the balance of topics right. You’ve spent too long on xxxx at the expense of yyyy. content comment

Category percentages

Were peers setting a lower standard than tutors (1)? Comparison of comments made by peers and tutor for any particular student showed that most of the difference in ‘praise’ and ‘confirm’ comments was because: –peers were being precise in addressing the sub-elements of meeting brief and structure in a way that tutors weren’t, and this accounted for much of the difference (silence from a tutor often implied a sub-element was OK) –peers nearly all made positive overall comments on the wiki page at the start and/or end of their feedback, whereas tutors often didn’t make any positive overall comments on the wiki page (even on the PT3) –peers tended to confirm style and technical level more than did tutors.

Were peers setting a lower standard than tutors (2)? Also Students were told in the question not just to focus on problems but to find positive things to say as well. So In general most peers were not setting a noticeably lower standard than tutors, and where they did it tended to be in one or two criteria, not across the board. In one or two cases the peers seemed to be setting a somewhat higher standard than the tutor!

‘Depth’ of comment Depth 1 – indicates e.g. ‘More needed here’; ‘Poor structure’, ‘Good’ Depth 2 – corrects/amplifies e.g. ‘You needed to mention xxxx’; ‘Your answer would have had a better structure if you had …’, ‘A good introduction’ Depth 3 – explains the correction/amplification e.g. ‘You needed to mention xxxx because …’; ‘Your answer would have had a better structure if you had … This is because …’, ‘This is a good introduction because you have …’

Depth percentages

Tone (when pointing out a problem) Tentative/suggesting Factual Exhortatory Questioning

Tone percentages

Is the feedback from peers used?

Did students use the peer comments they received? Of all the comments they received: they responded thoroughly to 40% they made some attempt to respond to 11% they did not respond to 49%

What made a difference to whether a student used a peer comment?

Did student’s ability make any difference to whether a student used a peer comment? No!

Did criterion make any difference to whether a student used a peer comment?

Did depth make any difference to whether a student used a peer comment?

Did tone make any difference to whether a student used a peer comment? No!

Some conclusions Most students have learned how to give feedback using the criteria in the module. The fact that peers gave a much higher proportion of motivating and confirming comments than tutors does not in general imply peers were setting a lower standard. Some students made much more use of peers’ feedback than others. The tone of a peer’s comment and the ability level of the recipient made no difference to whether the comment was used. Students were somewhat more likely to use comments that were related to structure or were of depth ‘explain’.

Answers to the research questions Is the feedback from peers usable? In general, yes. Is the feedback from peers used? By some students, yes – particularly so if a comment is about structure or of depth ‘explain’.