Presentation on theme: "Peer review with Aropa in Computing Science Dr Simon Rogers"— Presentation transcript:
Peer review with Aropa in Computing Science Dr Simon Rogers Simon.Rogers@glasgow.ac.uk
The course: PSI Professional Skills and Issues: – L4 and Msc students (>100) – Compulsory: Required for BCS accreditation – Non-technical: Ethics, IPR, Data protection etc – Not the most popular L4 course…… Assignment: – 1,500 word essay. – 2 stage peer review process. – Final mark (from academic) weighted by `engagement factor
Aropä Peer review management software developed by John Hamer (Auckland & SCS) and Helen Purchase (SCS). – email@example.com firstname.lastname@example.org – http://www.dcs.gla.ac.uk/~hcp/aropa Handles everything: – Submission (word/pdf, txt) – Assignment to reviews – Review rubric (design your own) – Returning reviews to reviewers – Reviewing of reviews Very easy to use. Local support available. – Willing to expand system to meet new requirements.
Peer review process 1.Trial exercise 1.500 word essay 2.2 reviews 3.Not assessed (almost 100% response) 2.Assessed exercise 1.Students submit draft essay on one of 3 topics (0.7). 2.Assigned 3 reviewers from same topic. 3.Reviewers assess work (numerical ratings and text justification) (0.05 x 3). 4.Reviews returned to authors who can use them to improve essay. 5.Each student also reviews 3 other reviews (0.05 x 3). 6.Final marking done by academics, weighted by factors (X)
Pros & Cons Pros: – Aropa very easy to use – assigning reviews, handling deadlines, monitoring submissions etc. – High level of engagement (almost all students completed everything). – Massive quantity of feedback (not always high quality) Cons: – Between cohort tension (L4 v MsC). – `Threat of plagiarism (by reviewers). – Lack of peer review experience – poor quality of feedback (first use of peer review for 85% of class). – Anonymity (.doc files).
Other feedback Only 20% of students would have been happy to be formally assessed by peers. Of those who thought the exercise improved their essay (42%!), 71% felt that the most helpful bit was receiving feedback.