Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis Dr Richard Parsons Centre for Learning and Teaching University of Dundee

Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis Overall position of CAA at University of Dundee Why self and peer assessment? Development story Key features –Academic author sets exercise online –Students use web, and text messaging –Robust reliable, secure –Checks for copying Demonstration – create an exercise, live Using the system – key points –Good question design –Involve the students –Teach the students Student activities and responses Marking moderation and plagiarism The future

University of Dundee Learning Technology Systems Blackboard VLE (5.5 Unix, upgrading to Enterprise 6.0 Unix) QuestionMark Perception 3.4, dual server system –1. Low stakes/formative server coupled to Blackboard (Oracle Unix database) –2. High stakes/summative server runs stand alone (Oracle Unix database) Provide full support for CAA including a “CAA Online” staff education course, initial authoring service Have Policy & Procedure documents for CAA (based on BS 7988), (also for VLE and eLearning strategies) Provide support for a range of additional services including online evaluation forms, plagiarism detection service, online staff eLearning tuition, publicise innovative solutions for supporting and enhancing learning (e.g Self and Peer Assessment Program)

P3 Student Management System e-Directory QMP formative server CAA-01 QMP Exam server CAA-02 VLE Integrated Systems University of Dundee MLE User exists 3 weeks before appointment Completed daily Student exists Staff details also stored in SMS Module enrolments of students Module assessment mark storage LDAP client? Create user name and process LDAP authentication Remove user process Library Catalogue access Resource management system LDAP client? Module enrolment Bb client Mark return Special exam enrolment Mark return Blackboard VLE – ELS 6.0 GroupWise Web Novell Fileservers Student Desktop LDAP Staff Assignment Record Secure web access Hardware 2 * 480s Version Users created from SITS SMS system Module enrolment Staff assignments from web system LDAP client Module assessment mark storage Consultancy visit 12 May – 16 May Training teaching staff Richard Parsons, CLT LEU

Why Self and Peer Assessment? Assessment is one of the keys to learning Being able to assess your own work is a vital skill Developing peer assessment skills is important for future careers By carrying out assessments, students learn about the assessment process Shifts tutor work from post to pre-assignment Web-based and innovative, so can interest the students

Development Story First a system to replace some lectures when I was overseas First self-assessment only, but peer assessment seen as important, results ed and texted to students Developed into a system that allowed online authoring Added the ability to tutor mark as well as self & peer assessment Made more robust based on usage –Remove the back button –Disable right click –Provide more guidance to staff and student users Added a check for copying and plagiarism Tidied up code to W3C compliance

Online Self & Peer Assessment of Text (Outline of the process) Question provided, answer researched and composed 1. Student submits their text answer (may copy and paste) 3. Student completes an assessment of one of their peers 2. Self assessment completed by student using criteria offered 5. Summary of marks presented. Student thanked for completing exercise, brief feedback completed 4. Student completes an assessment of the second of their peers Peer 1 assessment ed anonymously to student author Tutor can mark a selection of answers and moderate and collate results Final marks are checked and transferred to register Instructor designs question, sets word limits and details criteria and scoring ranges All files created, URLs provided to instructor

Technical Details Author completes webform Perl program creates HTML pages for input Perl files to process input HTML pages for output Perl file to process output User interaction And sets file protections Author access to results All operates on a Unix web server, with hyperlinks as required to Blackboard Sets question & marking criteria

Demonstration of the Program Create a NEW exercise, live and interactive?

Using the System – Key Points Good question design Involve the students Teach the students

Receive the first assignment, question and instructions Research the first topic, craft your first text answer Mark your first answer using the criteria provided and mark two peer’s answers Receive the second assignment and instructions Research the second topic, craft your second text answer Mark your second answer using the criteria provided and mark two peer’s answers Summary marks returned via VLE – Self, peer and tutor marks compared. Statistics provided overall to judge the success of the exercise Complete a short evaluation form where your opinion of the exercise will be sought. Week containing Tuesday 11 March Week containing Tuesday 15 April Student Instructions – Online Self and Peer Assessment Exercise Dr Richard Parsons Tutor marking of exercise 1 Tutor marking of exercise 2 Marks sent to student author

Student Performance Environmental science exercise - Write a short piece of text that describes the major scientific reasons why global warming is almost certainly occurring at present. The total length of the text must be between 200 to 300 words. Students performed well, range of marks awarded and obtained. Reasonable correlation between self, peer and tutor marking Students recognised that they were learning about assessment. Positive feedback received One of those exercises where you see the students learning

Criterion vs Norm-based Marking

Student comments demonstrate increased understanding “A good computer program that more classes should use” “A great learning tool for the students. Enjoyable too, better than the labs” “Useful for understanding what markers look for in a piece of work” “It was interesting to see how the marking process takes place, and how you can have an essay that in its own right is well written but does not contain the points needed by the marker” “Assessing others gives me comparison to my own” “Good to see how we can improve our work by our mistakes” “Useful for the ability to receive results via text message” “It was good to be able to compare your own work with others and offered an insight into the basic processes staff must use” “Makes you realise that you need more depth in your answer and to read the question” “It is quite funky”

Demonstration of plagiarism/copying/ collusion detection A check for copying is now embedded within the program Operates in the following manner: Recursive check for 6 word sets, ignoring most punctuation Tutor can insert text from web pages, textbooks and lecture notes as required No service is used to store the student’s work, so no complicated legal issues Will not pick up copying from year to year, or institution to institution

Online Self and Peer Assessment Exercise Formative to Summative Assessments: Considerations FormativeSummative Low stakesHigh stakes Random tutor marking, copying detection Comparative tutor marking, plagiarism detection No tutor marking, copying detection Checks Initial exercise limited training Students experienced at self and peer assessment Training in assessment Experience Little moderation of marking Students gain marks for marking correctly Moderation Anomalous marks adjusted

Further development Should be able to soon offer to include images check submission step present criteria to students before exercise students receive marks for marking And perhaps eventually include feedback on feedback computer marking of free text

Self and Peer Assessment What I leave you with Online system is available Relatively easy to use with supporting documentation It has some features that can engage students (text messaging) and staff (plagiarism detection) Academic issues remain important –Question design –Students trained in assessment techniques –Student understanding of what constitutes plagiarism –Policy for dealing with plagiarism (formative work) Further developments are possible, suggestions welcomed