1. A system for enhancement of manual exam reviewing process Ivan Budišćak Boris Milašinović Faculty of Electrical Engineering and Computing University of Zagreb, Croatia

2. Environment 1 st year of study Programming and Software Engineering Data Structures and Algorithms ◦ >=700 students on each course ◦ Continuous assessment  Using software for automatic evaluation of assignments (Ahyco)  Homeworks  Quizzes (Multiple choice tests)  Manually reviewing  2 mid-term exams and final exam  Class activity

3. Traditional manual exam reviewing (with pencil and paper) Pros: ◦ “Natural” way to review exams ◦ Can be done everywhere (in case you have access to exams ) Cons: ◦ Difficult to split exams on more than one reviewer (lot of merging and sorting) ◦ Unable to have dislocated reviewers ◦ Students have to be able to see reviewed exams (additional time and place) ◦ Needs place for storing exams

4. Digital exams archives Retain manually reviewing process Keep students exams and reviewing remarks together in digitalised archive Motivation: ◦ Potentially useful resource for further research  i.e. finding common mistakes ◦ No need to keep thousands of papers  3 x 700 students x more that 4 papers per each student => huge paper towers ◦ Enable dislocated reviewers ◦ Enable dividing particular exam on 2 or more reviewers ◦ Avoid long queues of students just want to see their exams

5. Former attempt (few years ago) Students were writing code in labs under assistants supervision and uploaded final code Code was manually reviewed using HTML editor Pros: ◦ No problems with recognizing handwriting ◦ Code indentation was able to be solved if program had been syntactically correct Cons: ◦ Time consuming on both sides of examination process:  Small lab’s capacity  Many of students in the first year of study still write faster than they type ◦ Technology drawback:  Reviewing extremely time consuming using rudimental HTML editor  Limited functionality of reviewing process

6. New technology – new chance Silverlight ◦ Cross platform and cross browser ◦ Rich (and fast) user interface inside browser  2D graphics  Animations ◦ HTML DOM access from managed code ◦ Web services

7. How it works (1)

8. How it works (2) Main toolbar ◦ Browse assessments ◦ Browse questions ◦ Enter score ◦ Save remarks

9. How it works (3) Exam toolbar Browse exam papers (pages) Enter text, draw line and polyline Select remark Zoom Save remark template

10. Reviewers opinions Takes little bit longer than “normal” reviewing with paper and pencil Not suitable for small numbers of exams ◦ Time needed to get a grip on tool’s interface and mouse movement Opposite situation: more exams – more same errors ◦ Once written, comments of common errors can be just dragged and dropped on the exam Saves time for exams insights ◦ Still has to be hold, but last significantly less

11. Students opinions - questionnaire Question Total answers 1(%)2(%)3(%)4(%)5(%)-(%) How often do you attend exam insights (for all courses): 1. Almost always 2. Often (cca 75%) 3. 50% 4. Rarely (cca 25%) 5. Almost never 1336,028,2717,334,633,10,75 When and why do you attend exam insights for manually reviewed exams: 1. Never, because there is always a large number of other students 2. I attend just to see what I did wrong but I achieved expected score 3. I attend because I want to try to get some extra points but I know what I had done wrong and I have achieved expected score 4. I usually attend because I like to know what I had done wrong and my score is less than I expected 5. I will attend because I don't understand official answers and would like reviewer to explain me difference between my answers and official answers 13318,86,774,5158,76,025,26 With score from 1 to 5 rate your satisfaction after exam insight (1 - not satisfied, 5 - very satisfied) 1333,761233,827,84,5118,1 Do you attend exam insight for online reviewed exams: 1. Never, because I saw what I had done wrong and I achieved adequate score 2. I attend because I know what I had done wrong but I think I should achieve greater score 3. I attend because I don't understand reviewer's comments 4. I attend to try to get some extra points 5. I never attend exam insight 13357,116,55,264,5111,35,26 With score from 1 to 5 rate usefulness of online exam reviewing (1 - not useful, 5 - very useful) 1330,751,5 23,366,96,02

12. Very useful because it is now possible to see my own mistakes, and the reviewer comments, and thus reducing the number of students who go on insights which are less crowded and everything is quickly resolved. Most of us are also curious and eager, we want to see why we do not have the expected number of points and that is why online access to the exams is very useful to us. Also, I think we are in most cases too lazy to go to the insights to see our mistakes and in this way, from pure curiosity, we all (or at least most of us) will see where we made a mistake and what official answer should look like. In this way we also learn a thing or two. Just one question... WHY THIS THING WAS NOT INTRODUCED FOR THE ENTIRE FACULTY A LONG TIME AGO! : P Students opinions – comments (1)

13. This is an excellent idea and should be used on as many courses as possible. On the Web, we can see where we made a mistake and there is no need to go to exam insight, which is usually crowded. Also, thanks to this system, the reviewing criteria are much clearer and more uniform. [Otherwise, this is the only questionnaire that I filled because I believe that it really can achieve positive change.] Once again, every honour for this initiative and I hope that more students will complete the questionnaire, and that this method of assessment will be used for all students on this course, and in the (near) future on other courses. Online insights are a great thing, because it allows all of us direct access to our exams. Thus saving us time that would have otherwise been lost (often in vain), on the insights and waiting in line. Praise the idea, performance and assistants in a detailed and well performed job even though this is additional burden for them! Students opinions – comments (2)

14. Might be useful if there is an official communication channel that would allow online access (something different from the e-mail). It would be very helpful for fellow colleagues that are traveling to the faculty. Distance from the faculty is one of my reasons why I don’t attend the insights and consultation. Only 2 “negative” comments: ◦ Online insights are not useful to me because I don’t understand official exam solutions and I want assistant to explain me the difference between official and my solution. ◦ Some assistants are to harsh  Note: Probably was comparing his/her and friends exams Students opinions – comments (3)

15. Conclusion Students are very satisfied Additional attention needed ◦ Reviews are now public (assistants should have almost unique criteria) Reviewing lasts longer (depending on the number of exams) ◦ Possible improvements  Touch screen monitors  Tablet PC