1. HEA-ICS 7 th Annual Conf. 31-08-06 Laboratory Exams in First Programming Classes Quintin Cutts, University of Glasgow quintin@dcs.gla.ac.uk and the Disciplinary Commoners Dave Barnes, Kent; Pete Bibby, Bolton; Jim Bown, Abertay; Vicky Bush, Gloucestershire; Phil Campbell, LSBU; Sally Fincher, Kent; Stephan Jamieson, Durham; Tony Jenkins, Leeds; Michael Jones, Bournemouth; Dimitar Kazakov, York; Thomas Lancaster, UCE; Mark Ratcliffe, Aberystwyth; Monika Seisenberger, Swansea; Dermot Shinners-Kennedy, Limerick; Carol Wagstaff, Teesside; Linda White, Sunderland; Chris Whyley, Swansea. http://www.cs.kent.ac.uk/~saf/dc

2. HEA-ICS 7 th Annual Conf. 31-08-06 Outline Motivation for laboratory examsMotivation for laboratory exams Seven Commons modelsSeven Commons models Comparing the modelsComparing the models

3. HEA-ICS 7 th Annual Conf. 31-08-06 Open assessed coursework Solution created bySolution created by –student –another student –last year’s student –book author –web author –contract coder Ensure ‘student’ by –trust their good nature –warn of severe penalties –manual code search –automatic code search –horrendous institutional disciplinary route –asking questions about understanding (almost)

4. HEA-ICS 7 th Annual Conf. 31-08-06 Coursework in an ITP course… Primary skill development mechanismPrimary skill development mechanism Should skill development be mixed with skill assessment anyway?Should skill development be mixed with skill assessment anyway? –No!! Increased stress etc. HenceHence –use a more formalised assessment of programming skills

5. HEA-ICS 7 th Annual Conf. 31-08-06 define: laboratory exam Strictly controlled set of rulesStrictly controlled set of rules PossibilitiesPossibilities –time limit –unseen questions –limited access to other materials –limited access to other people Rules upheld by strict invigilationRules upheld by strict invigilation

6. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics Studs Max students exam tried with Freq No. of examinations in course Sess No. of sittings required for each exam Vers No. of versions of paper required for each exam Len Length of exam Open? Is it an open book exam? Seen? Is the paper/question seen in advance? Indivi Can the student work with/confer with others? Ntwkd Are the laboratory machines networked?

7. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics AberStuds120 Freq4/yr Sess1 Vers1 Len120 Open?y Seen?n Indiviy Ntwkdy Smaller tail and std dev End of course MCQ answered much better Subm & marking on-line maximises feedback Tests comprehension not memory

8. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics AberUCEStuds120160 Freq4/yr2/smr Sess12smdy Vers16 Len12090 Open?yy Seen?nn Indiviyy Ntwkdyy 1 st year of use Low ave. mark, but many students absent this time Provision of early f’back on progress 2 nd test based on c’work Security paramount

9. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics AberUCEDurStuds12016080 Freq4/yr2/smr2/yr Sess12smdy5 Vers161sc+5 Len12090110 Open?yyy Seen?nnn Indiviyy tut avl Ntwkdyyy-m/e 10% used help facility, reduced stress (no tears) Allows students stuck at one point to succeed elsewhere

10. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics AberUCEDurGlaStuds12016080480 Freq4/yr2/smr2/yr2/yr Sess12smdy524 Vers161sc+51 Len12090110110 Open?yyylimited Seen?nnny Indiviyy tut avl y Ntwkdyyy-m/en Some rote learning Assesses coding, testing & debugging only One version of exam only for multiple sessions

11. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics AberUCEDurGlaLeeStuds12016080480200 Freq4/yr2/smr2/yr2/yr1/crs Sess12smdy5241 Vers161sc+511 Len12090110110~180 Open?yyylimitedy Seen?nnnyn Indiviyy tut avl yy Ntwkdyyy-m/enlimited Results match performance on other assessments Matches intended learning outcomes with assessment

12. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics AberUCEDurGlaLeeSunStuds12016080480200100 Freq4/yr2/smr2/yr2/yr1/crs3/yr Sess12smdy52414 Vers161sc+5116sc+3 Len12090110110~18075 Open?yyylimitedyy Seen?nnnynn Indiviyy tut avl yyy Ntwkdyyy-m/enlimitedy Students appreciate the validity of the exam Rapid feedback Heavy to set up, but scenario reuse ameliorates

13. HEA-ICS 7 th Annual Conf. 31-08-06 Vital Statistics AberUCEDurGlaLeeSunSwaStuds120160804802001005 Freq4/yr2/smr2/yr2/yr1/crs3/yr 1 rsit Sess12smdy524141 Vers161sc+5116sc+31 Len12090110110~18075120 Open?yyylimitedyyy Seen?nnnynnn Indiviyy tut avl yyyy Ntwkdyyy-m/enlimitedyy The reality of a practical exam wakes some students up to necessary study levels Hence, significant motivational effect

14. HEA-ICS 7 th Annual Conf. 31-08-06 Three categories Single session, single unseen paperSingle session, single unseen paper –Aberystwyth, Leeds, Swansea Multiple sessions, multiple unseen papersMultiple sessions, multiple unseen papers –UCE, Durham, Sunderland –Maintains fairness and reliability Multiple sessions, single seen paperMultiple sessions, single seen paper –Glasgow –Reduced assessment coverage

15. HEA-ICS 7 th Annual Conf. 31-08-06 Reliability No cheating/communicationNo cheating/communication –invigilators –limited/no network –different versions of exams Same complexity across all versions?Same complexity across all versions? –range of problems, all using same concepts –single overarching scenario, multiple problems drawn from it

16. HEA-ICS 7 th Annual Conf. 31-08-06 Validity Does assessment match real world activity?Does assessment match real world activity? access to books/web/earlier solutions goodaccess to books/web/earlier solutions good access to human assistance (at a price)access to human assistance (at a price) Do assessment exercises match skill development exercises?Do assessment exercises match skill development exercises?

17. HEA-ICS 7 th Annual Conf. 31-08-06 Scalability Large class sizes not a problem now, but…Large class sizes not a problem now, but… –we can hope Not all these models scaleNot all these models scale

18. HEA-ICS 7 th Annual Conf. 31-08-06 Conclusions Support for lab examsSupport for lab exams –Improved attitudes and study habits –Reliable feedback on progress –Assessing full range of programming skills –Testing comprehension not memory –Separating skill development from skill assessment –Reducing exam stress Further workFurther work –Analysis of the problems used –How were the students’ solutions evaluated? –Comparison of exam performance against other assessments Disciplinary CommonsDisciplinary Commons