1. Using QuestionMark Perception for summative on-line assessment Marija Cubric May 2006

2. Context  Level: UG1  Module: Introduction to programming Practical “hands-on” subject that requires some amount of thinking Cannot be learnt from books only  Number of students: ~100 Mix of BISF, AMIS, BS, MB,..  Last year failure rate: 56%  Main Issues: Low attendance (>50% never showed up!) Low motivation (does not count towards the final grade) Not enough contact time (1hpw tutorial – not sufficient)

3. Solution (first iteration )  Attendance Weekly on-line test (20% of final grade)  Motivation Publishing “league tables” (top 10, 10 most improved etc)  Contact time (tutorials) Increased from 1hpw to 2hpw

4. Theory  Objective questions (MCQ/MRQ) can be designed to test all levels of Bloom’s educational competencies: Knowledge Comprehension Application Analysis Evaluation  Objective questions can be used in summative/ not only formative assessment, but need to be mixed up with other forms of assessment Creativity, design and development, non-cognitive skills, writing style etc – cannot be tested with objective questions.  Receiving immediate feedback is important part of any constructivism*-based learning cycle such as Kolb’s learning cycle (*Constructivism - knowledge construction is social process that occurs through collaboration with others)  Amanda Jeffreys & Andy Oliver (BLUE team) run w/shops on how to design objective tests

5. Tool  QuestionMark Perception 3 rd party sw, licensed to UH (trial) Not part of StudyNet http://www.questionmark.com/uk/hom e.htm http://www.questionmark.com/uk/hom e.htm  Tool administrator: Angus Shaw To setup account for you To create participant groups (module code)

6. Features  Roles: administrator or participant  Administrator functions:  Question management (C/E/D)  Assessment management  Schedule management  Group management  Report management  Participants functions:  Start & submit test  Review test results

7. Login  http://www.perception.herts.ac.uk/ em/login.asp

8. Question database

9. Types of questions

10. Configuring assessment

11. Configuring schedule

12. Report example (full score list report)  Result index  Assessment ID  Assessment name  Assessment author  Assessment last modified  Participant  Participant group  Participant details  Hostname  IP address  Status  Date/time started  Date/time finished  Total score  Maximum score  Percentage score  Time taken  Topic percentage score  Topic result

13. Types of reports

14. Test example  Login to: http://www.perception.herts.ac.uk/q/p erception.dll http://www.perception.herts.ac.uk/q/p erception.dll

15. Feedback  Not immediate (except the score)  Run through the answers immediatelly after the test (extra 5- 10 minutes)

16. Results (tutor’s perception)  Increased attendance: Average tutorial attendance: 75%  Increased motivation: “League tables” very popular!  Better results? Yet to be seen Expecting raise of 10% in pass rate  “Liminality” (Glynis Cousin) “mimicking” rather than “thinking” (“cut&paste” strategy)

17. Results (students’ perception)  Data: anonymous questionnaire (74 respondents)

18. Results (usability)

19. Workload  Creating questions (10min/q)  Creating assessments & schedules (30min/w)  Importing reports to Excell & generating league tables (30min/w)  Beware: extra time >4hpw (incl. 20 new questions)

20. Questions?  Can questions be parameterized & parameters within question randomized NO  Can questions database be exported Questions-yes, not sure how to expeort answers???  Can it be used outside of the campus YES (with usual proxy settings for remote access)

21. URLs  Admin login: http://www.perception.herts.ac.uk/em/ login.asp http://www.perception.herts.ac.uk/em/ login.asp  Participant login: http://www.perception.herts.ac.uk/q/p erception.dll http://www.perception.herts.ac.uk/q/p erception.dll  More on QuestionMark Perception: http://www.questionmark.com/uk/hom e.htm http://www.questionmark.com/uk/hom e.htm

22. Top 10 – see StudyNet for full score table

23. Top 10 (most improved) – see StudyNet for full score table