1. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 1 CAA Pedagogy, standards and interoperability LTSN 27 th Jan 2003 Niall Sclater Manager, Learning Services University of Strathclyde

2. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 2 What is CAA? The use of computer-based tools for: the creation, storage and delivery of assessments to candidates the capture, marking, storage and analysis of their responses the collation, return and analysis of results

3. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 3 Types of CAA Diagnostic Formative Summative

4. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 4 Why use CAA? Improving learning Objective marking Reducing marking load Automated results collation, analysis and return Improving access Reducing opportunities for plagiarism

5. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 5 Costs Insitutional VLE? Obtaining or creating questions and tests Staff development – –Management – –Teaching staff – –Support staff Technical support Hardware

6. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 6 Authentication and security Knowledge Ownership Characteristic Impersonation without candidate’s permission Impersonation with permission –Use real time questioning –Get to know writing style Restricting access to other resources Restricting access to other resources

7. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 7 Selecting a CAA system Is your VLE adequate? Quality of vendor Navigation facilities Usability Help facilities Security features Candidates’ responses Interoperability Accessibility Accessibility

8. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 8 MCQ Name the capital of Saudi Arabia. Mecca Medina Jiddah Riyadh

9. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 9 True/false Yes/no Sydney is the capital of Australia True False Is Sydney the capital of Australia? Yes No

10. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 10 Short answer A body will continue in its state of rest or uniform velocity unless acted upon by an unbalanced force. What is this law known as?

11. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 11 Fill in the blank / gap fill Complete the following: Now is the of our discontent. Toledo es a setenta kilómetros de Madrid. está

12. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 12 Fill in the blank / gap fill Toledo es a setenta kilómetros de Madrid. está Justify your choice of es or está in the above sentence.

13. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 13 Matching Associate the following plants with the most appropriate type: lysichiton americanus evergreen perennial cornus alba deciduous shrub phormium tenax marginal aquatic perennial verbascum bombyciferumbiennial or short-lived evergreen perennial

14. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 14 Hot spot Click the stigma of this flower:

15. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 15 Sequencing Put the following colours of the rainbow into the correct order. Orange Yellow Indigo Blue Red

16. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 16 Adding complexity to MCQs An inelastic collision takes place between a moving object and a stationary object. Both objects have the same mass. In this situation, which of the following quantities is/are conserved for this system? 1 Momentum 2 Total energy 3 Kinetic energy 1 and 2 2 and 3 1, 2 and 3

17. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 17 Metadata Contact details of creator Date of creation Description Level

18. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 18 Automated text marking Keyword analysis Natural language processing

19. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 19 Scoring strategies Make candidates aware of strategy MCQs usually allocate one mark for a correct response Negative marking Multiple response scoring

20. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 20 Holding invigilated test sessions Develop evaluation procedures Trial system Register candidates for system Give candidates practice Give candidates prior information Prepare room and equipment Ensure adequate staffing Ensure security Specify technical failure procedures Specify emergency procedures Authenticate candidates

21. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 21 Why do I need to know about QTI? I’m authoring a bank of questions in my subject area I’m authoring a bank of questions in my subject area I’m involved in the selection of a CAA system for my institution I’m involved in the selection of a CAA system for my institution I have a general interest in online learning and assessment I have a general interest in online learning and assessment

22. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 22 The basics IMS QTI is a means for exchanging assessment information IMS QTI is a means for exchanging assessment information It uses XML It uses XML 2 key parts of the spec: 2 key parts of the spec: –Assessment Section Item –Results Report Also QTI lite Also QTI lite Can be implemented separately Can be implemented separately

23. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 23 Rendering example

24. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 24 True false example

25. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 25 True false example XML This is a simple True/False multiple-choice example. The rendering is a standard radio button style. No response processing is incorporated. Paris is the Capital of France ? True False

26. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 26 Where does QTI fit in with other specs? IMS Meta-data Specification IMS Meta-data Specification IMS Content Packaging Specification IMS Content Packaging Specification IMS Learner Information Packaging Specification IMS Learner Information Packaging Specification IMS Digital Repositories Interoperability Specification IMS Digital Repositories Interoperability Specification

27. Elect Drafts Scope IMS Spec Development Process IMS Members Technical Board Public Working Group Final Scope Base Document Public Draft Feedback Approval

28. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 28 Products which say they are implementing QTI E-Test 3, RIVA Technologies Inc E-Test 3, RIVA Technologies Inc Question Mark Perception Question Mark Perception Can studios Can studios Oracle i-learning Oracle i-learning Blackboard Blackboard WebCT WebCT Ultimate assessment Ultimate assessment E3AN E3AN TOIA TOIA

29. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 29 QTI: some problems For vendors For vendors –implementing QTI is costly –stops them from protecting their user base Vendors want simple specifications; users want broad, well-defined ones Vendors want simple specifications; users want broad, well-defined ones It is difficult to define specifications which do not have an effect on functionality It is difficult to define specifications which do not have an effect on functionality

30. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 30 Finally: why bother with QTI? Avoid lock in Avoid lock in –allow data to be moved in standard formats –allow one system to replace another Integrate systems from different vendors; allow choice of best assessment tool Integrate systems from different vendors; allow choice of best assessment tool Facilitate exchange of questions and tests between institutions, allow market to develop Facilitate exchange of questions and tests between institutions, allow market to develop Enable separation of content from presentation Enable separation of content from presentation Standard assessment records will facilitate student mobility Standard assessment records will facilitate student mobility

31. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 31 Further information BS7988 “A code of practice in the use of IT for the delivery of assessments” www.caacentre.ac.uk www.caaconference.com www.toia.ac.uk www.sclater.com