1. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 1 Interoperability: future- proofing CAA Niall Sclater CETIS Manager, Educational Systems University of Strathclyde Heriot-Watt University 6th February, 2002

2. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 2 Why do I need to know about QTI? I’m authoring a bank of questions in my subject area I’m authoring a bank of questions in my subject area I’m involved in the selection of a CAA system for my institution I’m involved in the selection of a CAA system for my institution I have a general interest in online learning and assessment I have a general interest in online learning and assessment

3. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 3 The basics IMS QTI is a means for exchanging assessment information IMS QTI is a means for exchanging assessment information It uses XML It uses XML 2 key parts of the spec: 2 key parts of the spec: –Assessment Section Item –Results Report Also QTI lite Also QTI lite Can be implemented separately Can be implemented separately

4. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 4

5. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 5 Response types Multiple choice Multiple choice True false True false Multiple response Multiple response Image hot spot Image hot spot Fill in the blank Fill in the blank Select text Select text Slide Slide Drag object Drag object Drag target Drag target Order objects Order objects Match item Match item Connect the points Connect the points

6. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 6 The concept of rendering The same item can be “rendered” in different ways to allow for different The same item can be “rendered” in different ways to allow for different –delivery platforms –technical capabilities –participant challenges (eg visual, audio, physical etc)

7. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 7 Rendering example

8. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 8 Basic response & render types Response types Response types –logical identifier –xy coordinate –string –number –logical group Render types Render types –choice –hot spot –fill in blank –slider –‘object’

9. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 9 True false example

10. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 10 True false example XML This is a simple True/False multiple-choice example. The rendering is a standard radio button style. No response processing is incorporated. Paris is the Capital of France ? True False

11. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 11 Response processing … T 1 Yes, you are right …

12. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 12 Multiple choice example

13. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 13 Multiple choice example XML Which one of the listed standards committees is responsible for developing the token ring specification ? IEEE 802.3 IEEE 802.5 IEEE 802.6 IEEE 802.11 None of the above.

14. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 14 Response processing … A 1 Yes, you are right. …

15. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 15 Multiple response example

16. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 16 Image hot spot example

17. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 17 Fill in blank (string) example

18. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 18 Numerical example

19. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 19 Composite example

20. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 20 Results reporting Specific to a single participant Specific to a single participant 4 constructs 4 constructs –summary (eg max score, num attempts) –assessment –section –item Each construct contains a context which specifies details such as name of participant Each construct contains a context which specifies details such as name of participant

21. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 21 Where does QTI fit in with other specs? IMS Meta-data Specification IMS Meta-data Specification IMS Content Packaging Specification IMS Content Packaging Specification IMS Learner Information Packaging Specification IMS Learner Information Packaging Specification IMS Digital Repositories Interoperability Specification IMS Digital Repositories Interoperability Specification

22. Elect Drafts Scope IMS Spec Development Process IMS Members Technical Board Public Working Group Final Scope Base Document Public Draft Feedback Approval

23. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 23 What’s next with QTI? V1.2 currently being approved V1.2 currently being approved –selection and ordering of sections and items –scoring roll-up in sections and assessments (from question scores to test scores) –results reporting Question and test metadata Question and test metadata Harmonisation with other specs Harmonisation with other specs Integration with ADL SCORM Integration with ADL SCORM Implementations now needed Implementations now needed V1.3 early in 2002 V1.3 early in 2002

24. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 24 UK Interoperability Developments Strong representation in QTI working group Strong representation in QTI working group CETIS QTI SIG CETIS QTI SIG –meetings –online discussions –repository of open source tools –hosting workshops CAA Centre & SCAAN CAA Centre & SCAAN JISC MLE CAA Project JISC MLE CAA Project e3an e3an

25. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 25 Products which say they are implementing QTI E-Test 3, RIVA Technologies Inc E-Test 3, RIVA Technologies Inc Question Mark Perception Question Mark Perception Can studios Can studios Oracle i-learning Oracle i-learning Blackboard Blackboard WebCT WebCT Ultimate assessment Ultimate assessment E3AN E3AN

26. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 26 IMS and other initiatives IMS Europe IMS Europe SCORM SCORM Open Knowledge Initiative Open Knowledge Initiative Prometeus Prometeus Questionmark -> Blackboard & WebCT Questionmark -> Blackboard & WebCT

27. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 27 www.imsproject.org

28. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 28 QTI page

29. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 29 QTI: some problems For vendors For vendors –implementing QTI is costly –stops them from protecting their user base Vendors want simple specifications; users want broad, well-defined ones Vendors want simple specifications; users want broad, well-defined ones It is difficult to define specifications which do not have an effect on functionality It is difficult to define specifications which do not have an effect on functionality

30. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 30 Finally: why bother with QTI? Avoid lock in Avoid lock in –allow data to be moved in standard formats –allow one system to replace another Integrate systems from different vendors; allow choice of best assessment tool Integrate systems from different vendors; allow choice of best assessment tool Facilitate exchange of questions and tests between institutions, allow market to develop Facilitate exchange of questions and tests between institutions, allow market to develop Enable separation of content from presentation Enable separation of content from presentation Standard assessment records will facilitate student mobility Standard assessment records will facilitate student mobility

31. © 2001 By Default! A Free sample background from www.pptbackgrounds.fsnet.co.uk Slide 31 Further information www.sclater.comwww.imsproject.orgwww.cetis.ac.uk/assessment