1. 1 Reference Model for Evaluating Intelligent Tutoring Systems Esma Aimeur, Claude Frasson Laboratoire HERON Informatique et recherche opérationnelle Université de Montréal

2. 2 Laboratoire HERON - TICE 2000 Introduction Goal of an ITS (Intelligent Tutoring System): produce the behaviour of an intelligent (competent) human tutor who can adapt his teaching to the learning rythm of the learner Ability to model and reason about domain knowledge, human thinking, learning processes, and teaching process building an ITS needs also to evaluate the system lack of evaluation methodology

3. 3 Laboratoire HERON - TICE 2000 Architecture of an ITS Student

4. 4 Laboratoire HERON - TICE 2000 Evaluation of ITS Evaluation can be formative or summative Summative: after the design process Formative: during the design (define and refine goals and methods) Evaluation techniques used in –ITS –Software engineering

5. 5 Laboratoire HERON - TICE 2000 Evaluation of some ITS SHERLOCK : experimental evaluation VCR Tutor : 4 versions compared Formative qualitative evaluation –iterative design –formative methods qualitative : characteristics of a situation quantitative : finding causes and consequences –case studies

6. 6 Laboratoire HERON - TICE 2000 Lester’s evaluation On animated pedagogical agents (Design a Plant) --- botanical and physiology 100 middle school students Important educational benefits : improved problem solving Better than less expressive animated agents

7. 7 Laboratoire HERON - TICE 2000 Littman- Soloway What is the educational impact on students ? What is the relationship between the architecture of an ITS and its behavior ? External evaluation of the student model (Proust) Internal evaluation : architecture, how ITS respond to input values

8. 8 Laboratoire HERON - TICE 2000 Van Lehn Teaching metacognitive skills to implement and evaluate an ITS (SE- Coach guide self-explanation) Empirical evaluation Fundamental questions –How the design tools are used efficiently ? –How the learner improve his knowledge ?

9. 9 Laboratoire HERON - TICE 2000 Software engineering techniques Product : Boehm’model –software doing what the user want it to do –use resources correctly –easy to learn –well designed, code –easily tested and maintained

10. 10 Laboratoire HERON - TICE 2000 HCI techniques Evaluating design –cognitive walthrough (how easy a system is to learn) –heuristic evaluation (visibility, consistency, flexibility, helps for the user) –model-based evaluation (GOMS) predict user performance

11. 11 Laboratoire HERON - TICE 2000 Evaluationg Implementation Empirical and experimental methods Observational methods (user completes a set of tasks) Query techniques –interviews of the users about their experience –questionnaires : questions fixed in advance

12. 12 Laboratoire HERON - TICE 2000 Learner model Layers : – cognitive –affective –inferential Learning time : speed of knowledge acquisition Tracability : of learner’s actions

13. 13 Laboratoire HERON - TICE 2000 Learner model Levels of knowledge –Based on works of Gagné (1985) –7 levels of knowledge –4 principal levels Novice Beginner Intermediate Expert

14. 14 Laboratoire HERON - TICE 2000  One-on-one  Co-learner  Learning Companion  Learning by Disturbing Learning strategies

15. 15 Laboratoire HERON - TICE 2000 Learning strategies Diversity Adaptability (switching to different strategies) Modification Memorization Feedback Reduce cognitive load

16. 16 Laboratoire HERON - TICE 2000 Curriculum Generalisation Consistency Knowledge articulation Reusability Navigability Maintenance

17. 17 Laboratoire HERON - TICE 2000 Interface Intuitive : learner should understand easily the functions to apply Interactivity : allowing the learner to be active in his learning environment Matching : using words, phrase and concepts

18. 18 Laboratoire HERON - TICE 2000 General Productivity rate : the most important factor. Time spent to produce one hour of ITS based course Learning outcomes : performance obtained by the learner after the session Ease of use of the tools

19. 19 Laboratoire HERON - TICE 2000 Conclusion Evaluation process is complex but needs to be realistic Develop first, evaluate later Evaluation criteria as a guideline for estimation

20. 20 Laboratoire HERON - TICE 2000 Merci de votre attention

21. 21 Laboratoire HERON - TICE 2000 Stratégies d ’apprentissage TuteurCo-apprenant CompagnonApprentissage par explication