1. Evaluation of a Hybrid Self-improving Instructional Planner Jon A. Elorriaga and Isabel Fernández-Castro Computer Languages and Systems Dept. University of the Basque Country 649 Postakutxa, E-20080 Donostia. e-mail: jipelarj@si.ehu.es

2. Contents Introduction Our approach: HSIIP –Case-based Instructional Planner –How it works –Methodology of application Evaluation Conclusions Related and Future Work

3. Introduction Self-improving Vs. Adaptive ITSs –Self-improving ITSs: generalise the acquired knowledge and use it in future instructional sessions (Dillenbourg, 1989) Few self-improving systems ITSs can learn in each of its modules –Tutor Module –Student Model –Domain Module –Interface By using data –From the Student Model (ITS) –Directly from the student –From the teacher

4. Our Approach: HSIIP HSIIP: Hybrid Self-Improving Instructional Planner Objectives: –To improve the adaptation ability –To incorporate a learning ability to existing ITSs Focus: Tutor Module, Instructional Planning Learning Techniques: –Case-Based Reasoning –Learning from memorization –Statistical learning

5. The HSIIP Approach Proposal: To incorporate a CBIP into existing ITSs Aim: To enhance the ITS with learning ability Result: SIITS ITS+CBIP ===> SIITS

6. Case-Based Reasoning

7. Case-Based Instructional Planner

8. CBIP: Case Structure Application Context Instructional Plan Results of the application

9. CBIP: Detail of Instructional Plan Memory

10. Search Hierarchical Organisation of the IPM Exhaustive and Heuristic Search Heuristic function: Similarity function Similarity Threshold CBIP: Generation Component (1) Retrieval of Cases Matching Nearest neighbour matching Similarity function

11. Adaptation of cases Critic-Based Adaptation Production System Knowledge intensive task CBIP: Generation Component (2) CRITIC IF + THEN + Priority: Adaptation-degree:(0.. 1)

12. CBIP: Learning Component (1) Revision of Cases Evaluation items Trace of the learning session Two Dimensions: –Educational Beliefs of the ITS (Student Model) Beliefs of the Learner: Interaction –Computational Case Reuse level

13. CBIP: Learning Component (2) Revision of Cases Evaluation items: –Knowledge Acquisition Levels –Misconceptions –Student Beliefs about KAL –Student Beliefs about the session –Replanning –... Result Object –Elementary results –Collective results Normalisation of values (0.. 1) Statistical Learning Creation of new cases

14. CBIP: Learning Component (3) Storage of new Cases Find the appropriate GE –Adapted Search Algorithms Generalisation –On-line –Off-line –Thresholds

15. CBIP: Heuristic Assessment Component Heuristic Formulae Adaptable - Weights Assessed objects –Retrieved cases –Built Instructional Plan –Global result of the executed Instructional Plan Assessment Factors –Beliefs of the STI (SM) –Beliefs of the learner –Similarity –Reuse rating –Adaptation level –Influence level Assesses some Instructional Planning Objects

16. TUTOR MODULE Instructional Plan Memory DIDACTIC DISPATCHER Learning Component Generation Component Classical Instructional Planner DIDACTIC INSTRUCTOR Assessment Component Session Data Estimates Cases Instructional Plan Estimates Cases Estimate > Threshold SHIIP Working (1)

17. Generation Component Classical Instructional Planner DIDACTIC INSTRUCTOR TUTOR MODULE Session Data Estimates Cases NIL Estimates Cases Instructional Plan Estimate < Threshold Learning Component Assessment Component Instructional Plan Memory DIDACTIC DISPATCHER SHIIP Working (2)

18. HSIIP Working –Initially empty IPM –Training Phase –Co-operation Phase Kernel of CBIP –An object-oriented framework that facilitates the development of Case-Based Instructional Planners –Represent explicitly and separately the characteristics of the concrete ITS Methodology of Application –Procedure and Guidelines

19. HSIIP: Methodology of Application (1) Analysis of the ITS (levels, plan-items, attributes) ** Most important task ** Knowledge Engineering Adaptation of the framework –Representing the ITS related knowledge –Setting of the parameters: Thresholds and Weights –Construction of the adaptation module Integration of the CBIP Test

20. HSIIP: Methodology of Application (2) Analysis of the ITS (instructional planning) Level structure Items of each level Attributes –Useful for retrieving Matching Indexing –Useful for evaluating Student Model Strudent

21. HSIIP: Evaluation (1) Objective: Evaluate the performance of the HSIIP in terms of the changes in the student’s knowledge Design of the experiments: –Four classical instructional planners (CIP) –Their corresponding HSIIP –A Population of simulated students (4 groups) To test isolated modules To perform a significative number of experiments in the same conditions

22. HSIIP: Evaluation (2)

23. HSIIP: Evaluation (3)

24. Conclusions HSIIP: An hybrid approach to enhance ITSs with learning capabilities based on a Case-Based Instructional Planner. –Case-Based Learning –Statistical Learning –Learning from Memorization Sound performance (combines two planners). CBIP kernel: A framework for developing Case-Based Instructional Planners. –Generic Module for Instructional Planning –Adaptable Positive evaluation results Simulated student a useful tool for formative evaluation

25. Related and Further Work Related Work –Tool for interacting with the teacher (supervision of plans and results) –Data: System, Student, Teacher Further Work –Experimentation –More aspects taken into account in revision –Application to other planning problems

26. Result Object Object Result (is-a Case-component Result-object) result-history: result-average: Object Elementary-result (is-a Result-object) evaluation-item-list: learner: date: learner-estimate: [0..1] global-estimate:[0..1] Object Collective-result (is-a Result-object) direct-use-number: use-number: evaluation-item-list: learner-estimate: [0..1] global-estimate:[0..1] Object Evaluation-item (is-a Result-object) evaluation-attribute: final-value:[0..1] change:[0..1] learner-estimate:[0..1]

27. Elementary Result of a Subplan (ERS) ERS=Elementary Result of a Subplan ARS=Average Result of a Subplan WB T =Weighted Belief of the Tutor (applied to each individual feature) WB L =Weighted Belief of the Learner (applied to each individual feature) WOB L =Weighted Overall Belief of the Learner (applied to a subplan) AEIS=Average of the Evaluation Items of a Subplan NEI=Number of Evaluation Items NA=Number of Applications of the case EI j =Evaluation Item j S=Subplan W X =Weight related to factor X

28. Retrieved Case Estimate (RCE) RCE=Retrieved Case Estimate ARS=Average Result of a Subplan (see figure 6) RCAF=Retrieved Case Appropriateness Factor WOB L =Weighted Overall Belief of the Learner (applied to a case) WSF=Weighted Similarity Factor WRF=Weighted Reuse Factor C=Case S (C)=Subplan attached to the case C W X =Weight related to factor X

29. Created Plan Estimate (CPE) CPE=Created Plan Estimate NL=Number of Levels in the Instructional Plan CLE=Created Level Estimate PCN=Number of Primary Cases RCE=Retrieved Case Estimate (see figure 7) CPF =Created Plan Factor WTF=Weighted Stability Factor WIF=Weighted Importance Factor IP=Instructional Plan L, L i =Levels of the Instructional Plan C, C i =Cases W X =Weight related to factor X