The Behavior of Tutoring Systems Kurt VanLehn University of Pittsburgh

Underlying Claim Behaviors of tutoring systems are quite similar, despite difference on Task domain User interface Software structure Knowledge base etc.

6 Illustrative Tutoring Systems Algebra Cognitive Tutor Andes AutoTutor Sherlock SQL-Tutor Steve

Learning Event & Knowledge Component Empirical tech. for establishing KC Modeling transfer Predicting learning curve Determining grain size of KC Instructive objective of designers “the designers’ decisions about what learning events should be detected and advised by the tutor determine what knowledge components are included in the tutor’s representation of knowledge”

Two loops Outer loop Inner loop select a task for student that will help to learn Obtain a task set to select from Inner loop select the right step and help learning

I like this… “a tutoring system does not have to replace a teacher or run an after-school remediation session. Its role in the student’s learning ecology can be anything—a smart piece of paper; an encouraging coach; a stimulating peer, etc” (p6-7)

The Outer Loop Task selection Student control Homework helper e.g. Andes Fixed sequence (random sequence) e.g. AutoTutor, Assistments linear order curriculum Mastery learning Self-paced, e.g. Sherlock Macro-adaptation Knowledge component (KC) based. e.g. Cognitive Tutor Systems modeling also incorrect KC (e.g.??) Systems also represent learning styles and preferences (e.g. different pedagogical agents??)

The Inner Loop The goal is to help student to learn from a task Common services: Minimal feedback Error-specific feedback Hints on the next step. Assessment of knowledge. Review of the solution.

Minimal Feedback Categories When to give Correct, incorrect, non-optimal, unrecognizable When to give Immediate Delayed Demand Feedback policy can be a function of student competence (Razzaq & Heffernan, 2007) Delayed feedback for high performing students Immediate one for less competent students

Next-step Hints To avoid repeated guessing and frustration When Which Based on student data (DT Tutor) Hint on demand (Andes, Sherlock) Hint abuse/refusal “Proactive” help (DT Tutor, Steve) Which Following student’s plan How Single KC: Point, Teach, Bottom-out Multi KC: bottom-out, refer to history; Scaffolding? Contingent tutoring on student recent performance

Error-specific Feedback Analyzing an incorrect step to determine the incorrect learning event and to prevent future occurrence To help self-debugging How Minimal feedback + a sequence of hints When Immediate; when same error happens again; on demand etc. Step analyzer seems a magic, black box to me. It seems it is impossible, at least very hard, to detect what KC a student applied in case of an incorrect answer.

Assessing Knowledge What kind of assessment is required? Assessments are decision aids Coarse-grained assessment Combination of different measures Fine-grained assessment Based on learning events on steps “Mastery” really means the probability that a knowledge component will be applied when it should be applied. Assignment credit & blame problem Counting learning events Counting failures Other issues Here comes the step analyzer again: given a correct student step, it returns the set of knowledge components that were used to derive the step. It is kind of easy to understand “blame” is a difficult problem. But why credit too? Is it possible for the tutor to credit different KCs in case of a correct step? E.g. given an external and an internal angles’ degree, asking for another one

Reviewing the Whole Solution Real-time task domains Non-real-time task domains Pedagogical reason: delayed feedback Plans and policies Learn from human tutor (bottom-hint during performance, reasoning during reviewing) Other issues

Final Remarks The dream of having an ITS for every student in every course is within our grasp. 

Q? Is Assistment single-loop or double-loop? Knowledge component can be incorrect (p3)? Is it saying the KC itself is wrong or not applied at proper step? #steps per task is not fixed, varying by student; Steps <-> learning events (x = 18.46) Mastery learning vs. Macroadptation? Student model A pointer to next task Attribute-value pairs Student data, such as standardized test score, major, etc Step analyzer and generator