1. Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU

2. Outline u Benefits of assessment embedded in SBT u Synthesis of several common ITS assessment methods –Terminology –Where does assessment fit into an ITS? –A synthesis

3. Why should step-based tutors (SBT) be better than multiple choice tests (MCT) for assessment? u Why is a 30 minute task (SBT) better than a 1 minute task (MCT)? –Authenticity, validity, prediction of valued tasks –“teaching to the test” becomes valuable u Why is seeing the steps better than seeing just the answer? –Assess problem solving strategy –Assess meta-cognitive strategy (e.g., use of hints)

4. Why is embedding assessment in daily homework or seatwork better than monthly in-class testing? u Frees up time for teaching u Can’t be absent on day of test u “doesn’t test well” and “only good at tests” are no longer excuses u No test anxiety u No cramming u Instructors can see changes in competence daily

5. Homework is unsupervised but testing is supervised. How does that affect assessment? u What if students refer to textbook, web, friends or instructor? –Sequestered problem solving is inauthentic –Assessment may improve if tutor can “see” such references u What about cheating? –Technology can thwart most copying –Would have to hire a friend to do almost all of one’s homework

6. Outline u Benefits of embedding assessments in SBTs u Synthesis of several common ITS assessment methods –Terminology –Where does assessment fit into an ITS? –A synthesis Next

7. Definition: Knowledge component u A piece of knowledge u Generic – can be applied in to multiple tasks u Any form: rule, principle, concept, fact, procedure… u Examples: –Two angles are called “supplementary” if…. –To buckle your seat belt, insert the … –The gravitational constant of earth, g, is 9.8 m/s 2.

8. Definition: Mastery of a knowledge component (KC) u No more training needed –E.g., The ITS need no longer pick problems that involve it u Typical signs of mastery –Applies the KC when should –Applies the KC only when should –Applies without referring to text, notes, … –Can explain the KC, when to apply, etc.

9. Main goal of assessment u Never have enough data –E.g., typically do not have student’s explanations u Must estimate P(mastery) for every knowledge component given only steps as data

10. Definition: The response pattern of a student’s step u How a step was accomplished by the student u E.g., 4 response patterns: (S = student; T = tutoring system) 1.S: Correct. 2.S: Error; T: Hint; S: Correct. 3.S: Help; T: Hint; S: Correct. 4.S: Help; T: Hint; S: Help; T: Hint; S: Help; T: Bottom out hint; S: Correct

11. Definition: The relevant knowledge components of a student’s step u The knowledge components involved in deriving the correct step –given the information available at the time the step was done u The knowledge the student should apply to generate this step u The topic of the hints (if any were given)

12. Outline u An introduction to a common type of ITS –Terminology –Three examples u Synthesis of several common ITS assessment methods –Terminology (KC, mastery, response pattern, KCs relevant to a step) –Where does assessment fit into an ITS? –A synthesis Next

13. The 3 major computations of intelligent tutoring systems P(mastery) for each knowledge component Problem to be solved All correct steps in all orders Response pattern for each student step Expert Assessor Helper

14. The expert’s computation u Expert can be humans or an expert system u Solve the problem in all acceptable ways u Record steps taken u Record knowledge components used at each step P(mastery) for each knowledge component Problem to be solved All correct steps in all orders Response patterns for each student step Expert Assessor Helper

15. The helper’s computation u When the student enters a step, match it to a correct step u Give feedback & hints as necessary u Record response pattern P(mastery) for each knowledge component Problem to be solved All correct steps in all orders Response patterns for each student step Expert Assessor Helper

16. The assessor’s computation u Given –Reponse patterns for each step taken by the student –Old P(mastery) for each knowledge component u Calculate –New P(mastery) P(mastery) for each knowledge component Problem to be solved All correct steps in all orders Response patterns for each student step Expert Assessor Helper

17. Outline u An introduction to a common type of ITS –Terminology –Three examples u Synthesis of several common ITS assessment methods –Terminology –Where does assessment fit into an ITS? –A synthesis Next

18. Some assessment methods Knowledge tracing (Corbett & Anderson) -- Only two response patterns: “S:Correct” & “other” -- Only one relevant knowledge component per step SMART (Shute) -- Multiple response patterns Andes (VanLehn et al.) -- Multiple knowledge components per step Today: A synthesis of the above DT tutor (Murray) -- Dynamic Bayesian networks

19. The problem: Given step responses & prior P(mastery), calculate posterior P(mastery) KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Step 1 S: error; T: Hint; S: Correct Step 2 S: Correct Step 3 S: Help; T: Hint; S: Help; T: Hint; S: Help; T: B.O. Hint; S: Correct.05.11.10.05.75.12.50.99.05 Prior P(mastery) for each knowledge component (KC) Posterior P(mastery) for each knowledge component

20. A dynamic Bayesian network represents that P(mastery) changes during steps Bayesian network KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Bayesian network Step 1 Step 2 P(mastery) of knowledge component 4 at time 1 P(mastery) of knowledge component 4 at time 2

21. Assume P(mastery) changes only for relevant knowledge components Bayesian network KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Bayesian network Step 1 Involves only KC2 and KC3 Step 2 Involves only KC1 and KC4

22. Knowledge tracing: Assumes just one relevant knowledge component per step Bayesian network KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Bayesian network Step 1 Involves only KC2 Step 2 Involves only KC4

23. Knowledge tracing: Network topology and conditional probability tables KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Step 1 Step 2 Step response KC2 mastered:YesNo S: Correct.92.10 Other.08.90 Was mastered: YesNo Mastered 1.47 Not mastered 0.53 P(Step response not correct | KC2 mastered) Called “slip parameter” P(Step response correct | KC2 not mastered) Called “guess parameter” P(Mastered | Was not mastered, relevant to step) Called “acquisition parameter”

24. Need to allow multiple relevant knowledge components per step KC1 KC3 KCn KC2 KC4 Step response KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Step 1 Step 2 Step response Use AND node to represent that the step response depends only on whether or not all relevant knowledge components are mastered

25. The student’s meta-cognitive strategy also affects the student’s step response KC1 KC3 KCn KC2 KC4 Step response KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Step 1 Step 2 Step response Meta-cognitive strategy

26. Conditional probability table for the first “Step response” node Meta-cognitive strategy LearnPerform Both KC2 and KC3 mastered? YesNoYesNo S: Correct.99.50.99.10 S: Error; T: Hint; S: Correct.01.35.00.01 S: Help;… T: Bottom out hint; S: Correct.00.01.85 All other response patterns.00.14.00.04 P(step response was “S:Correct” | Meta-cognitive strategy was “Learn”, Not all knowledge components mastered before step Strong tendency to abuse help when the meta-cognitive strategy is “Perform” and some relevant knowledge component is not mastered yet.

27. Need to assume new P(mastery) depends on the step response as well as the old P(mastery) KC1 KC3 KCn KC2 KC4 Step response KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Step 1 Step 2 Step response Meta-cognitive strategy

28. Conditional probability table for a “knowledge component” node Step response:S:Correct1 hintBottom out hintOther Mastered before?YesNoYesNoYesNoYesNo Mastered after step 1.0.50.95.30.90.01.55.05 Not mastered after step 0.0.50.05.70.10.99.45.95

29. How the network should be used u Given log file – all steps taken by the student u Construct the whole network – all steps in all problems u Enter prior P(mastery) on knowledge components at time 1 u Clamp all step responses u Update – a huge computation u Read out P(mastery) on knowledge components after last step.

30. How the network is actually used u Step 1 –Construct network –Enter prior P(mastery) –Clamp step 1 response –Update –Read out P(mastery) u Step 2 –Construct network –Enter prior P(mastery) –Clamp step 2 response –Update –Etc. KC1 KC3 KCn KC2 KC4 Step response KC1 KC3 KCn KC2 KC4 Step 1 Meta-cognitive strategy KC1 KC3 KCn KC2 KC4 KC1 KC3 KCn KC2 KC4 Step 2 Step response Meta-cognitive strategy

31. Future work Knowledge tracing SMARTAndesToday Design  Implementation  Calibration  Validity  Sensitivity 

32. Summary u An introduction to a common type of ITS –Steps replace answers as main student response u A synthesis of several Bayesian assessment methods –Many knowledge components –Probability of mastery of each knowledge component –One (or more) knowledge components per step –Can use step response patterns to infer mastery –Global traits e.g., meta-cognitive strategy