Kate’s Ongoing Work on Uncertainty Adaptation in ITSPOKE

Outline  PSLC Project  ALT Project  Differences from PSLC Project  NAACL-HLT 2007 Study

PSLC Project: Background  Does Treating Student Uncertainty as a Learning Impasse Improve Learning in Spoken Dialogue Tutoring?  Learning Impasse (VanLehn, Siler & Murray, 2003):  “occurs when a student realizes that he or she lacks a complete understanding of a specific piece of knowledge [...] when a student gets stuck, detects an error, or does an action correctly but expresses uncertainty about it”  uncertain answers and wrong answers

PSLC Project: Background  Does Treating Student Uncertainty as a Learning Impasse Improve Learning in Spoken Dialogue Tutoring?  Uncertainty relates to learning (Craig et al. 2004):  positively correlates with learning in AutoTutor  Hypothesis: uncertainty can be accompanied by deliberation/inquiry aimed at increasing understanding

PSLC Project: Hypothesis  Does Treating Student Uncertainty as a Learning Impasse Improve Learning in Spoken Dialogue Tutoring?  uncertain: opportunity to learn something  incorrect: opportunity to learn something  Most tutoring systems respond only to correctness  Our hypothesis: treating uncertainty as a learning impasse will increase learning

PSLC Project: Experimental Condition  Keep ITSPOKE response to incorrect answers:  Simple questions: give answer and move on  Complex questions: remediation dialogue with more questions to increase understanding, then move on  ADD ITSPOKE response to uncertain answers:  Respond as if answer were incorrect  Keep ITSPOKE response to other correct answers:  move on to next question

PSLC Project: Control Condition #1  Keep ITSPOKE response to incorrect answers  Keep ITSPOKE response to correct answers  NO ITSPOKE response to uncertainty

PSLC Project: Control Condition #2  Keep ITSPOKE response to incorrect answers  Keep ITSPOKE response to 85% correct answers  CHANGE ITSPOKE response to a random 15% of correct answers  Respond as if answer were incorrect  15% of student answers are uncertain (HC-wade)  control for extra tutoring in experimental condition; should be less effective than targeting uncertainty

PSLC Project: Experimental Procedure  Read background material (reduced)  Take pretest (reduced)  Work 1 st problem with ITSPOKE (#58)  Take posttest (reduced)  Tests modified to be “fill in blank” (no chance correct)  May not see significant learning gains after 1 problem  Work isomorphic problem with ITSPOKE (#59)  Experimental condition may have more correct answers  20 subjects/condition (low pretest only); 2 hours

PSLC Project: Isomorphic Problems  Problem #58: Suppose a man is in a free-falling elevator and is holding his keys motionless right in front of his face. He then lets go. What will be the position of the keys relative to the man's face as time passes? Explain.  Problem #59: A professor is sitting in his armchair holding his spectacles motionless in front of his face, in order to inspect how well he just cleaned them. All of a sudden, an earthquake causes his office floor to collapse. The professor is startled when he finds that both he and his armchair are in free-fall, and he drops his spectacles. What will be the position of the spectacles relative to the professor's face as time passes? Explain.

PSLC Project: Implementation  WOZ version of ITSPOKE (“ideal” system conditions)  Speech input/output  No essay (everyone gets walk-through)  Human Wizard  Perform speech recognition and NLU (select anticipated response components)  Label uncertainty for each condition (checkbox; data)  Remaining Issues:  students hit stop/start buttons when speak?  random 15% of correct turns (non-uncertain?)  need piloters so we can check logs etc ;)

PSLC Project DEMO

ALT Project: Differences from PSLC  Adapting to Student Uncertainty over and above Correctness in a Spoken Tutoring Dialogue System  Experiment 1: 4 conditions, WOZ, all 5 problems  Adaptation 1: treat uncertainty as learning impasse (baseline/obvious/least-change strategy)  Adaptation 2: new system responses to uncertain + correct and uncertain + incorrect student answers, based (in part) on analysis of human tutor responses  Hypothesis: significantly improved learning in adaptive conditions as compared to controls (same 2 as PSLC)

ALT Project: Differences from PSLC  Adapting to Student Uncertainty over and above Correctness in a Spoken Tutoring Dialogue System  Experiment 2: 3 conditions, ITSPOKE, all 5 problems  Adaptation: “best-performing” adaptation from Exp1  via learning, dialogue efficiency, survey, etc  automatic uncertainty detection  automatic speech recognition/NLU  Hypothesis: significantly improved learning in adaptive condition as compared to controls (same 2), but less improvement as compared to WOZ

ALT Project: Adaptation 2  Adaptation 2: new system responses to uncertain + correct and uncertain + incorrect based (in part) on human tutor responses  Pon-Barry et al., 2006: implemented human tutor responses, but didn't find increased learning in experimental condition  uncertain + correct: paraphrase answer  corpus#1: 4/5 and 0/34 other corrects  corpus#2: 10% and 1% other corrects  uncertain + incorrect: refer to past dialogue  corpus#1: 3/10 (point out misconception, follow incorrect reasoning, state answer) and 0/7 other corrects  corpus#2: 30% and 5% other incorrects  Human tutors use multiple strategies depending on context

NAACL07: Towards Adaptation 2  Exploring Context-Dependent Affect Adaptations in Spoken Dialogue Systems (with Mihai, Diane, Joel)  Proposal: develop multiple adaptations for affective state, whose use depends on prior context in which state occurs  Motivations:  Human tutors seem to do this (Pon-Barry et al., 2006)  Prior work largely context-independent and no improved learning (e.g., Aist et al., 2003, Pon-Barry et al., 2006)  Call Center: apologize after user frustration, except if multiple frustrated turns in row give to human operator

NAACL07: Towards Adaptation 2  Exploring Context-Dependent Affect Adaptations in Spoken Dialogue Systems (with Mihai, Diane, Joel)  Questions: in what contexts to adapt, and what strategy to use in each context? (not clear for uncertainty in tutoring)  Our Empirical Approach:  Apply Chi 2 test is system corpora to find contexts significantly associated with increased/decreased affect  Formulate hypotheses explaining association  Base context-dependent adaptations on hypotheses  Refine adaptations via analysis of human tutor responses

NAACL07: Towards Adaptation 2  Apply approach in ITSPOKE  Targeted affect: uncertain+correct, uncertain+incorrect  Contexts are question properties: Question Act, Discourse Structure Depth/Transition, Goal Repetitions  “Tutor Question – Student Answer”, but 275 questions  Results:  multiple contexts significantly associated with increased or decreased uncertainty states  hypotheses explain association and suggest adaptations  FW: Refine via analysis of human tutor response in contexts

NAACL07: Annotated Example SYS4: Good. Does the packet have an acceleration? If yes please specify its direction. [SAQ, Advance, D1, GR0] STUDENT4: horizontally forward yes [other+incorrect] SYS5: Well... To find the packet's acceleration we will use Newton's second law. What is the definition of this law? [LAQ, Push, D2, GR0] STUDENT5: an object in motion tends to stay in motion until it's act by an outside force [uncertain+incorrect] SYS6: Well... [...] What can we determine about the acceleration of the packet given that we know the net force acting on it is not zero and points straight down? [DAQ, Advance, D2, GR0] STUDENT6: I don't know [uncertain+incorrect] SYS7: Well... [...] Does the packet have an acceleration? If yes please specify its direction. [SAQ, PopUp, D1, GR1]

Question Act Dependencies (p<.05)  LAQ/DAQ are hard questions: definition/deep reasoning  Adaptation: reformulate as ``fill in blank” SAQ to work through uncertainty by (also) answering simpler version correctly

Question Act Dependencies (p<.05)  SAQ are easier “fill in blank” questions: uncertain students may be having difficulty with path to solution  Adaptation: remind how SAQ relates to overall solution (and/or monitor frequency of uncertainty after SAQs)

Depth Dependencies (p<.05)  Deeper depths correspond to deeper knowledge gaps about topics in solution (+otherI, -otherC): uncertainty may concern perceived lack of cohesion between sub- topic and larger topic/solution  Adaptation: show how sub-topic fits in to overall solution

Depth Dependencies (p<.05)  D1 on direct path to solution (+otherC, -otherI): uncertainty may relate to difficulty understanding necessary steps on path  Adaptation: reminder of steps on path, and/or monitor frequency of uncertainty at D1: occasional may be constructive (VanLehn et al., 2003) but frequent may be destructive and require intervention

Transition Dependencies (p<.05)  Pushes correspond to deeper knowledge gaps (+otherI): uncertainty may concern perceived lack of cohesion between sub-topic and larger topic/solution (like D<1)  Adaptation: show how sub-topic fits in to overall solution

Transition Dependencies (p<.05)  After PopUpAdv, uncertainty may relate to losing track of original question (not +otherI); after PopUp, to goal of sub-topic question(s)  Adaptation: remind how PopUpAdv relates to original question; clarify PopUp's subtopic goal

Transition Dependencies (p<.05)  Uncertainty may relate to current topic/depth(-otherI)  Adaptation: can be based on other contexts simultaneously monitored (e.g., Depth, QACT)

Transition Dependencies (p<.05)  SameGoal associated with increased incorrect overall; RPT with decreased correct: if student repeats answer, uncertainty may relate to original answer  (Rotaru and Litman, 2006): RPTs should be removed

Goal Repetition Dependencies (p<.05)  Uncertainty may be increased simply because students haven't seen question before (not +otherC); some may be constructive  Adaptation: can be based on continued uncertainty at GR0 (monitor frequency like after SAQs, D0 )

Goal Repetition Dependencies (p<.05)  Most students may remember answer from last time it was asked (no uncertainty) (not +otherC)  Adaptation: remind students about last time question asked (akin to Pon-Barry et al., 2006)

Conclusions & Future Work  Approach shows dependencies between uncertainty states and contexts, suggests hypotheses leading to adaptations  Will refine adaptations by analysing human tutor responses to uncertainty states in these contexts  Will study other automatically-monitorable contexts:  Topic Repetition: track similar + identical questions about topics, within and across 5 SYS problems (e.g. gravity, freefall); topic-dependent adaptations can be derived if uncertainty relates to specific topics  Will develop Combined/Precedence rules when multiple contexts trigger context-dependent adaptations