1. Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems Kate Forbes-Riley, Diane Litman, Scott Silliman, Amruta Purandare University of Pittsburgh Pittsburgh, PA, USA

2. 2 Outline u Introduction u WOZ-TUT System u Experimental Design u Uncertainty Corpus Description u Uses of the Uncertainty Corpus

3. 3 Overview: Towards Affect-Adaptive Spoken Dialogue Systems u Automatic Detection: promising across affective states and applications, e.g. (Craig et al., 2006; Litman & Forbes-Riley, 2006; Lee & Narayanan, 2005; Vidrascu & Devillers, 2005; Batliner et al., 2003) u Larger goal is automatic adaptation, but results are sparser u More public affect-annotated corpora of human-computer dialogues could help, e.g. HUMAINE project u SYMPAFLY, AIBO (Batliner et al., 2004) (German) u Communicator (Walker et al., 2001; Ang et al., 2002) (English) u Uncertainty Corpus: u new complex domain: spoken dialogue tutoring u new affect annotation: student uncertainty

4. 4 Uncertainty Corpus Collection: WOZ-TUT System u WOZ-TUT: Adaptive Wizard of OZ Tutoring System u modified version of ITSPOKE (Litman & Silliman, 2004) u system responses based on combined uncertainty and correctness u human recognizes speech, annotates uncertainty and correctness u Why uncertainty? u Most frequent in ITSPOKE corpora (Forbes-Riley & Litman, 2007) u Most systems respond only to correctness, but literature suggests uncertain and incorrect answers signal learning impasses u What uncertainty adaptation? u Treating uncertain+correct answers as incorrect should provide additional knowledge to bridge impasse

5. 5 WOZ-TUT Screenshot

6. 6 Experimental Design u 3 Conditions: used parameterized WOZ-TUT dialogue manager u Experimental: treat all uncertain+correct turns as incorrect u First Control: ignore uncertainty (logged) u Second Control: ignore uncertainty (logged), but treated a percentage of random correct answers as incorrect

7. 7 u TUTOR: What will the velocity of the object be a second after that (where the initial velocity is 9.8m/s and the acceleration is 9.8m/s2)? u STUDENT: Nineteen point six meters per second?? [uncertain+correct] u TUTOR in First Control Condition moves on: Good. So at every point in time during the fall of the man and his keys, how do their velocities compare with each other? u TUTOR in Experimental Condition remediates: Okay. As we have seen, if a falling object has an acceleration of 9.8m/s2, its velocity changes by 9.8m/s every second. So if a second after it began falling its velocity is 9.8m/s, a second later its velocity will be 9.8m/s + 9.8m/s = 19.6m/s. So what will its velocity be a second after that? Corpus Excerpts

8. 8 Experimental Procedure u 60 subjects randomly assigned to 3 conditions (gender-balanced) u Native English speakers with no college physics u Procedure: 1) read background material, 2) took pretest, 3) worked training problem with WOZ- TUT, 4) took posttest, 5) worked isomorphic test problem with non-adaptive WOZ-TUT

9. 9 Corpus Description u 120 dialogues from 60 students (.ogg format) u 20 total hours of dialogue u Student turns manually transcribed, including disfluency and non-syntactic question annotation u Tutor turns and Wizard annotations in log files StudentTutor Total Turns21712531 Total Uncertain Turns796- Total Words13533111829 Average Words per Turn6.2344.20

10. 10 Student Answer Attributes u One-way ANOVAs showed no significant differences: u number of correct, uncertain, or uncertain+correct turns u number adapted-to turns (EXP vs CTRL2) Training ProblemEXPCTRL1CTRL2 Ave Turns20.6518.6019.75 Ave Correct Turns13.8012.5514.20 Ave Uncertain Turns9.958.6011.15 Ave Uncertain+Correct Turns4.753.756.10 Ave Adapted-To Turns4.7503.65 Ave Uncertain+Correct and Adapted-To Turns 100%0%36%

11. 11 Uses of the Uncertainty Corpus I Isomorphic Test ProblemEXPCTRL1CTRL2 Ave Turns16.5016.8016.25 Ave Correct Turns14.6014.3514.10 Ave Uncertain Turns3.303.153.65 u Compare student performance across conditions to isolate impact of uncertainty adaptation u No significant differences in learning u We are comparing dialogue-based metrics in the isomorphic test problem (Forbes-Riley, Litman and Rotaru, 2008) - Feedback confound identified and rectified in larger follow-on study

12. 12 Uses of the Uncertainty Corpus II u Resource for analyzing linguistic features of naturally- occurring user affect in human-computer dialogue u Models built from elicited emotions generally transfer poorly to naturally-occurring dialogue (Cowie and Cornelius, 2003; Batliner et al., 2003) u Uncertainty Corpus provides a new resource for modeling naturally-occurring dialogue u Large number of features in speech, transcript, log files

13. 13 Summary and Current Directions u The Uncertainty Corpus is a collection of tutorial dialogues between students and an adaptive Wizard-of-Oz spoken dialogue system u Corpus (speech, transcripts, uncertainty and correctness annotations) publicly available by request through the Pittsburgh Science of Learning Center: https://learnlab.web.cmu.edu/datashop/index.jsp u Follow-on experiments and corpora u Larger WOZ study just completed, with learning results! u Fully automated study to begin Fall 2008

14. 14 Questions? Further Information? web search: ITSPOKE or PSLC Thank You!