1. Towards Emotion Prediction in Spoken Tutoring Dialogues
Diane Litman, Kate Forbes, and Scott Silliman
Learning Research and Development Center
and Computer Science Department
University of Pittsburgh
Pittsburgh, PA USA

2. Outline
Introduction
System and Corpora
Pilot Study
Summary

3. Motivation
Human tutors listen to both “what” and “how” (e.g. “confident” vs. “uncertain”)
Speech supplies acoustic-prosodic information about user state; some spoken dialogue applications already handle “problem” dialogues specially (Ang et al. 2002, Batliner et al. 2003, Litman et al. 2001)
 Can effectiveness of computer dialogue tutors increase by detecting/adapting to student emotional state (Evens 2001)

4. ITSPOKE: Intelligent Tutoring SPOKEn Dialogue System
“Back-end” is (text-based) Why2-Atlas intelligent tutoring dialogue system (VanLehn et al., 2002)
Speech input via Sphinx2 speech recognizer
Speech output via Festival text-to-speech synthesizer

5. ITSPOKE Screen Shot

6. Parallel Human-Human Corpus
Target size
- 20 subjects
- up to 10 dialogues per subject
Size on 5/20/03
- 10 subjects
- 86 dialogues
- 62 dialogues transcribed
manually segmented student turns

7. Human-Human Corpus Transcription and Annotation

8. Annotating Emotion 14 transcribed dialogues (n=553 student turns)
Each student turn was annotated (intuition of 1 coder) with one of 3 general categories:
-negative (e.g. ‘uncertain’ or ‘frustrated’): n=141
-positive (e.g. ‘confident’ or ‘certain’): n=167
-neutral/indeterminate: n=248
KVL: first bullet is confusing
KVL: You could just put the numbers after the italic bullets above; clearer

9. Example Annotated Excerpt
…6.5 minutes after essay…
Tutor: Now this law that force is equal to mass times acceleration, what's this law called? This is uh since this it is a very important basic uh fact uh it is it is a law of physics. Um you have you have read it in the background material. Can you recall it?
Student: Um no it was one of Newton's laws but I don't- remember which one. (laugh) (EMOTION = NEGATIVE)
Tutor: Right, right- That- is Newton's second law of motion.
Student: he I- Ok, because I remember one, two, and three, but I didn't know if there was a different name (EMOTION = POSITIVE)
Tutor: Yeah that's right you know Newton was a genius-
KVL: This text will not be visible on the 9th floor setup. How about just including the student turns and not the tutor turns on the slide?

10. Predicting Emotion Ripper (machine learning program)
Input:
1) classes to be learned (our 3 emotion categories)
2) names and possible values for a set of features
(next slide)
3) training examples with class and feature values (the annotated student turns)
Output: an ordered set of if…then rules
for classifying future examples

11. Pilot Machine Learning Results
Six turn features
- Problem, Student, Duration, StartTime, Transcription, #Words
-all features automatically available in real-time
Cross-validated error (33.03%) significantly lower than majority class baseline (55.69%)
if (duration ≥ 0.65) & (text has “I”) then negative
else if (duration ≥ 2.98) then negative
else if (duration ≥ 0.93) & (startTime ≥ ) then positive
else if (text has “right”) then positive
else neutral

12. Summary and Current Directions
1) Pilot study suggests there are indeed features that can be used to automatically predict emotion in tutoring dialogues
Wider variety of features from many knowledge sources (e.g., pitch, amplitude, timing, other acoustic/prosodic, syntactic, semantic, discourse)
Reliable Emotion Annotation Guidelines
Analysis of Human-Computer Corpus
2) Empirical comparisons with typed tutorial dialogues (Building Educational Applications Using NLP paper)

13. Text-Feature Ruleset 1 Feature: Text in Turn
Figure 2: Text-Feature Ruleset for Emotion Prediction (excerpt from 21 rules))
if (text has “the”) & (text has “don't”) then negative
else if (text has “I”) & (text has “don't”) then negative …
else if (text has “um”) & (text has “<hn>”) then negative
else if (text has “the”) & (text has “<fs>”) then negative
else if (text has “right”) then positive
else if (text has “so”) then positive
else if (text has “(laugh)”) & (text has “that's”) then positive
else neutral
Estimated mean error and standard deviation: 39.03% +/- 2.40%, based on 25-fold cross-validation