1. Is This Conversation on Track? Utterance Level Confidence Annotation in the CMU Communicator spoken dialog system Presented by: Dan Bohus (dbohus@cs.cmu.edu) Work by: Paul Carpenter, Chun Jin, Daniel Wilson, Rong Zhang, Dan Bohus, Alex Rudnicky Carnegie Mellon University – 2001

2. 09-06-2001Is This Conversation on Track ? Outline The Problem. The Approach Training Data and Features Experiments and Results Conclusion. Future Work

3. 09-06-2001Is This Conversation on Track ? The Problem Systems often misunderstand, take misunderstanding as fact, and continue to act using invalid information Repair costs Increased dialog length User Frustration Confidence annotation provides critical information for effective confirmation and clarification in dialog systems.

4. 09-06-2001Is This Conversation on Track ? The Approach Treat the problem as a data-driven classification task. Objective: accurately label misunderstood utterances. Collect a training corpus. Identify useful features. Train a classifier ~ identify the best performing one for this task.

5. 09-06-2001Is This Conversation on Track ? Data Communicator Logs & Transcripts: Collected 2 months (Oct, Nov 1999). Eliminated conversations with < 5 turns. Manually labeled OK (67%) / BAD (33%) BAD ~ RecogBAD / ParseBAD / OOD / NONSpeech Discarded mixed-label utterances (6%). Cleaned corpus of 4550 utterances / 311 dialogs.

6. 09-06-2001Is This Conversation on Track ? Feature Extraction 12 Features from various levels: Decoder Features: Word Number, Unconfident Percentage Parsing Features: Uncovered Percentage, Fragment Transitions, Gap Number, Slot Number, Slot Bigram Dialog Features: Dialog State, State Duration, Turn Number, Expected Slots Garble: handcrafted heuristic currently used by the CMU Communicator

7. 09-06-2001Is This Conversation on Track ? Experiments with 6 different classifiers Decision Tree Artificial Neural Network Naïve Bayes Bayesian Network Several network structures attempted AdaBoost Individual feature-based binning estimators as weak learners, 750 boosting stages Support Vector Machines Dot, Polynomial, Radial, Neural, Anova

8. 09-06-2001Is This Conversation on Track ? Evaluating performance Classification Error Rate (FP+FN) CDR = 1-Fallout = 1-(FP/NBAD) Cost of misunderstanding in dialog systems depends on Error type (FP vs. FN) Domain Dialog state Ideally, build a cost function for each type of error, and optimize for that

9. 09-06-2001Is This Conversation on Track ? Results – Individual Features Features (top 8)Mean Err. Rate Uncovered Percentage19.93% Expected Slot20.97% Gap Number23.01% Bigram Score23.14% Garble25.32% Slot Number25.69% Unconfident Percentage27.34% Dialog State31.03% Baseline error 32.84% (when predicting the majority class) All experiments involved 10-fold cross-validation

10. 09-06-2001Is This Conversation on Track ? Results – Classifiers ClassifierMean Err. RateF/P RateF/N Rate AdaBoost16.59%11.43%5.16% Decision Tree17.32%11.82%5.49% Bayesian Network17.82%9.41%8.42% SVM18.40%15.01%3.39% Neural Network18.90%15.08%3.82% Naïve Bayes21.65%14.24%7.41% T-Test showed there is no statistically significant difference between the classifiers except for the Naïve Bayes Explanation: independence between feature assumption is violated Baseline error 25.32% (GARBLE)

11. 09-06-2001Is This Conversation on Track ? Future Work Improve the classifiers Additional features Develop a cost model for understanding errors in dialog systems. Study/optimize tradeoffs between F/P and F/N; Integrate value and confidence information to guide clarification in dialog systems

12. 09-06-2001Is This Conversation on Track ? Confusion Matrix OKBAD System says OKTPFP System says BADFNTN FP = False acceptance FN = False detection/rejection Fallout = FP/(FP+TN) = FP/NBAD CDR = 1-Fallout = 1-(FP/NBAD)