2. Error Awareness and Recovery in Task-Oriented Spoken Dialogue Systems Thesis Proposal Dan Bohus Carnegie Mellon University, January 2004 Thesis Committee Alex Rudnicky (Chair) Roni Rosenfeld Jeff Schneider Eric Horvitz (Microsoft Research)

3. 2 Problem  Spans most domains and interaction types  Has a significant impact on performance Lack of robustness when faced with understanding errors

4. 3 An example S: Are you a registered user ? U: No I'm not. No [NO I'M NOT NO] S: What is your full name ? U: Adam Schumacher [ATHENS IN AKRON] S: There is an Athens in Georgia and in Greece. Which destination did you want ? U: Neither [NEITHER] S: I'm sorry, I didn't catch that. There is an Athens in Georgia and in Greece. Which destination did you want ? U: Georgia [GEORGIA] S: A flight from Athens... Where do you want to go ? U: Start over [START OVER] S: Do you really want to start over ? U: Yes [YES] S: What city are you leaving from ? U: Hamilton, Ontario [HILTON ONTARIO] S: Sorry, I'm not sure I understood what you said. Where are you leaving from ? U: Hamilton [HILTON] S: I'm still having trouble understanding you. To go on, I need you to answer the following question. What city are you leaving from ? U: Toronto [TORONTO]

5. 4 Some Statistics …  Corrections [Krahmer, Swerts, Litman, Levow]  30% of utterances correct system mistakes  2-3 times more likely to be misrecognized  Semantic error rates CMU Communicator [CMU] 32% CU Communicator [CU] 27% How May I Help You? [AT&T] 36% Jupiter [MIT] 28% SpeechActs [SRI] 25%

6. 5 Significant Impact on Interaction  CMU Communicator 40% 26% Contain understanding errors Failed  Multi-site Communicator Corpus [Shin et al] 37% Failed sessions 33% 63%

7. 6 Outline  Problem  Approach  Infrastructure  Research Program  Timeline & Summary problem : approach : infrastructure : indicators : strategies : decision process : summary

8. 7 Increasing Robustness …  Increase the accuracy of speech recognition  Assume recognition is unreliable, and create the mechanisms for acting robustly at the dialogue management level  ASR performance increases / demands increase  More general problem : approach : infrastructure : indicators : strategies : decision process : summary

9. 8 Snapshot of Existing Work: Slide 1  Theoretical models of grounding  Contribution Model [Clark], Grounding Acts [Traum]  Practice: heuristic rules  Misunderstandings Threshold(s) on confidence scores  Non-understandings problem : approach : infrastructure : indicators : strategies : decision process : summary Analytical/Descriptive, not decision oriented Ad-hoc, lack generality, not easy to extend

10. 9 Snapshot of Existing Work: Slide 2  Conversation as Action under Uncertainty [Paek and Horvitz]  Belief networks to model uncertainties  Decisions based on expected utility, VOI-analysis  Reinforcement learning for dialogue control policies [Singh, Kearns, Litman, Walker, Levin, Pieraccini, Young, Scheffler, etc]  Formulate dialogue control as an MDP  Learn optimal control policy from data Do not scale up to complex, real-world domains problem : approach : infrastructure : indicators : strategies : decision process : summary

11. 10 A task-independent, adaptive and scalable framework for error recovery in task-oriented spoken dialogue systems Research Program: Goals & Approach  Decision making under uncertainty Approach: problem : approach : infrastructure : indicators : strategies : decision process : summary

12. 11 1.Error awareness 2.Error recovery strategies 3.Error handling decision process Three Components Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary 0.Infrastructure problem : approach : infrastructure : indicators : strategies : decision process : summary

13. 12 Infrastructure  RavenClaw  Modern dialog management framework for complex, task-oriented domains  RavenClaw spoken dialogue systems  Test-bed for evaluation problem : approach : infrastructure : indicators : strategies : decision process : summary Completed

14. 13 RavenClaw Dialogue Task (Specification) Domain-Independent Dialogue Engine RoomLine Login Welcome AskRegisteredAskName GreetUser GetQuery DateTimeLocationProperties NetworkProjectorWhiteboard GetResultsDiscussResults user_nameregistered query results RoomLine Login AskRegistered Dialogue Stack registered: [No]-> false, [Yes] -> true Expectation Agenda Error Handling Decision Process Strategies Indicators ExplicitConfirm problem : approach : infrastructure : indicators : strategies : decision process : summary

15. 14 RavenClaw-based Systems  RoomLine  CMU Let’s Go!! Bus Information System  LARRI [Symphony]  TeamTalk [11-741]  Eureka [11-743] problem : approach : infrastructure : indicators : strategies : decision process : summary

16. 15 0.Infrastructure 1.Error awareness 2.Error recovery strategies 3.Error handling decision process Three Components Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

17. 16 Existing Work  Confidence Annotation  Traditionally focused on speech recognizer [Bansal, Chase, Cox, and others]  Recently, multiple sources of knowledge [San-Segundo, Walker, Bosch, Bohus, and others] Recognition, parsing, dialogue management  Detect misunderstandings: ~ 80-90% accuracy  Correction and Aware Site Detection [Swerts, Litman, Levow and others]  Multiple sources of knowledge  Detect corrections: ~ 80-90% accuracy problem : approach : infrastructure : indicators : strategies : decision process : summary

18. 17 S: Where are you flying from? U: [CityName={Aspen/0.6; Austin/0.2}] S: Did you say you wanted to fly out of Aspen? U: [No] [CityName={Boston/0.8}] Proposed: Belief Updating  Continuously assess beliefs in light of initial confidence and subsequent events [CityName={Aspen/?; Austin/?; Boston/?}]  An example: problem : approach : infrastructure : indicators : strategies : decision process : summary initial belief + system action + user response updated belief

19. 18 contents Belief Updating: Approach  Model the update in a dynamic belief network CC system action User response features tt + 1 problem : approach : infrastructure : indicators : strategies : decision process : summary CC system action initial beliefupdated belief confidence correction contentsconfidence correction CurrentTopCurrent2nd Current3r d Confidence YesNo Positive Markers Negative Markers Utterance Length

20. 19 0.Infrastructure 1.Error awareness 2.Error recovery strategies 3.Error handling decision process Three Components Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

21. 20 Is the Dialogue Advancing Normally? Locally, turn-level:  Non-understanding indicators  Non-understanding flag directly available  Develop additional indicators Recognition, Understanding, Interpretation Globally, discourse-level:  Dialogue-on-track indicators  Summary statistics of non-understanding indicators  Rate of dialogue advance problem : approach : infrastructure : indicators : strategies : decision process : summary

22. 21 0.Infrastructure 1.Error awareness 2.Error recovery strategies 3.Error handling decision process Three Components Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

23. 22 Error Recovery Strategies  Identify  Identify and define an extended set of error handling strategies  Implement  Construct task-decoupled implementations of a large number of strategies  Evaluate  Evaluate performance and bring further refinements

24. 23 List of Error Recovery Strategies Help Where are we? Start over Scratch concept value Go back Channel establishment Suspend/Resume Repeat Summarize Quit Restart subtask plan Select alternative plan Start over Terminate session / Direct to operator Local problems (non-understandings) Global problems (compounded, discourse-level problems) Switch input modality SNR repair Ask repeat turn Notify non-understanding Explicit confirm turn Targeted help WH-reformulation Keep-a-word reformulation Generic help You can say Ask rephrase turn problem : approach : infrastructure : indicators : strategies : decision process : summary User InitiatedSystem Initiated Explicit confirmation Implicit confirmation Disambiguation Ask repeat concept Reject concept Ensure that the system has reliable information (misunderstandings) Ensure that the dialogue on track

25. 24 List of Error Recovery Strategies Help Where are we? Start over Scratch concept value Go back Channel establishment Suspend/Resume Repeat Summarize Quit Restart subtask plan Select alternative plan Start over Terminate session / Direct to operator Local problems (non-understandings) Global problems (compounded, discourse-level problems) Switch input modality SNR repair Ask repeat turn Notify non-understanding Explicit confirm turn Targeted help WH-reformulation Keep-a-word reformulation Generic help You can say Ask rephrase turn problem : approach : infrastructure : indicators : strategies : decision process : summary User InitiatedSystem Initiated Explicit confirmation Implicit confirmation Disambiguation Ask repeat concept Reject concept Ensure that the system has reliable information (misunderstandings) Ensure that the dialogue on track

26. 25 Error Recovery Strategies: Evaluation  Reusability  Deploy in different spoken dialogue systems  Efficiency of non-understanding strategies  Simple metric: Is the next utterance understood?  Efficiency depends on decision process  Construct upper and lower bounds for efficiency Lower bound: decision process which chooses uniformly Upper bound: human performs decision process (WOZ) problem : approach : infrastructure : indicators : strategies : decision process : summary

27. 26 0.Infrastructure 1.Error awareness 2.Error recovery strategies 3.Error handling decision process Three Components Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

28. 27  Dialogue control ~ Markov Decision Process  States  Actions  Rewards  Previous work: successes in small domains  NJFun [Singh, Kearns, Litman, Walker et al]  Problems  Lack of scalability  Once learned, policies are not reusable Previous Reinforcement Learning Work problem : approach : infrastructure : indicators : strategies : decision process : summary S1S1 S2S2 S3S3 A

29. 28 Proposed Approach Overcome previous shortcomings:  Focus learning only on error handling  Reduces the size of the learning problem  Favors reusability of learned policies  Lessens the system development effort  Use a “divide-and-conquer” approach  Leverage independences in dialogue problem : approach : infrastructure : indicators : strategies : decision process : summary

30. 29 Gated Markov Decision Processes RoomLine Login Welcome AskRegisteredAskName GreetUser user_nameregistered Gating Mechanism Concept-MDP Topic-MDP  Small-size models  Parameters can be tied across models  Easy to design initial policies  Decoupling favors reusability of policies  Accommodate dynamic task generation Independence assumption problem : approach : infrastructure : indicators : strategies : decision process : summary No Action Explicit Confirm No Action Explicit Confirmation

31. 30 Reward structure & learning Gating Mechanism MDP Action Global, post-gate rewards Reward Gating Mechanism MDP Action Local rewards Reward  Rewards based on any dialogue performance metric  Atypical, multi-agent reinforcement learning setting  Multiple, standard RL problems  Model-based approaches problem : approach : infrastructure : indicators : strategies : decision process : summary

32. 31 Evaluation  Performance  Compare learned policies with initial heuristic policies  Metrics Task completion Efficiency Number and lengths of error segments User satisfaction  Scalability  Deploy in a system operating with a sizable task  Theoretical analysis problem : approach : infrastructure : indicators : strategies : decision process : summary

33. 32 Outline  Problem  Approach  Infrastructure  Research Program  Summary & Timeline problem : approach : infrastructure : indicators : strategies : decision process : summary

34. 33  Overall Goal: develop a task-independent, adaptive and scalable framework for error recovery in task-oriented spoken dialogue systems  Modern dialogue management framework  Belief updating framework  Investigation of an extended set of error handling strategies  Scalable data-driven approach for learning error handling policies Summary of Contributions problem : approach : infrastructure : indicators : strategies : decision process : summary

35. 34 Timeline proposal milestone 1 milestone 2 milestone 3 defense end of year 4 end of year 5 now 5.5 years Data collection for belief updating and WOZ study Develop and evaluate the belief updating models Implement dialogue-on-track indicators Misunderstanding and non-understanding strategies Investigate theoretical aspects of proposed reinforcement learning model Evaluate non-understanding strategies; develop remaining strategies Error handling decision process: reinforcement learning experiments Data collection for RL training Data collection for RL evaluation data indicatorsstrategiesdecisions Contingency data collection efforts Additional experiments: extensions or contingency work problem : approach : infrastructure : indicators : strategies : decision process : summary

36. 35 Thank You! Questions & Comments committee members, then floor

37. 36 Indicators: Goals  Goal: Increase awareness and capacity to detect problems  Develop indicators which can inform the error handling process about potential problems Understanding process System acquires information System does not acquire information Non-understanding System acquires correct information System acquires incorrect information Misunderstanding OK

38. 37 problem : approach : support work : indicators : strategies : decision process : summary

39. 38 Three Desired Properties  Task-Independence  Reuse the proposed architecture across different spoken dialogue systems with a minimal amount of authoring effort  Adaptability  Learn from experience how to adapt to the characteristics of various domains  Scalability  Applicable in spoken dialogue systems operating with large, practical tasks

40. 39 HC ExplConf ImplConf NoAct LC ExplConf ImplConf NoAct MC ExplConf ImplConf NoAct 0

41. 40 Belief Updating: Approach  Model the update in a dynamic belief network CC System Action User response features Yes No CurrentTopCurrent2nd Current3r d Confidence Positive Markers Negative Markers Utterance Length tt + 1  Top-N values  Fixed structure  Learn parameters  Data collection  Evaluation  Accuracy  Soft-error problem : approach : infrastructure : indicators : strategies : decision process : summary CC System Action CurrentTopCurrent2nd Current3r d Confidence Yes No Positive Markers Negative Markers Utterance Length

42. 41 Gated Markov Decision Processes Issues:  Structure of individual MDPs  Gating mechanism  Reward structure and learning problem : approach : infrastructure : indicators : strategies : decision process : summary RoomLine Login Welcome AskRegisteredAskName GreetUser user_nameregistered Gating Mechanism Concept-MDP Topic-MDP No Action Explicit Confirm No Action Explicit Confirmation

43. 42 Structure for individual MDPs  State-space:  informative subset of corresponding indicators  Concept-MDPs: confidence / beliefs  Topic-MDPs: non-understanding, dialogue-on- track indicators  Action-space  corresponding system-initiated error handling strategies problem : approach : infrastructure : indicators : strategies : decision process : summary

44. 43 Gating Mechanism  Heuristic derived from domain-independent dialogue principles  Give priority to topics over concept  Give priority to entities closer to the conversational focus problem : approach : infrastructure : indicators : strategies : decision process : summary