1 error handling – Higgins / Galatea Dialogs on Dialogs Group July 2005

2 work by … Gabriel Skantze ph.d. student KTH, Stockholm “I am doing research on spoken dialogue systems. More specifically, I am interested in studying miscommunication and error handling, but also in the representation and modelling of utterances and dialogue, as well as conducting experiments with users.“ and co-authors: J. Edlund, D. House, R. Carlson

3 3 papers Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

4 1 st paper Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

5 Higgins practical goal of Higgins project  build a collaborative dialog system in which error handling ideas can be tested empirically error handling issues, plus  incremental dialogue processing  on-line prosodic feature extraction  robust interpretation  flexible generation and output

6 domain pedestrian city navigation and guiding  user gives system a destination  system guides user by giving verbal instructions complex  large variety of error types  semantic structures can be quite complex  reference resolution domain can be extended even further

7 architecture follow-up from Adapt everything is XML  domain objects  utterance semantics  discourse model  database content  system output (before surface)  3D city model

8 research issues early detection and correction late detection incrementality error recovery

9 early detection and correction KTH LVCSR – output likely to contain errors robust interpretation Pickering:  some syntactic analysis is needed e.g. relations between objects  but handles insertions and non-agreement phrases humans - good at early detection (woz)

10 late detection and correction discourse modeller (GALATEA)  joins several results from Pickering into a discourse model  adds grounding information  can be manipulated later remove concepts which turn out not to be grounded

11 incrementality end-pointers cause trouble  even more so in this domain better:

12 incrementality [2] all components support incremental processing several issues  when to barge in? (semantic content and prosody)  longer-than-utterance units: interpreter or dialog manager?  rapid and unobtrusive feedback: challenge for synthesis

13 error recovery signaling non-understandings  decreased experience of task success  slower recovery ask other task-related question

14 2 nd paper Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

15 GALATEA a discourse modeller for conversational spoken dialog systems  builds a discourse model (what has been said during the discourse) resolution of ellipses & anaphora tracks the grounding status  who said what when (plus confidence information)  can be used for concept-level error handling

16 should do grounding at concept level explicit and implicit verification on whole utterance can be tedious and unnatural 45% of clarifications in BNC are fragmentary / elliptical

17 should do grounding at concept level Traum (1994) – utterance level computational model of grounding Larsson (2002) – issue-level computational model of grounding in Issue-Based DM Rieser (2004), Schlangen (2004): systems capable of fragmentary clarification requests, but models do not handle user reactions systems should keep grounding information at the concept level like RavenClaw?

18 semantic representation rooted unordered trees of semantic concepts  nodes: attr-value pairs, objects, relations, properties

19 semantic representation enhanced with “meta”-information  confidence  communicative acts  info is new / given

20 ellipsis resolution transforms ellipsis into full propositions rule based  ~10 rules  domain-specific

21 anaphora resolution keeps a list of entities (talked about)  assigns ids when given entities are added to the discourse, look up the antecedent if found, unification (and move to the top of the entity list) unification also allows entities to be referred to in new ways how does this fare and compare?

22 grounding status who added the concept? in which turn? how confident? may be used by the action manager  for instance remove all items with high grounding status when referring to an entity

23 updating grounding status

24 late error detection discover inconsistencies in discourse model look at grounding status to see where error may be  concept can be removed

25 future methods for automatic tuning of strategy selection extend to track confidence and grounding status at different levels evaluate how people respond to incorrect confirmations, and how can that information be used to update grounding status error recovery after non-understandings other domains

26 3 rd paper Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

27 prosody in clarifications effects of prosodic features on interpretation of elliptical clarifications  U: Further ahead on the right I see a red building…  S: Red (?) vary prosodic features study impact on user’s understanding of the system’s intention

28 motivation long (whole utterance) confirmations are not good  tedious, unnatural  BNC corpus: 45% of clarifications are elliptical short confirmations  make dialog more efficient by focusing on the actual problematic fragments however  interpretation depends on context and prosody

29 3 readings U: Further ahead on the right I see a red building… S: Red (?) Ok, red [all positive] Do you really mean red? What do you mean by red? [positive perception, negative understanding] Did you say red? [positive contact, negative perception]

30 stimuli 3 test words [red, blue, yellow] di-phone voice (MBROLA) manipulated  peak position [mid, early, late / 100ms]  peak height [130Hz / 160 Hz]  vowel duration [normal, long / +100ms]

31 subjects + design 8 speakers: 2f / 6m, 2nn / 6n introduced to Higgins listen to all 42 (only once); random order 3 options  Okay, X  Did you really mean X?  Did you say X?

32 results no effects for  color, subject, duration significant effects for  peak position, peak height, & their interaction

33 results Statement: early, low peak Question: late, high peak Clear division between “did you mean” and “did you say”

34 food for thought how about English? red red? red!? how many ways can you say it?

35 conclusion strong relationship between intonation and meaning statement: early, low peak question: late, high peak clear division between “did you mean” and “did you say”

36 the end