2. 1 Natural Language Processing Lecture Notes 14 Chapter 19

3. 2 Today Spoken dialogue systems

4. 3 Talking to Computers Spoken dialogue systems make it possible to accomplish real tasks without talking to a real person Keys to success –goal-directed interactions in a limited domain –Priming users to adopt a vocabulary you can recognize –Segmenting the task into manageable stages –Judicious use of system vs. mixed initiative

5. 4 Dialogue vs. Monologue Monologue and dialogue both involve interpreting –Information status (given and new info) –Coherence issues –Reference resolution –Speech acts, implicature, intentionality Dialogue involves managing –Turn-taking –Grounding –Detecting and repairing misunderstandings –Initiative and confirmation strategies

6. 5 Example Here’s an (unfair) example from the ATT Toot system.

7. 6 Segmenting Speech into Utterances What is an `utterance’? –Single syntactic sentence may span several turns A: We've got you on USAir flight 99 B: Yep A: leaving on December 1. –Multiple syntactic sentences may occur in single turn A: We've got you on USAir flight 99 leaving on December. Do you need a rental car? –Utterance segmentation: cue words, n-gram word or POS sequences, prosody (pitch, accent, phrase-final lengthening, pause duration)

8. 7 Turns and Utterances Dialogue is characterized by turn-taking: –Who should talk next –When they should talk Turns in recorded speech: –Little speaker overlap (around 5% in English) –But little silence between turns either

9. 8 Turn Taking How do we know when a speaker is –Giving up or taking a turn? –Holding the floor? –Interruptable? How do I know when –Its my turn obligatorily –Optionally?

10. 9 Simple Turn-Taking Rules At each transition-relevance place (TRP) of each turn: –If current speaker has selected A as next speaker, then A must speak next –If current speaker does not select next speaker, any other speaker may take next turn –If no one else takes next turn, the current speaker may take next turn TRPs are where the structure of the language allows speaker shifts to occur

11. 10 Turn Taking Scripts Adjacency pairs set up next speaker expectations –GREETING/GREETING –QUESTION/ANSWER –COMPLIMENT/DOWNPLAYER –REQUEST/GRANT Significant silence is dispreferred A: Is there something bothering you or not? (1.0s) A: Yes or no? (1.5s) A: Eh? B: No.

12. 11 Overall System Strategies System Initiative (Control freak) S: Please give me your arrival city name. U: Baltimore. S: Please give me your departure city name U: Boston S:… Rigid, unnatural, difficulty with chatty users

13. 12 Overall System Strategies Mixed initiative S: How may I help you? U: I want to go to Boston. S: What day do you want to go to Boston? User Initiative S: How may I help you? U: I want to go from Boston to Baltimore on November 8.

14. 13 Grounding Participants are trying to come to a meeting of minds, they’re trying to establish common ground (or a set of mutual beliefs)

15. 14 Grounding Hearers must ground a speaker’s utterances by making it clear whether or not understanding has occurred Various ways to do this… S: I can upgrade you to an SUV at that rate. User: ????

16. 15 Grounding (Clark and Schaefer 1989) S: I can upgrade you to an SUV at that rate. –Continued attention/permission to proceed (User gazes appreciatively at S) –Relevant next contribution U: Do you have an Explorer available? –Acknowledgement/backchannel U: Ok/uh-huh/Great! –Display/repetition U: An SUV? U: You can upgrade me to an SUV at the same rate? –Request for repair U: Huh?

17. 16 Evaluation Performance of a dialogue system is affected both by what gets accomplished by the user and the dialogue agent and how it gets accomplished Maximize Task Success MinimizeCosts EfficiencyMeasuresQualitativeMeasures

18. 17 Metrics Efficiency of the Interaction:User Turns, System Turns, Elapsed Time Quality of the Interaction: ASR rejections, Time Out Prompts, Help Requests, Barge-Ins, Cancellation Requests, … User Satisfaction Task Success: perceived completion, information extracted, information learned (in tutorial setting)

19. 18 User Satisfaction Metrics TTS Performance –Was system easy to understand in this conversation? ASR Performance –In this conversation, did system understand what you said? Task Ease –In this conversation, was it easy to do what you wanted? Interaction Pace –Was the pace of interaction appropriate in this conversation? User Orientation –In this conversation, did you know what you could say at each point of the dialog? System Response –How often was the system sluggish and slow to reply to you in this conversation? Expected Behavior –Did system work the way you expected it to in this conversation?

20. 19 Misrecognition Repair Recognizing when the conversation has gone astray and recovering… –Mainly by analyzing the user’s utterances for signals that things are going astray

21. 20 Wrapup Spoken dialogue systems present new problems -- but also new possibilities –Recognizing speech introduces a new source of errors –Additional information provided in the speech stream offers new information about users’ intended meanings, emotional state (grounding of information, speech acts, reaction to system errors)