1. U1, Speech in the interface:2. Dialogue Management1 Module u1: Speech in the Interface 2: Dialogue Management Jacques Terken HG room 2:40 tel. (247) 5254 j.m.b.terken@tue.nl

2. U1, Speech in the interface:2. Dialogue Management2 contents 1. Tasks of the dialogue manager 2. Initiative/Control 3. Dialogue structure 4. Dealing with channel and technology limitations

3. U1, Speech in the interface:2. Dialogue Management3 Dialogue phenomena n Natural language phenomena: User: “a flight from Boston to New York” System: “7:15 with Continental” User: “A later one?” [ flight from B. to NY.]

4. U1, Speech in the interface:2. Dialogue Management4 n Recognition of user’s dialogue act U: “I want a flight from Boston to New York” syntactically: statement pragmatically: request for information n Dealing with recognition errors, misunderstandings and other communication problems U: A flight from Boston to New York … S; I’ve booked a flight from Houston to Newark … In addition, users often have problems to know what kind of reaction is expected for successful communication

5. U1, Speech in the interface:2. Dialogue Management5 DM tasks n Interpreting user contribution in dialogue context and situational context –Dealing with natural language phenomena –Recognition of user’s dialogue act –Involves dealing with recognition errors, misunderstandings and other communication problems n Deciding on next system contribution

6. U1, Speech in the interface:2. Dialogue Management6 Deciding upon next system contribution n Interaction as co-operation –User and system co-operate to achieve a goal –transmission of information across communication channel may induce distortions  distinction between task-oriented acts and dialogue control acts 

7. U1, Speech in the interface:2. Dialogue Management7 n Task-oriented dialogue acts –Bring the dialogue purpose closer –WH-question, YN-question, Inform, WH-answer, … n Dialogue control acts –Directed towards keeping dialogue on the track and prevent and deal with problems, e.g. Requests for clarification Verification Feedback/confirmation Error recovery strategies But also greetings, apologies etc.

8. U1, Speech in the interface:2. Dialogue Management8 contents 1. Tasks of the dialogue manager 2. Initiative/Control 3. Dialogue structure 4. Dealing with channel and technology limitations

9. U1, Speech in the interface:2. Dialogue Management9 Initiative n System initiative S: Where are you travelling to U: London S: What day do you wish to travel U: Friday S: At what time U: 9 a.m

10. U1, Speech in the interface:2. Dialogue Management10 n User initiative: U: How many employees living in the London area earn more than 50000 ₤ S: Fifty four U: How many are female S: Eleven U: And managers S: Nine

11. U1, Speech in the interface:2. Dialogue Management11 n Mixed initiative S: Where are you travelling to U: I want to fly to London on Friday S: At what time do you want to fly to London U: Are there any cheap flights

12. U1, Speech in the interface:2. Dialogue Management12 contents 1. Tasks of the dialogue manager 2. Initiative/Control 3. Dialogue structure 4. Dealing with channel and technology limitations

13. U1, Speech in the interface:2. Dialogue Management13 Finite-state dialogue grammar n Dialogue structure represented as finite state transition network Departure town  arrival town  date  time  carrier n With repair sub-dialogues: abcdabcd rrrr

14. U1, Speech in the interface:2. Dialogue Management14 Destination? Was that $Destination? Day? Was that $Day? yesno yes London Friday

15. U1, Speech in the interface:2. Dialogue Management15 Frame-based approach n Dialogue Frame: Departure airport Destination airport Date Time Carrier n Onset condition: All parameters unknown n Condition action pairs produce dialogue structure: Condition: Origin and destination unknown Question: Which route do you want to travel

16. U1, Speech in the interface:2. Dialogue Management16 DestinationDayTime

17. U1, Speech in the interface:2. Dialogue Management17 Constraint relaxation and narrowing n “I want a flight from Boston to New York Friday morning, leaving between 7:15 and 7:45” n If no records are found, ask user to be more general n If many options are found (e.g. ≥ 5), ask user to be more specific n If 1 ≤ number of found options < N (e.g. 5), generate response that outputs the retrieved records: “I found the following flights: ….”

18. U1, Speech in the interface:2. Dialogue Management18 Cooperative dialogue n Don’t ask user to be more specific or more general, but propose solutions E.g. U: Is there a flight from Boston to New York between 7:15 and 7:45? S’: No S’’: No, but there is a flight at 7:55. Do you want me to book that one? n Requires understanding the user’s intentions and priorities: S’’’: * No but there is a flight to Philadelphia between 7:15 and 7:45

19. U1, Speech in the interface:2. Dialogue Management19 Theoretical frameworks for dialogue management n Plan-based approaches Dialogue as a special instance of rational (goal-directed) behaviour System job is to discover and react adequately to the speaker’s plan rather than the utterance n Modelling of Beliefs-Desires-Intentions Adds modeling of assumptions and beliefs that speaker has, in order to identify common ground n Rational agency (dialogue management from first principles) n Agent-based approaches (combination of plan-based approaches and local repair mechanisms with heterogeneous control)

20. U1, Speech in the interface:2. Dialogue Management20 contents 1. Tasks of the dialogue manager 2. Initiative/Control 3. Dialogue structure 4. Dealing with channel and technology limitations

21. U1, Speech in the interface:2. Dialogue Management21 n Technology limitations: speech recognition errors n Channel limitations: –Speech is sequential and volatile n Requires feedback at different levels of communication: –Perception –Interpretation –Evaluation  Grounding theory (Clark): dialogue as joint activity; reaching agreement over what was said and meant

22. U1, Speech in the interface:2. Dialogue Management22 Dialogue strategies n Zooming Asking general questions first and narrowing down if no adequate response is received “How may I help you” n Adequate prompts are important –They inform the user about what kind of reply is expected –Require iterative design in combination with user studies (collecting extensive set of user reactions)

23. U1, Speech in the interface:2. Dialogue Management23 Verification strategies n Verification provides feedback and ensures that both participants stay in agreement about the state of the dialogue (what was said and agreed) n Explicit verification –So you want to go to New York n Implicit verification –What time do you want to arrive in New York n Confidence-based verification –Repeat question with very low confidence –Explicit verification with low to intermediate confidence –Implicit verification with higher confidence –No verification with very high confidence

24. U1, Speech in the interface:2. Dialogue Management24 Project n Exercises –Start from the original pizza application –Implement different verification strategies, both explicit and implicit –Try the system while simulating speech recognition errors Hint: wreck one of the options and run a scenario n Project –Conduct a task analysis (or construct use cases) and define the basic dialogue structure