1. © Löckelt, Becker, Pfleger, Alexandersson; DFKI Edilog 2002 Workshop Jan Alexandersson (Tilman Becker, Markus Löckelt, Norbert Pfleger) German Research Center for Artificial Intelligence – DFKI GmbH Stuhlsatzenhausweg 3, Geb. 43.8 - 66123 Saarbrücken phone: (+44) 681 302 5347 Email: janal@dfki.de http://www.dfki.de/~janal Overlay – a non-monotonic operation for discourse processing

2. © Alexandersson; DFKI Nancy 18.10.2002 What is this Talk about Dialogue systems must deal with utterances relating elliptically to previous dialogue User: „half past five“ System: „at what time would you like to start recording?“ User: „I would like to start recording at half past five“... or utterances only partially related to the previous discourse System: [Showing a list of films] „Here is a list of films“ User: „what films are on TV tonight?“ User: ``Thats a boring program. I‘d rather go to the movies´´ User: ``Thats a boring program. I‘d rather go to the movies tonight´´

3. © Alexandersson; DFKI Nancy 18.10.2002 What is this Talk about? ``...an expensive large portrait of the queen by Wainright hanging in the library´´  ``and one of the princess over the mantel´´ There are similar elliptical phenomena like one-anaphora  ``and an expensive large portrait of the princess hanging over the mantel´´

4. © Alexandersson; DFKI Nancy 18.10.2002 Overview SmartKom Domain Model Overlay Discourse Modelling Action Planning Processing Partial Utterances Conclusion

5. © Alexandersson; DFKI Nancy 18.10.2002 What is the challenge? General tasks for a dialogue manager –Enrichment: Given incomplete information, enrich it from other sources in order to proceed. –Validation: Validate hypotheses against the current discourse state

6. © Alexandersson; DFKI Nancy 18.10.2002 Our Approach One general operation – Overlay –Domain representation: (Typed) Feature Structures –Non-monotonic: Always succeeds –Computes a score: Reflecting the structural consistency of the two arguments of overlay

7. © Alexandersson; DFKI Nancy 18.10.2002 SmartKom: Task-Oriented Dialogue System MM Dialogue Back- Bone Home: Consumer Electronics EPG Public: Cinema, Phone, Fax, Mail, Biometrics Mobile: Car and Pedestrian Navigation Application Layer SmartKom-Mobile SmartKom-Public SmartKom-Home/Office

8. © Alexandersson; DFKI Nancy 18.10.2002 Module Overview on the SmartKom Control GUI

9. © Alexandersson; DFKI Nancy 18.10.2002 The True Story

10. © Alexandersson; DFKI Nancy 18.10.2002 The DFKI Dialogue Back-Bone Communication pools Main data flow Context information Analysers External Services Modality Fusion Discourse Modelling DiM Action Planning AP Presentation Manager Generators Speech Gesture Speech Graphics Gesture

11. © Alexandersson; DFKI Nancy 18.10.2002 theater: MovieTheater movie: Movie reservationNumber: PositiveInteger Domain Model Used for communication in the back-bone Frame-based ontology; representation as Typed Feature Structures in M3L (XML) name: String director: Person cast: PersonList yearOfProduction: PositiveInteger… address: Address seats: SeatStructure… CinemaReservation Application objects composed of subobjects Slots: Feature paths meaningful for the dialogue (entities that can be talked about / referenced to); e.g. movie:director:lastName in a CinemaReservation object Slots can recursively contain other slots firstName: String lastName: String…

12. © Alexandersson; DFKI Nancy 18.10.2002 Discourse Modeling Lattice of intention hypothesis sequences from Modality Fusion Enrichment and Validation –compare and enrich with a selected number of discourse states: fill in consistent information compute a score –for each hypothesis - background pair: Overlay (covering, background) Overlay gives a score representing how well the covering fits the background. Scores of overlay, speech interpretation etc. give overall score One augmented hypothesis sequence with best score is passed on to Action Planning Covering: Background: Intention lattice Selected augmented hypothesis sequence

13. © Alexandersson; DFKI Nancy 18.10.2002 Example for Overlay with TFS User: What films are on TV tonight? System: [presents list of films] User: That‘s a boring program, I‘d rather go to the movies. How do we inherit “tonight” ?

14. © Alexandersson; DFKI Nancy 18.10.2002 The Domain Model A named entertainment at some time A named TV program at some time on some channel A named Movie at some time at some cinema

15. © Alexandersson; DFKI Nancy 18.10.2002 Unification Simulation Films on TV tonightGo to the movies Fail – type clash

16. © Alexandersson; DFKI Nancy 18.10.2002 Overlay Simulation Go to the moviesFilms on TV tonight Assimilation Background Covering

17. © Alexandersson; DFKI Nancy 18.10.2002 ``Formal´´ Definition Overlay Let –co be covering –bg be background Step 1: –Assimilate(co,bg) T bg co Step 2: –Overlay(co,assimilate(co,bg)) If co and bg are frames: recursion If co is empty: use bg If bg is empty: use co If conflict: use co

18. © Alexandersson; DFKI Nancy 18.10.2002 Domain Models with Multiple Inheritance Assimilate(co,bg) –Compute the set of minimal upper bounds (MUB) –Specialize the MUBs –Unify the specialized MUBs T cobg Overlay remains untouched MUB

19. © Alexandersson; DFKI Nancy 18.10.2002 Processing Partial User Contributions Our Dialogues contain a lot of partial contributions –``Three´´ –``Three o‘clock´´ –``At three o‘clock´´ –``At three [  ] o‘clock´´ –``Later´´ –``Information [  ]´´ –... Requirement –Advanced discourse modelling and action planning –Bidirection communication DiM - AP

20. © Alexandersson; DFKI Nancy 18.10.2002 Discourse Modelling - Three-Tiered Context Representation DO 1 DO 2 LO 1 DO 10 DO 3 DO 9 Modality layer Discourse layer System: Here [pointing gesture] I show you a list of films running in Heidelberg. show heidelberg list LO 2 LO 3... Domain layer DomainObject 1 ticketfirst DO 11 DO 12 reserve LO 4 LO 5 LO 6 DomainObject 2 GO 1...  User: Reserve a ticket for the first!

21. © Alexandersson; DFKI Nancy 18.10.2002 Action Planner – Regression Planning AP processes plan operators –Uniform description of mixed initiative with user or applications based on Games and Moves. Example: User: request – System: response System: request – Application: response Processing steps 1.Constructs a plan from plan operators –A Plan specifies a partial order of games 2.Interprets the plan –Execute the games Expectations are published during execution of the plan –Guide the interpretation of input

22. © Alexandersson; DFKI Nancy 18.10.2002 Action Planning - Regression planning provides(Y) needs(D) condition(D=x) Goal provides() needs(X,Y) provides(X) needs(A,B) provides(C) needs() provides(B) needs(D) provides(A) needs(C) provides(F,G) needs(H) provides(D) needs(E) For Example: Slot D=y Current State provides(?) needs(?) successor states

23. © Alexandersson; DFKI Nancy 18.10.2002 AP - Example Plan: VCR Goal: VCR_record VCR_checkMediumVCR_getChannelVCR_getStartTimeVCR_getEndTime VCR_rewindMedium

24. © Alexandersson; DFKI Nancy 18.10.2002 Action Planning Initiation of sub-dialogues to gather necessary information Order proposed by the system, but the user may not follow the trail Several cases of user responses to requests: –Providing a matching answer, –Supplying information not (yet) asked for, –Changing already established information –Acting uncooperatively or changing goals

25. © Alexandersson; DFKI Nancy 18.10.2002 Expectations When AP initiates a sub-dialogue, it expects a certain answer An Expectation structure is published to help the analysis modules –Expected Slots: corresponding to the request –Possible Slots: plausible in context –Filled Slots: already provided –The currently active Application Allows for a fine-grained distinction between types of possible utterances Discourse Modelling can employ different rankings for each category

26. © Alexandersson; DFKI Nancy 18.10.2002 Integration of Partial Partial Utterances are analyzed as subobjects –validation and enrichment include the integration into the preceding context (to achieve a coherent discourse) –we consider three types of partial utterances: 1.The system has the initiative and the user responds elliptically to a system request 2.An elliptical user utterance does not correspond to a request but can be interpreted in context 3.Misinterpretation or non-cooperative user behaviour

27. © Alexandersson; DFKI Nancy 18.10.2002 Integration of Partial - Possible Situations User Initiative System Initiative Expected slotN A expected, unify, plan continues Possible slot plausible, unify, replanning plausible, unify, replanning Filled slot plausible, overlay, replanning plausible, overlay, replanning Otherimplausible – recover strategies User contribution DiM processing AP processing

28. © Alexandersson; DFKI Nancy 18.10.2002 DiM - Processing User Responses User provides expected slots –Bridging: a new application object is created (of same type as the current application object) for each subobject in the hypotheses its type is compared to the expected slots –matching subobjects are integrated into the new application object –remaining subobjects are interpreted as plausible partial utterances (next slide) –Overlay: The extended application object is overlayed over the application object currently in focus

29. © Alexandersson; DFKI Nancy 18.10.2002 Processing Plausible Partial Utterances User provides possible slots –processed like expected slots (but get a score penalty) User changes filled slots –search for a discourse object DO of the same type in the discourse context –create application object AO1 of the same type as the application object AO2 within which DO was mentioned –overlay AO1 over AO2 –gets a score penalty based on position in discourse history If integration fails, all subobjects are passed on unchanged (integration fails)

30. © Alexandersson; DFKI Nancy 18.10.2002 Example: Expected and Possible Slots (1) User: I want to record a film. (2) System: When should I start recording? (3) User: 1:30pm on channel two. Analysis of (1) contains the setting of a goal VCR_record (empty application object of type VCR introduced) AP triggers presentation (2) and publishes an expectation containing: –VCR as the type of the current application object –VCR_startTime as the (only) expected slot –some possible slots (e.g., VCR_endTime and VCR_channel ) –and maybe some filled slots (e.g. VCR_mediumPresent ) Analysis of (3) contains two subobjects –the expected slot is filled by “1:30pm” –but also one possible slot is filled by “channel two”

31. © Alexandersson; DFKI Nancy 18.10.2002 Example: Filled Slots (1) User: What is currently running on TV? (2) System: The following [Smartakus points at a list of programmes] programmes are currently running. (3) User: And tonight?

32. © Alexandersson; DFKI Nancy 18.10.2002 Example: Filled Slots show programcurrently Modality layer Discourse layer Domain layer DO 1 DO 2 LO 1 LO 2 DO 3 LO 3 tonight DO 4 LO 4 DO 3

33. © Alexandersson; DFKI Nancy 18.10.2002 Conclusion Uniform, flexible and robust mechanism for interpreting possible partial utterances Pragmatical classification of partial utterances from the point of view of the action planner Modality independent Approach integrates seamlessly with other aspects of discourse processing, e.g. anaphora resolution Implemented and running in, e.g., a large multimodal dialogue system

34. © Alexandersson; DFKI Nancy 18.10.2002 Papers Jan Alexandersson and Tilman Becker. Overlay as the Basic Operation for Discourse Processing in a Multimodal Dialogue System. In: Proceedings of the IJCAI Workshop ``Knowledge and Reasoning in Practical Dialogue Systems,´´ Seattle, 2001. Norbert Pfleger, Jan Alexandersson, and Tilman Becker. Scoring Functions for Overlay and their Application in Discourse Processing. In Proceedings of ``KONVENS 2002,´´ Saarbrücken, Germany, 2002. Markus Loeckelt, Tilman Becker, Norbert Pfleger and Jan Alexandersson Making Sense of Partial. In: Bos, Foster & Matheson (eds): ``Proceedings of the sixth workshop on the semantics and pragmatics of dialogue (EDILOG 2002),´´ Edinburgh, UK, Pages 101-107. Jan Alexandersson and Tilman Becker. The Formal Foundations Underlying Overlay. Submitted to IWCS5

35. © Alexandersson; DFKI Nancy 18.10.2002 Thank you very much for your attention! Merci infiniment de votre attention! Babelfish.altavista.com