1. GoDiS (Gothenburg Dialogue System) with application to instructional text and dialogue ESSLLI, Helsinki 21st Aug 2001 Staffan Larsson sl@ling.gu.se

2. this lecture introduction to GoDiS –information state –moves, plans, actions –accommodation –optimistic grounding and acceptance demo information-oriented vs. action oriented dialogue instructional dialogue & text (natural command dialogue & menus)

3. originally: information-seeking dialogue –extended to action-oriented dialogue information state based Ginzburg’s notion of Questions Under Discussion (QUD) dialogue plans to drive dialogue simpler than general reasoning and planning more versatile than frame-filling and finite automata GoDiS features

4. input inter- pret TIS IS proper 5 module interface variables 3 resource interface variables DATABASE LEXICON DOMAIN data- base control updateselect gene- rate output lexicon domain knowledge DME

5. basic GoDIS infostate type PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) LU: SPEAKER: Speaker MOVES: assocSet( Move ) + module interface variables INPUT : String LATEST-MOVES: Set(Move) LATEST-SPEAKER: Speaker NEXT-MOVES: Set(Move) OUTPUT: String + resource interface variables LEXICON : Lexicon DOMAIN : Domain DATABASE : Database

6. PRIVATE =PLAN = AGENDA = { findout(?return) } SHARED = findout(? x.month(x)) findout(? x.class(x)) respond(? x.price(x)) COM = dest-city(paris) transport(plane) task(get_price_info) QUD = LU = SPEAKER = sys MOVES = {? x.depart-city(x)) - false } BEL = { } TMP = (same structure as SHARED) Sample GoDiS information state

7. Activity-based moves in GoDiS Moves are determined by the relation of the content to the domain –utterance U is an answer if the content A of U is a relevant answer to a question Q in the domain –moves are not necessarily speech acts! GoDiS basic dialogue moves –ask(Q), Q:question –answer(A), A:answer (proposition or fragment) –inform(P), P:proposition –request-repetition –greet, quit

8. integrateAnswer Before an answer can be integrated by the system, it must be matched to a question on QUD pre: eff: in($ SHARED.LM, answer(usr, A)) fst($ SHARED.QUD, Q) $ DOMAIN :relevant_answer(Q, A) pop( SHARED.QUD ) DOMAIN: reduce(Q, A, P) add( SHARED.COM, P) Sample update rule

9. Dialogue plans domain-specific partial specifications of how the system should act in a dialogue –specifies default behaviour –may be overridden by user initiative operational semantics of plan constructs –determined by information state update rules

10. Dialogue plan constructs action A exec(S) –S is a task/subplan sequence if P then C –P is a proposition and C a construct –If P is (believed to be) true, execute C case(,...,, C n )

11. Actions findout(Q): find the answer to Q raise(Q): raise Q respond(Q): provide answer to Q if there is one in the private beliefs inform(P): inform of P (Q is a question, P a proposition)

12. Questions y/n-question: ?P –”Do you want a return ticket?” –”Do you want to call NN?” wh-question: ?x.P(x) –“What city do you want to go to?” –”Who do you want to search for?” alt-question: {?P 1, ?P 2,..., ?P n } –”Do you want to search, add to, erase from or change the phonebook?”

13. Typical human-computer dialog S: Hello, how can I help you? U: I want price information please S: Where do you want to go? U: Paris S: How do you want to travel? U: A flight please S: When do you want to travel U: April S: what class did you have in mind? … S: The price is $123

14. Dialogue plans for information-seeking dialogue Find out how user wants to travel Find out where user wants to go to Find out where user wants to travel from Find out when user wants to travel … Lookup database Tell user the price

15. Dialogue plan, formal representation findout(?x.transport(x)) findout(?x.dest-city(x)) findout(?x.depart-city(x)) findout(?return) … consultDB(?x.price(x)) respond(?x.price(x))

16. Typical human-human dialogue S(alesman), C(ustomer) S: hi C: flights to paris S: when do you want to travel? C: april, as cheap as possible...

17. Accommodation Lewis (1979): –If someone says something at t which requires X to be in the conversational scoreboard, and X is not in the scoreboard at t, then (under certain conditions) X will become part of the scoreboard at t A: My cat is hungry –B accommodates “A has a cat” Has been applied to referents and propositions, as parts of the conversational scoreboard / information state

18. Question accommodation If questions are part of the information state, they too can be accommodated If –the latest move was an answer, and –there is an action in the plan to ask a matching question, –then, push that question on QUD Requires that the number of possible matching questions is not too large –or can be narrowed down by asking clarification question

19. Update rule for question accommodation QuAcc pre: eff: in($ SHARED.LM, answer(usr, A)) in($ PRIVATE.PLAN, findout(Q)) $ DOMAIN :relevant_answer(Q, A) delete( PRIVATE.PLAN, findout(Q)) push( SHARED.QUD, Q)

20. question accommodation… PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) LU: SPEAKER: Speaker MOVES: assocSet( Move )

21. Task accommodation In some cases, the system may not even know what task the user wants the system to perform If latest move was an answer, and there is currently no plan, –find (in the domain resource) a task and corresponding plan containing a matching question; –accommodate the task and load the appropriate plan Similar to plan recognition, but no dynamic plan reconstruction or assumption that the full plan is shared

22. Update rule for task accommodation taskAcc - An answer move matches a task if the plan contains a question matching the answer - More complex version generates clarification question if the number of plans > 1 pre: eff: in($ SHARED.LM, Move) $ DOMAIN :: match_task(Move, Task, Plan) add( SHARED.BEL, task(Task)) set( PRIVATE.PLAN, Plan)

23. task accommodation… DATABASE= LEXICON = DOMAIN= travel travel_english travel database travel lexicon travel english domain travel PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) LU: SPEAKER: Speaker MOVES: assocSet( Move )

24. Question and task accommodation in information-seeking dialogue S: hi U: flights to paris system finds plan containing appropriate questions, and loads it into the plan field in the information state system accommodates questions: how does user want to travel + where does user want to go, and integrates the answers “flight” and “to paris” system proceeds to next question on plan S: when do you want to travel?

25. An optimistic approach to grounding DPs assume their utterances are understood –If A asks a question with content Q, A will update SHARED.LM with ask(A,Q) If addressee indicates that the previous utterance was not understood (reqRep), backtrack –uses the PRIVATE.TMP field, which contains a copy of the SHARED field from before the latest utterance was made The alternative is a pessimistic approach –If A asks Q, A will not update SHARED.LM until B indicates understanding (implicitly or explicitly)

26. Update rule for optimistic grounding assumeSysMovesGrounded pre: eff: $LATEST-SPEAKER == sys SHARED.LU.SPEAKER := sys clear( SHARED.LU.MOVES ) forall_do( in( LATEST-MOVES, Move), add( /SHARED/LU/MOVES, Move, false) )

27. input inter- pret updateselect gene- rate output PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) LU: SPEAKER: Speaker MOVES: assocSet( Move ) INPUT : String LATEST-MOVES: Set(Move) LATEST-SPEAKER: Speaker NEXT-MOVES: Set(Move) OUTPUT: String optimistic grounding

28. Optimistic approach to acceptance DPs assume their utterances are accepted (and integrated into SHARED) –If ask(A,Q) is in SHARED.LM, A will make Q topmost on SHARED.QUD If addresse indicates rejection, backtrack –using the PRIVATE.TMP field No need to indicate acceptance explicitly; it is assumed The alternative is a pessimistic approach –If A asks a question with content Q, A will wait for an acceptance (implicit or explicit) before putting Q on top of QUD

29. optimistic acceptance PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) LU: SPEAKER: Speaker MOVES: assocSet( Move )

30. Adapted for travel agency and autoroute domains, as well as acting as interface to handheld computer or mobile phone Question and task accommodation to enable mixed initiative Simple “optimistic” grounding strategy Focus intonation based on information state contents Has been extended to handle instructional dialogue (IMDiS) Also being extended to handle negotiative dialogue (SIRIDUS) GoDiS features

31. GoDiS demo

32. Action- and information- oriented dialogue

33. Information- vs. action- oriented dialogue Information oriented dialogue (IOD) has the primary goal of exchanging information, regardless of how this information will be used in future actions Action oriented dialogue (AOD) has the primary goal of one or several participants performing or being obliged to perform an action (or plan, i.e. a complex action)

34. Information-oriented dialogue Information-seeking dialogue (ISD): one DP asks the questions, the other answers them Information-exchange (information oriented) dialogue: both DPs ask questions and provide answers IOD can be seen as a sequence of infoseeking dialogues, possibly with embedded subdialogues shared information is stored in shared.com

35. Action-oriented dialogue In simple AOD, only one participant becomes obliged/comitted to some action or plan Actions can either be performed ”online” while the dialogue is happening, or they may be stored as a plan to be performed after the dialogue (”offline”) Requires a store of obliged actions in the IS ( SHARED.ACTIONS ) Two subtypes; –instructional dialogue –natural command dialogue

36. GoDIS AOD-extended infostate type PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) ACTIONS : stack( Action ) LU:…

37. Instructional dialogue user carries out actions goals of this research: –explore the relation between monologue and dialogue in instructional discourse –investigate possibilities of using domain task plans to generate dialogue plans providing various levels of interactivity –adapt GoDiS to handle instructional dialogue (and monologue) NB. We are not dealing with educational (a.k.a. instructional) dialogue TrindiBook chapter 9; Larsson & Zaenen (2000) SigDial paper

38. Practical motivation for document transformation from Xerox Practical advantages of being able to adapt the “same” document to different uses and media Costly to develop presentations from scratch for each new media

39. Theoretical issues Written discourse and human-human dialogue are the extremes of the scale of interactivity We explore some intermediate levels of interactivity using the TRINDI information state approach How can we model the differences between these levels, independently of domain-specific knowledge? Concepts: –interactivity in monologue and dialogue –interactivity vs. initiative –spoken vs. written discourse –maxim of quantity in monologue and dialogue

40. Degrees of interactivity Dialogue is interactive, since participants can influence each other’s moves Monologue is typically non-interactive Written monologue is minimally interactive; reader can influence order & speed of reading highnone written monologue human-human dialogue spoken monologue inter- activity minimal

41. Interactivity vs. initiative Interactivity does not imply shared initiative –it is possible to have a dialogue with both task and dialogue initiative staying on one side In instructional dialogue, –task initiative is typically held by instructor; –dialogue initiative may shift

42. Spoken vs. written instructional discourse separate dimension from interactivity (and initiative) written monologue has advantage of providing minimal interactivity spoken monologue (e.g. tape recording) has advantage of leaving hands and eyes free to do other things –prerecorded repetitions and reformulations can be used as a substitute for interactivity

43. Quantity of information in monologue manuals are cooperative, and are based on implicit assumptions about the reader setting: instructor has know-how information which instructee does not all information, and preferably only that information, which is needed by the instructee to complete the task should be transferred (Grice’s maxim of quantity) –however, if education is an issue, additional information (explanations) should be supplied in manuals, instructor is forced to provide more information than may be necessary for any given instructee

44. Quantity of information in dialogue interactivity makes it possible for instructee to control the amount of information given by the instructor instructee must be able to inform instructor about her state, thus influencing the instructors moves instructor needs to keep track of the state of the instructee to determine what information to provide

45. From written manual to spoken dialogue – 3 levels of interactivity 1. written/spoken monologue –no overt interaction (but minimal interaction in written monologue) 2. recapture minimal interactivity in spoken presentation –system can ask yes/no questions and deals with simple user feedback (“done”, “OK”, “what did you say?”) 3. increase interactivity –user can control level of detail by indicating whether she already knows certain subprocedures –requires more detailed domain task plan

46. Example from a Xerox manual Reinstalling the print head Caution: make sure that the green carriage lock lever is STILL moved all the way forward before you reinstall the print head 1. Line up the hole in the print head with the green post on the printer carriage Lower the print head down gently into position 2. Gently push the green carriage lock lever up until it snaps into place This secures the print head 3. Close the tops cover and re-attach the scanner 4. Press and release the yellow LED button The printer will prepare the cartridge for printing Note: if the carriage does not move from the center position after you press the carriage change button, remove and reinstall the print head

47. monologue mode 2 moves: Instruct, Inform All actions use the update rule autoConfirm –the instructor (GoDiS) assumes all actions are performed by instructee rule autoConfirm pre: eff: val( SHARED.ACTIONS, A ) pop( SHARED.ACTIONS ) add( SHARED.COM, done(A))

48. dialogue mode 8 moves (6 standard GoDiS moves, + Instruct and Confirm) Confirmations are integrated by assuming that the current top- most action of the shared.actions has been performed rule integrateUsrConfirm (slightly simplified) pre: eff: val( SHARED.ACTIONS, A ) pop( SHARED.ACTIONS ) add( SHARED.COM, done(A)) in( SHARED.LU.MOVES, confirm )

49. Example from a Xerox manual (repeated) Reinstalling the print head Caution: make sure that the green carriage lock lever is STILL moved all the way forward before you reinstall the print head 1. Line up the hole in the print head with the green post on the printer carriage Lower the print head down gently into position 2. Gently push the green carriage lock lever up until it snaps into place This secures the print head 3. Close the tops cover and re-attach the scanner 4. Press and release the yellow LED button The printer will prepare the cartridge for printing Note: if the carriage does not move from the center position after you press the carriage change button, remove and reinstall the print head

50. Simple domain plan EFFECT: reinstalled (print_head) NAME: reinstall(print_head) PRECOND: moved_forward(carriage_lock) close(top_cover) reattach(scanner) press_and_release(button) remove(print_head) reinstall(print_head) Final state Action line_up(hole, post) moved_from_center(carriage) N Y lower(print_head) push(lever) condition

51. From domain task plan to monologue and dialogue plans

52. Sample information state PRIVATE =PLAN = AGENDA = { findout(?moved(carriage)) } SHARED = if_then( not moved( carriage ), [ remove(print_head), reinstall(print_head) ] ) COM = moved_forward(carriage_lock) done(secure(print_head)) done(close(top_cover)) done(reattach(scanner)) QUD = LU.MOVES { instruct(sys, press_and_release(yellow_button) } BEL = { } TMP = (same structure as SHARED) ACTIONS=

53. Monologue and dialogue behavior interactivity level 1: (spoken) monologue –the text you have seen above but presented using speech, so the user can direct her attention to the task

54. level 2 - dialogue behavior with minimal interactivity yes/no questions confirm (”ok”) requestRepeat (”what?”) grounding and avoidance of irrelevant information example: S: Press and release the yellow button U: ok S: Has the carriage moved from the center position? U: yes S: the print head is now installed U: what? S: the print head is now installed

55. Complex task plan EFFECT: reinstalled (print_head) NAME: reinstall(print_head) PRECOND: moved_forward(carriage_lock) close(top_cover) reattach(scanner) press_and_release(button) remove(print_head) reinstall(print_head) Final state Complex action Action secure(print_head) moved_from_center(print_head) Condition N Y

56. Subtask plan for complex action EFFECT: secured (print_head) NAME: secure(print_head) PRECOND: - line_up (hole, post) lower(print_head) push(lever)

57. level 3: increased interactivity skipping subtask instructions –S: Put the print head in place –U: Ok, done, what now? –S: Close the top cover requesting subtask instructions; popping out of subdialogue –S: Put the print head in place –U: how? –S: Line up the hole in the print head with the green post on the printer carriage –U: done, I now remember the rest, the print head is secured

58. Domain task plans – where can we get them? What we did: –read the text and construct a corresponding domain plan that seems plausible; –this is not a principled approach Alt.1: Annotating existing manuals with procedural relations –(Scott et. al. 1995); –use this to reconstruct domain plan Alt.2: Author uses authoring tool –specify domain plan (“knowledge model”); –this has been done in the context of multilingual manual authoring (DRAFTER project, GIST project)

59. Conclusions (instructional dialogue) We provide a simple schema modeling the relation between task plan and discourse plans for intermediate interactivity level –The schema also gives a model of how different levels of interactivity (from monologue to dialogue) are related –The degree of interactivity is limited by the level of detail with which the manual encodes the underlying task Document transformation requires separation of task plans and assumptions about interactions –Task plans can be obtained either by marking up existing manuals, or by constructing them using authoring tools Extended existing information-seeking dialogue system to handle instructional dialogue –reusability!

60. Prospect (instructional dialogue) Use existing domain task plans produced by annotating text or using authoring tools TrindiKit dialogue system for information seeking dialogue, adapted to instructional dialogue; uses dialogue plans (NB: current system very simple) Our model sketches a mapping from the domain plans to dialogue plans So, building on existing components, this indicates a possible way to get instructional dialogue systems without a lot of extra work

61. Possible future work on instructional dialogue Explore and implement further levels of interactivity Scale up experiment, e.g. a complete manual Explore use of domain plans generated using existing tools Extend GoDiS to handle referent resolution subdialogues Do real NL generation, possibly using existing components

62. conclusions GoDiS is based on QUD and dialogue plans –QUD has independent motivation in dialogue semantics & ellipsis resolution (Ginzburg) –no planning needed in many domains; use ready- made plans but allow user initiative QUD accommodation –theoretically motivated –enables mixed initiative Demonstrates reusability –Originally implemented for information-oriented dialogue –but straightforward to extend to handle simple action-oriented dialogue Demonstrate exploratory use of TrindiKit –explore relation between instructional text and dialogue

63. Menu-based natural command dialogue (D’Homme) system carries out actions conversion menu 2 dialogue –menu-based interfaces can be automatically or semi-automatically converted into dialogue plans flexible dialogue –dialogue plans are used in a way that enables more flexible interaction than the original menu-based interface Larsson, Cooper, Ericsson (2001): menu2dialog. IJCAI workshop paper.

64. Menus vs. dialogue: Drawbacks of menus User must follow predesigned menu structure this structure may be complicated and difficult to learn menus allow only one activity at a time

65. Menu vs. dialogue: Advantages of flexible dialogue provide information in any order provide information without pre- specifying the task not necessary to learn menu structure doing several things at once

66. Menu components multiple-choice list –the user selects one of several prespecified choices dialogue window –user types requested information tickbox –user ticks box (or not) system message –information from system; user clicks ”OK”

67. Sample menu from a Nokia 3210 mobile phone search phonebook –dialogue window to type in name Name: _______ –message showing the number Kalle 031 12 34 56 –tickbox: whether to call this person Call? –if a call is made, a message: Calling 031 12 34 56

68. Converting menus to plans MenuPlan multi-choice list L 1, L 2,..., L n findout({?L 1,?L 2,...,?L n }); case(,..., ) where T 1 is the task corresponding to L 1 tick-box +/-Pfindout(?P) dialogue window p=___ findout(?x.p(x)) pop-up message M inform(M)

69. Converting menus to plans, cont’d in addition, we need –if-then statements to check conditions and pick out propositions –device actions, e.g. “call up number N” “add name N with number M to phonebook” … these are not currently assumed to be part of the menu specification

70. the menu again search phonebook –dialogue window to type in name Name: _______ –message showing the number Kalle 031 12 34 56 –tickbox: whether to call this person Call? –if a call is made, a message: Calling 031 12 34 56

71. Corresponding plan findout(?x.name(x)); lookup_database(?x.number(x)); if number(N) then respond(?x.number(x) ), findout( ?call ), if call and name(M) then inform(calling_name(M))))

72. Another plan: change ringtone findout(?x.name(x)); findout(?x.ringtone(x)); if name(N) and ringtone(R) then save-ringtone( N, R ), inform(ringtone-changed);

73. Example 1: standard menu navigation U:Go to the phonebook S: Phonebook. Do you want to search, add, or remove an entry, or assign a ringing tone? U: Assign ringing tone S: What name? U: John S: Which ringtone? U: Bossanova S: The ringtone for John is now Bossanova.

74. Example 2: question accommodation U: Assign ringtone S: What name? U: Bossanova for John accommodate ”which ringtone?” S: The ringing tone for John is now Bossanova.

75. Example 3: task accommodation U: Bossanova for John accommodate task: assign ringtone accommodate ”which ringtone?” accommodate ”what name?” S: The ringing tone for John is now Bossanova.

76. Example 4: task clarification U: 5 o’clock –matches two tasks; ask for clarification S: Do you want to set the clock or the alarm clock? U: alarm clock –accommodate ”what time?” S: the alarm clock is now set to 5

77. Example 5 (video): task and question accommodation U: record from 19:30 to 20:00 today –accommodate task: add program –accommodate ”what starting time?” –accommodate ”what end time?” –accommodate ”what day?” S: What channel? U: Seven S: OK, I will record channel 7 from 19:30 to 20:00 today.

78. Example 6: doing several things at once U: Search the phonebook for Jim S: The number is 031 997321 U: Assign ringtone “Attraction” name(jim) is already a shared proposition S: The ringtone for Jim is now Attraction. Do you want to make a call? U: No Note: proposition name(jim) is used by both plans