1. The Information State approach to dialogue modelling + TrindiKit AI-course, Chalmers April 2002 Staffan Larsson

2. Dialogue modeling and management Information state approach; TrindiKit architecture & concepts what’s in TrindiKit? building a system with TrindiKit other architectures this talk

3. Dialogue modelling Theoretical motivations –find structure of dialogue –explain structure –relate dialogue structure to informational and intentional structure Practical motivations –build dialogue systems to enable natural human-computer interaction –speech-to-speech translation –...

4. Informal approaches to dialogue modelling speech act theory (Austin, Searle,...) –utterances are actions –illocutionary acts: ask, assert, instruct etc. discourse analysis (Schegloff, Sacks,...) –turn-taking, pre-sequences etc. dialogue games (Sinclair & Coulthard,...) –structure of dialogue segments (rather than separate utterances) –can e.g. be encoded as regular expressions or finite automata qna-game -> question qna-game* answer

5. Computational approaches implemented in systems and toolkits finite state automata (CLSU toolkit, Nuance) frame-based (Philips, SpeechWorks) plan-based (TRAINS, Allen, Cohen, Grosz, Sidner,...) general reasoning (Sadek,...) information states (TRINDI: Traum, Bos,...)

6. Why build dialogue systems? theoretical: test theories –e.g. what kind of information does the system need to keep track of? –problem: complex system with many components practical: natural language interfaces –databases (train timetables etc) –electronic devices (mobile phones,...) –instructional/helpdesk systems –booking flights etc –tutorial systems

7. What does a system need to be able to do? speech recognition parsing, syntactic and semantic interpretation –resolve ambiguities –anaphora and ellipsis resolution, etc... dialogue management –how does an utterance change the state of the dialogue? –given the current state of the dialogue, what should the system do? natural language generation speech synthesis

8. Why spoken dialogue? Spoken dialogue is the natural way for people to communicate –computers should adapt to humans rather than the other way around important to enable system and user to communicate in a natural (human-like) way –mixed initiative –turntaking, feedback, barge-in –handle embedded subdialogues –...

9. What’s happening with dialogue systems Beginning to be used commercially Limited domains –need to encode domain-specific knowledge; a general system would require general world knowledge –speech recognition is harder with large lexicon Simple dialogue types –mostly information-seeking Need to bridge gap between dialogue theory and working systems

10. The Information State Approach; TrindiKit architecture & concepts

11. The information state approach – key concepts Information states represent information available to dialogue participants, at any given stage of the dialogue Dialogue moves trigger information state updates, formalised as information state update rules Update rules consist of conditions and operations on the information state Dialogue move engine updates the information state based on observed moves, and decides on next move(s)

12. What is TrindiKit? a toolkit for – building and experimenting with dialogue move engines and systems, – based on the information state approach

13. Key ideas use of global information state –all modules can access all information –enable e.g. context sensitive interpretation distributed decision making asynchronous interaction thinking in terms of IS updates –update rules functions IS (+moves)  IS (+moves) generic task-independent dialogue management –requires modularity

14. module 1 module … Total Information State (TIS) Information state proper (IS) Module Interface Variables Resource Interface Variables resource 1 control module i module j module … module n resource … resource m DME

15. an abstract data structure (record, DRS, set, stack etc.) accessed by modules using conditions and updates the Total Information State (TIS) includes –Information State proper (IS) –Module Interface variables –Resource Interface variables Information State (IS)

16. module or group of modules responsible for –updating the IS based on observed moves –selecting moves to be performed dialogue moves are associated with IS updates using IS update rules –there are also update rules no directly associated with any move (e.g. for reasoning and planning) update rules: rules for updating the TIS –rule name and class –preconditon list: conditions on TIS –effect list: updates to TIS update rules are coordinated by update algorithms Dialogue Move Engine (DME)

17. Modules (dialogue move engine, input, interpretation, generation, output etc.) –access the information state –no direct communication between modules only via module interface variables in TIS modules don’t have to know anything about other modules increases modularity, reusability, reconfigurability –may interact with user or external processes Resources (device interface, lexicons, domain knowledge etc.) –hooked up to the information state (TIS) –accessed by modules –defined as object of some type (e.g. ”lexicon”) Modules and resources

18. What’s in TrindiKit?

19. What does TrindiKit provide? High-level formalism and interpreter for implementing dialogue systems –promotes transparency, reusability, plug- and-play, etc. –allows implementation and comparison of dialogue theories –hides low-level software engineering issues Ready-made modules and resources –speech –interfaces to databases, devices, etc. –reasoning, planning

20. a library of datatype definitions (records, DRSs, sets, stacks etc.) –user extendible a language for writing information state update rules (improved version in 3.0) GUI: methods and tools for visualising the information state (coming in 3.0) debugging facilities –typechecking –logs of communication modules-TIS –etc. TrindiKit contents (1)

21. A language for defining update algorithms used by TrindiKit modules to coordinate update rule application A language for defining basic control structure, to coordinate modules A library of basic ready-made modules for input/output, interpretation, generation etc.; A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc. TrindiKit contents (2)

22. Special modules and resources included with TrindiKit Speech recognition and synthesis modules –TrindiKit shells for off-the-shelf recognizers and synthesizers –currently ViaVoice, Nuance, Festival OAA interface resource –enables interaction with existing software (e.g. Longbow planner, in Lisp) Possible future modules –robust parser –planning and reasoning modules –multimodal input and output

23. Asynchronous TrindiKit Internal communication uses either –OAA (Open Agent Architecture) from SRI, or –AE (Agent Environment), a stripped-down version of OAA, implemented for TrindiKit enables asynchronous dialogue management –e.g.: system can listen and interpret, plan the dialogue, and talk at the same time

24. TrindiKit and OAA TrindiKit uses a stripped-down version AE (Agent Architecture) OAA can be used with TrindiKit in various ways –use OAA as base for TrindiKit instead of AE; TrindiKit modules and infostate are OAA agents –Whole TrindiKit system is an OAA agent –TrindiKit modules are OAA agents or AE agents

25. How to build a system

26. TRINDIKIT dialogue theory (IS, rules, moves etc) domain knowledge (resources) domain-specific system Relation TrindiKit – dialogue system domain-independent DME software engineering (basic types, control flow)

27. Come up with a nice theory of dialogue Formalise the theory, i.e. decide on –Type of information state (DRS, record, set of propositions, frame,...) –A set of dialogue moves –Information state update rules, including rules for integrating and selecting moves –DME Module algorithm(s) and basic control algorithm –any extra datatypes (e.g. for semantics: proposition, question, etc.) Building a domain-independent Dialogue Move Engine

28. Domain independence of the Dialogue Move Engine The DME is domain independent, given a certain type of dialogue –information-seeking –instructional –negotiative –... Domain independence of DME is not enforced by TrindiKit, but is good practice –promotes reuse of components –forces abstraction from domain-specific details, resulting in a more general theory of dialogue

29. Specifying Infostate type the Total Information State contains a number of Information State Variables –IS, the Information State ”proper” –Interface Variables used for communication between modules –Resource Variables used for hooking up resources to the TIS, thus making them accessible from to modules use prespecified or new datatypes

30. sample infostate type declaration infostate_variable_of_type( is, IS ) :- IS = record( [ private : Private, shared : Shared ] ), Shared = record( [ com : set( proposition ), qud : stack( question ), lm : set( move ) ] ), Private = record( [ agenda: stack( action ), plan : stackset( action ), bel : set( proposition ), tmp : Shared ] ) ] ).

31. example infostate type PRIVATE : PLAN : Stack( Action ) AGENDA : OpenQueue( Action ) SHARED : BEL : Set( Prop ) TMP : (same type as SHARED) COM : Set( Prop ) QUD : OpenStack( Question ) LM: Set( Move )

32. Sample interface variable type declarations interface_variable_of_type( input, string ). interface_variable_of_type( output, string ). interface_variable_of_type( latest_speaker, speaker ). interface_variable_of_type( latest_moves, set(move) ). interface_variable_of_type( next_moves, set(move) ).

33. Specifying a set of moves amounts to specifying objects of type move (a reserved type) –there may be type constraints on the arguments of moves preconditions and effects of moves –formalised in update rules, not in the move definition itself –a move may have different effects on the IS depending e.g. on who performed it

34. sample move specifications % Social of_type( quit, move ). of_type( greet, move ). of_type( thank, move ). % Q&A of_type( ask(Q), move ) <- of_type( Q, question ). of_type(inform(P), move ) <- of_type( P, proposition). of_type( answer(R), move ) <- of_type( R, proposition) or of_type( R, ellipsis ).

35. Writing rules rule = conditions + operations –if the rule is applied to the TIS and its conditions are true, the operations will be applied to the TIS –conditions may bind variables with scope over the rule (prolog variables, with unification and backtracking)

36. A sample rule rule( integrateUsrAnswer, [ $/shared/lu/speaker = usr, in( $/shared/lu/moves, answer(R) ), fst( $/shared/qud, Q ), $domain : relevant_answer( Q, R ), $domain : reduce(Q, R, P) ], [ pop( /shared/qud ), add( /shared/com, P ) ] ).

37. Writing rules, cont’d available conditions and operations depend on the infostate type –the infostate is declared to be of a certain (usually complex) type datatype definitions provide –relations: Rel(Arg1, …, ArgN) in( $/shared/lu/moves, answer(R) ), –functions: $$Fun(Arg1, …, ArgN) –selectors: Obj/Sel –operations: Op(Arg1, …, ArgN) add( /shared/com, P ) New datatypes may be added

38. Writing rules: locations in TIS objects may be specified by giving a path to a location in the infostate; –paths are specified using selectors, which are similar to functions $Fun2($Fun1) ~ $Sel1/Sel2 $fst($/shared/qud) ~ $/shared/qud/fst –”$” evaluates a path and gives the object at the location specified example: –is/shared/com is a path, pointing to a location in the TIS –$is/shared/com is the object in that location –the is can be left out, giving $/shared/com –if A is a stack, A/fst is the topmost element

39. Writing rules: conditions (1) conditions do not change the information state if a condition fails, backtracking ensues condition syntax (incomplete) –Rel(Arg1, …, ArgN), e.g. fst($/shared/qud,Q) –Arg1:Rel(Arg2,…,ArgN), e.g. $/shared/qud:fst(Q) $domain:relevant_answer(Q,A) –Arg1 = Arg2 Q = $fst($/shared/qud) –Cond1 and Cond2 –Cond1 or Cond2 –not Cond1 –forall(Cond1, Cond2) –(Arg is object or prolog variable)

40. Writing rules: conditions (2) quantification, binding and backtracking –if an instantiation a of a variable V in a condition C is found that makes condition C true, V is bound to a –backtracking occurs until a successful instantiation of all variables in the list of conditions has been found example list of conditions fst($/shared/qud,Q), in($/shared/com,P), $domain:relevant_answer(P,Q) Explicit quantification  Q.  P. fst($/shared/qud,Q) & in($/shared/com,P) & $domain:relevant_answer(P,Q)

41. Writing rules: updates updates change the information state if an update fails, an error is reported variable bindings survive from conditions to updates update syntax (incomplete) –Op(Path,Arg1,…,ArgN) push(/shared/qud, Q) –Path : Op(Arg1, …,ArgN) /shared/qud : push(Q) –Store := $Fun(Obj,Arg1,…,ArgN) /private/tmp/qud := push($/shared/qud,Q)

42. Specifying update algorithms uses rule classes constructs include –Rule –RuleClass –if Cond then S else T –repeat R until C –repeat R –try R –R orelse S –test C –SubAlgorithm

43. Sample update algorithm grounding, if $latest_speaker == sys then try integrate, try database, repeat downdate_agenda, store else repeat integrate orelse accommodate orelse find_plan orelse if (empty#rec( private^agenda ) then manage_plan else downdate_agenda repeat downdate_agenda if empty($/private/agenda)) then repeat manage_plan repeat refill_agenda repeat store_nim try downdate_qud

44. Specifying serial control algorithms serial constructs include –Module{:Algorithm} –if Cond then S else T –repeat R until C –repeat R –try R –R orelse S –test C –SubAlgorithm

45. Sample control algorithm (1) repeat ( [ select, generate, output, update, test( $program_state == run ), input, interpret, update ] )

46. Specifying concurrent control algorithms Agent 1 | Agent 2 | … | Agent N where Agent i is AgentName : { –import Module 1, – … –import Module p, –Trigger 1 => SerialAlgoritm 1, –… –Trigger m => SerialAlgoritm m } triggers: –condition(C) (C is a subset of the full condition set) –init –new_data(Stream)

47. Sample control algorithm (2) input: { init => input:display_prompt, new_data(user_input) => input } | interpretation: { import interpret, condition(is_set(input)) => [ interpret, print_state ] } | dme: { import update, import select, init => [ select ], condition(not empty(latest_moves)) => [ update, if val(latest_speaker,usr) then select else [] ] } | generation: { condition(is_set(next_moves)) => generate } | output: { condition(is_set(output)) => output } )).

48. From DME to dialogue system Build or select from existing components: Modules, e.g. –input –interpretation –generation –output Still domain independent the choice of modules determines e.g. the format of the grammar and lexicon

49. Domain-specific system Build or select from existing components: Resources, e.g. –domain (device/database) interface –dialog-related domain knowledge, e.g. plan libraries etc. –grammars, lexicons

50. Building resources Resource –the resource itself; exports a set of predicates Resource interface –defines the resource as a datatype T, i.e. in terms of relations, functions and operations Resource interface variable –a TIS variable whose value is an object of the type T By changing the value of the variable, resources can be switched dynamically –change laguage –change domain

51. sample resource variable type declarations (incl. resource interface) resource_type( lexiconT ). resource_variable_of_type( lexicon, lexiconT ). of_type( lexicon_travel_english, lexiconT ). of_type( lexicon_autoroute_english, lexiconT ). of_type( lexicon_travel_svenska, lexiconT ). of_type( lexicon_cellphone_svenska, lexiconT ). resource_condition(lexiconT,input_form(Phrase,Move),Lexicon) :- Lexicon : input_form( Phrase, Move ). resource_condition(lexiconT,output_form(Phrase,Move),Lexicon):- Lexicon : output_form( Phrase, Move ).

52. Explicit information state datastructure makes systems more transparent Update rules provide an intuitive way of formalising theories in a way which can be used by a system Domain knowledge encoded in resources; –the rest of the system is domain independent –resources can be switched dynamically Modular architecture promotes reuse TrindiKit Features

53. Features, cont’d Allows both serial and asynchronous systems –however, asynchronicity remains largely unexplored Interfaces to OAA Runs on UNIX, Windows, Linux Needs SICStus Prolog –but considering moving to YAP prolog (freely available Sicstus clone) Version 2.0 is available, next version coming soon… www.ling.gu.se/projects/trindi/trindikit Larsson & Traum: NLE Special Issue on Best Practice in Dialogue Systems Design, 2000

54. GoDiS – information state based on Questions Under Discussion (Larsson et al 2000) –currently being reimplemented for thesis MIDAS – DRS information state, first-order reasoning (Bos & Gabsdil, 2000) EDIS – information state based on PTT (Matheson et al 2000) –extended to handle tutorial dialogue by Moore, Zinn, Core et al SRI Autoroute – information state based on Conversational Game Theory (Lewin 2000); robust interpretation (Milward 2000) Systems developed during TRINDI

55. Post-TRINDI applications SIRIDUS –connected to speech input and output –command and negotiative dialogues –Spanish, Swedish –GoDiS, SRI system D’Homme (EU 2001) –Dialogues in the Home Environment –GoDiS, SRI system Instruction Based Learning for mobile robots (U Edinburgh) –MIDAS Tutoring Dialogue (U Edinburgh) –BEETLE (based on EDIS) Student projects (Gothenburg) adapting GoDiS to various domains

56. TrindiKit in SIRIDUS added modules for connecting speech improved update rule language GUI –monitoring dialogue –generate dialogue printouts incl. infostat improved debugging facilities –tracing (other than Sicstus trace) –typechecking extending coverage of individual systems to AOD, ND

57. Other architectures

58. DARPA Communicator Open Agent Architecture SOAR Verbmobil VoiceXML What is supported? (What is possible? – less interesting)

59. relevant properties support for information state approach –which datastructures/datatypes are supported? support for managing control flow –not limited to pipelining asynchronous processing modularity potential for scalability

60. DARPA Communicator Hub-and-spoke architecture modularity, flexible control, asynchronous processing hub functions –message routing between servers (spokes) –state maintenance –control flow; script decides which server to call next

61. DARPA communicator cont’d limited support for information states –”tokens”: frames with name, index, and list of feature-value pairs –tokens processed by scripts set of rules determining when to call a server + which arguments (features) to pass Hub scripts too limited to do real dialogue management –most actual systems have separate dialogue manager –hub mostly used for data routing

62. OAA ”Facilitator” (hub) manages communication between ”agents” (spokes) facilitator can maintain global information state –what datastructures are available? asynchronous processing modular similar to scriptless version of DARPA Communicator

63. SOAR toolkit for modeling cognitive agents similar to TrindiKit in some respects keeps single information state –however, only one datastructure whereas TrindiKit has extendible range of (possibly complex) datatypes update rules –some nice ways of triggering rules which can be considered for TrindiKit supports ”chunking” of rules

64. Verbmobil not dialogue system per se several local information states, not one global –”partitioned blackboard” –which datastructures? extendible? scalability limited control support

65. VoiceXML Frame-based dialogue manager not really an architecture, more like a domain-independent dialogue system Requires scripting dialogues in detail Combine with GoDiS? –use GoDiS as VoiceXML server, dynamically building VoiceXML scripts –use VoiceXML specifications converted to GoDiS