1. Goteborg University Dialogue Systems Lab Introduction to TrindiKit Dialogue Systems 2004 Staffan Larsson

2. Goteborg University Dialogue Systems Lab What is TrindiKit? a toolkit for – building and experimenting with dialogue move engines and systems, – based on the information state approach not a dialogue system!

3. Goteborg University Dialogue Systems Lab architecture & concepts what’s in TrindiKit? extending TrindiKit building a system This lecture

4. Goteborg University Dialogue Systems Lab Architecture & concepts

5. Goteborg University Dialogue Systems Lab module 1 module … Total Information State (TIS) Information state proper (IS) Module Interface Variables Resource Interface Variables resource 1 control module i module j module … module n resource … resource m DME

6. Goteborg University Dialogue Systems Lab an abstract data structure (record, DRS, set, stack etc.) accessed by modules using conditions and updates the Total Information State (TIS) includes –Information State proper (IS) –Module Interface variables –Resource Interface variables Information State (IS)

7. Goteborg University Dialogue Systems Lab Dialogue moves abstract description of utterances e.g. speech acts

8. Goteborg University Dialogue Systems Lab module or group of modules responsible for –updating the IS based on observed moves –selecting moves to be performed dialogue moves are associated with IS updates using IS update rules –there are also update rules no directly associated with any move (e.g. for reasoning and planning) update rules: rules for updating the TIS –rule name and class –preconditon list: conditions on TIS –effect list: updates on TIS update rules are coordinated by update algorithms Dialogue Move Engine (DME)

9. Goteborg University Dialogue Systems Lab Modules (dialogue move engine, input, interpretation, generation, output etc.) –access the information state –no direct communication between modules only via module interface variables in TIS modules don’t have to know anything about other modules increases modularity, reusability, reconfigurability –may interact with user or external processes Resources (device interface, lexicons, domain knowledge etc.) –hooked up to the information state (TIS) –accessed by modules –defined as object of some type (e.g. ”lexicon”) Modules and resources

10. Goteborg University Dialogue Systems Lab What’s in TrindiKit?

11. Goteborg University Dialogue Systems Lab What does TrindiKit provide? High-level formalism and interpreter for implementing dialogue systems –promotes transparency, reusability, plug- and-play, etc. –allows implementation and comparison of dialogue theories –hides low-level software engineering issues GUI, WWW-demo Ready-made modules and resources –speech –interfaces to databases, devices, etc. –reasoning, planning

12. Goteborg University Dialogue Systems Lab a library of datatype definitions (records, DRSs, sets, stacks etc.) –user extendible a language for writing information state update rules GUI: methods and tools for visualising the information state debugging facilities –typechecking –logs of communication modules-TIS –etc. TrindiKit contents (1)

13. Goteborg University Dialogue Systems Lab A language for defining update algorithms used by TrindiKit modules to coordinate update rule application A language for defining basic control structure, to coordinate modules A library of basic ready-made modules for input/output, interpretation, generation etc.; A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc. TrindiKit contents (2)

14. Goteborg University Dialogue Systems Lab Special modules and resources included with TrindiKit OAA interface resource –enables interaction with existing software and languages other than Prolog Speech recognition and synthesis modules –TrindiKit shells for off-the-shelf recognisers –currently only ViaVoice, but more on the way Possible future modules: –planning and reasoning modules –multimodal input and output

15. Goteborg University Dialogue Systems Lab Asynchronous TrindiKit Internal communication uses either –OAA (Open Agent Architecture) from SRI, or –AE (Agent Environment), a stripped-down version of OAA, implemented for TrindiKit enables asynchronous dialogue management –e.g.: system can listen and interpret, plan the dialogue, and talk at the same time

16. Goteborg University Dialogue Systems Lab How to build a system

17. Goteborg University Dialogue Systems Lab TrindiKit information state approach How to use TrindiKit We start from TrindiKit –Implements the information state approach –Takes care of low-level programming: dataflow, datastructures etc.

18. Goteborg University Dialogue Systems Lab TrindiKit basic dialogue theory basic system information state approach How to build a basic system Formulate a basic dialogue theory –Information state –Dialogue moves –Update rules Add appropriate modules (speech recognition etc)

19. Goteborg University Dialogue Systems Lab TrindiKit basic dialogue theory basic system information state approach genre-specific theory additions genre-specific system How to build a genre-specific system Add genre-dependent IS components, moves and rules

20. Goteborg University Dialogue Systems Lab TrindiKit basic dialogue theory domain & language resources basic system application information state approach genre-specific theory additions genre-specific system How to build an application Add application-specific resources

21. Goteborg University Dialogue Systems Lab Come up with a nice theory of dialogue Formalise the theory, i.e. decide on –Type of information state (DRS, record, set of propositions, frame,...) –A set of dialogue moves –Information state update rules, including rules for integrating and selecting moves –DME Module algorithm(s) and basic control algorithm –any extra datatypes (e.g. for semantics: proposition, question, etc.) Building a domain-independent Dialogue Move Engine

22. Goteborg University Dialogue Systems Lab Domain independence of the Dialogue Move Engine The DME is domain independent, given a certain type of dialogue –information-seeking –instructional –negotiative –... Domain independence of DME is not enforced by TrindiKit, but is good practice –promotes reuse of components –forces abstraction from domain-specific details, resulting in a more general theory of dialogue

23. Goteborg University Dialogue Systems Lab Specifying Infostate type the Total Information State contains a number of Information State Variables –IS, the Information State ”proper” –Interface Variables used for communication between modules –Resource Variables used for hooking up resources to the TIS, thus making them accessible from to modules use prespecified or new datatypes

24. Goteborg University Dialogue Systems Lab sample infostate type declaration infostate_variable_of_type( is, IS ) :- IS = record( [ private : Private, shared : Shared ] ), Shared = record( [ com : set( proposition ), qud : stack( question ), lm : set( move ) ] ), Private = record( [ agenda: stack( action ), plan : stackset( action ), bel : set( proposition ), tmp : Shared ] ) ] ).

25. Goteborg University Dialogue Systems Lab resulting infostate type PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) LU: [SPEAKER:dp, MOVES:set( Move )]

26. Goteborg University Dialogue Systems Lab Sample interface variable type declarations interface_variable_of_type( input, string ). interface_variable_of_type( output, string ). interface_variable_of_type( latest_speaker, speaker ). interface_variable_of_type( latest_moves, set(move) ). interface_variable_of_type( next_moves, set(move) ).

27. Goteborg University Dialogue Systems Lab Specifying a set of moves amounts to specifying objects of type move (a reserved type) –there may be type constraints on the arguments of moves preconditions and effects of moves –formalised in update rules, not in the move definition itself –a move may have different effects on the IS depending e.g. on who performed it

28. Goteborg University Dialogue Systems Lab sample move specifications % Social of_type( quit, move ). of_type( greet, move ). of_type( thank, move ). % Q&A of_type( ask(Q), move ) <- of_type( Q, question ). of_type( inform(P), move ) <- of_type( P, proposition). of_type( answer(R), move ) <- of_type( R, proposition) or of_type( R, ellipsis ).

29. Goteborg University Dialogue Systems Lab Writing rules rule = conditions + updates –if the rule is applied to the IS and its conditions are true, the updates will be applied to the IS –conditions may bind variables with scope over the rule (prolog variables, with unification and backtracking)

30. Goteborg University Dialogue Systems Lab A sample rule rule( integrateUsrAnswer, [ $/shared/lu/speaker = usr, assoc( $/shared/lu/moves, answer(R), false ), fst( $/shared/qud, Q ), $domain : relevant_answer( Q, R ), $domain : reduce(Q, R, P) ], [ set_assoc( /shared/lu/moves, answer(R),true), shared/qud := $$pop( $/shared/qud ), add( /shared/com, P ) ] ).

31. Goteborg University Dialogue Systems Lab A sample rule (old syntax) rule( integrateUsrAnswer, [ –val#rec( shared^lu^speaker, usr ), –assoc#rec( shared^lu^moves, answer( R ), false ), –fst#rec( shared^qud, Q ), –domain :: relevant_answer( Q, R ), –domain :: reduce(Q, R, P) –], [ –set_assoc#rec( shared^lu^moves, answer(R),true), –pop#rec( shared^qud ), –add#rec( shared^com, P ) ] ).

32. Goteborg University Dialogue Systems Lab Writing rules available conditions and updates depend on the infostate type –the infostate is declared to be of a certain (usually complex) type datatype definitions provide –selectors: Sel(InObj,SelObj) –relations: Rel(Arg1, …, ArgN) –functions: Fun(Arg1, …, ArgN,Result) –operations: Op(ObjIn,Arg1, …, ArgN,ObjOut) New datatypes may be added

33. Goteborg University Dialogue Systems Lab Writing rules: locations in TIS objects may be specified by giving a path to a location in the infostate; –paths are specified using selectors, which are similar to functions $$Sel2($$Sel1) ~ $Sel1/Sel2 $$fst($/shared/qud) ~ $/shared/qud/fst –”$” evaluates a path and gives the object at the location specified –”$$” evaluates a function –$$fst($/shared/qud) = $/shared/qud/fst example: –is/shared/com is a path, pointing to a location in the TIS –$is/shared/com is the object in that location –the is can be left out, giving $/shared/com

34. Goteborg University Dialogue Systems Lab Writing rules: conditions (1) conditions do not change the information state if a condition fails, backtracking ensues condition syntax (incomplete) –Rel(Arg1, …, ArgN), e.g. fst($/shared/qud,Q) –Arg1:Rel(Arg2,…,ArgN), e.g. $/shared/qud:fst(Q) $domain:relevant_answer(Q,A) –Arg1 = Arg2 Q = $$fst($/shared/qud) –Cond1 and Cond2 –Cond1 or Cond2 –not Cond1 –forall(Cond1, Cond2) –(Arg is object or prolog variable)

35. Goteborg University Dialogue Systems Lab Writing rules: conditions (2) quantification, binding and backtracking –if an instantiation a of a variable V in a condition C is found that makes condition C true, V is bound to a –backtracking occurs until a successful instantiation of all variables in the list of conditions has been found example list of conditions fst($/shared/qud,Q), in($/shared/com,P), $domain:relevant_answer(P,Q) Explicit quantification  Q.  P. fst($/shared/qud,Q) & in($/shared/com,P) & $domain:relevant_answer(P,Q)

36. Goteborg University Dialogue Systems Lab Writing rules: updates updates change the information state if an update fails, an error is reported variable bindings survive from conditions to updates update syntax (incomplete) –Op(Path,Arg1,…,ArgN) push(/shared/qud, Q) –Path : Op(Arg1, …,ArgN) /shared/qud : push(Q) –Store := Fun(Obj,Arg1,…,ArgN) /private/tmp/qud := $$push($/shared/qud,Q)

37. Goteborg University Dialogue Systems Lab Specifying update algorithms uses rule classes constructs include –Rule –RuleClass –if Cond then S else T –repeat R until C –repeat R –try R –R orelse S –test C –SubAlgorithm

38. Goteborg University Dialogue Systems Lab Sample update algorithm grounding, if $latest_speaker == sys then try integrate, try database, repeat downdate_agenda, store else repeat integrate orelse accommodate orelse find_plan orelse if (empty ( $/private/agenda ) then manage_plan else downdate_agenda repeat downdate_agenda if empty($/private/agenda)) then repeat manage_plan repeat refill_agenda repeat store_nim try downdate_qud

39. Goteborg University Dialogue Systems Lab Specifying serial control algorithms serial constructs include –Module{:Algorithm} –if Cond then S else T –repeat R until C –repeat R –try R –R orelse S –test C –SubAlgorithm

40. Goteborg University Dialogue Systems Lab Specifying concurrent control algorithms Agent 1 | Agent 2 | … | Agent N where Agent i is AgentName : { –import Module 1, – … –import Module p, –Trigger 1 => SerialAlgoritm 1, –… –Trigger m => SerialAlgoritm m } triggers: –condition(C) (C is a subset of the full condition set) –init –new_data(Stream)

41. Goteborg University Dialogue Systems Lab Sample control algorithm (1) repeat ( [ select, generate, output, update, test( $program_state == run ), input, interpret, update ] )

42. Goteborg University Dialogue Systems Lab Sample control algorithm (2) input: { init => input:display_prompt, new_data(user_input) => input } | interpretation: { import interpret, condition(is_set(input)) => [ interpret, print_state ] } | dme: { import update, import select, init => [ select ], condition(not empty(latest_moves)) => [ update, if $latest_speaker == usr then select ] } | generation: { condition(is_set(next_moves)) => generate } | output: { condition(is_set(output)) => output } )).

43. Goteborg University Dialogue Systems Lab From DME to dialogue system Build or select from existing components: Modules, e.g. –input –interpretation –generation –output Still domain independent the choice of modules determines e.g. the format of the grammar and lexicon

44. Goteborg University Dialogue Systems Lab Domain-specific application Build or select from existing components: Resources, e.g. –domain (device/database) interface –dialog-related domain knowledge, e.g. plan libraries etc. –grammars, lexicons

45. Goteborg University Dialogue Systems Lab Extending TrindiKit

46. Goteborg University Dialogue Systems Lab You can add Datatypes –Whatever you need Modules –e.g. General interfaces to speech recognizers and synthesizers Resources –E.g. General interfaces to (passive) devices Important that all things added are reasonably general, so they can be reused in other systsems

47. Goteborg University Dialogue Systems Lab Datatype definitions relations –relations between objects; true or false –format: relation(Rel,Args). –Example definition: relation(fst,[stack([E|S]),E]). condition: fst($/shared/qud,Q)

48. Goteborg University Dialogue Systems Lab Datatype definitions functions –functions from arguments to result –format: function(Fun,Args,Result). –Example definition: function(fst,[stack([E|S])],E). in condition: –Q = $$fst($/shared/qud) –Q = $/shared/qud/fst in effect: –next_move/content := $$fst($/shared/qud) –every function corresponds to a relation relation(Fun,[Args@[Result]]).

49. Goteborg University Dialogue Systems Lab Datatype definitions (2) selectors –selects an object ( Obj ) embedded in another object ( Arg ) –selector(Sel,Arg,Obj,ArgWithHole,Hole ). –e.g. selector(fst,stack([E|S]),E,stack([H| S]),H). –Every selector corresponds to a function function(Sel,[Arg],Object).

50. Goteborg University Dialogue Systems Lab Datatype definitions (3) operations –operation(Op,InObj,Args,OutObj). –e.g. operation(push,stack(S),[E],stack([E| S])). –every operation corresponds to a relation relation(Op,[InObj|Args]@[OutObj]). –push($/shared/qud,Q,$/shared/qud).

51. Goteborg University Dialogue Systems Lab Building modules DME modules –Specific to a certain theory of dialogue management –Best implemented using rules and algorithms Other modules –Should be more general, less specific to certain theory of dialogue management –May be easier to implement directly in prolog or other language DME-ADL only covers checking and updating the infostate These modules may also need to interact with other programs or devices

52. Goteborg University Dialogue Systems Lab Building resources Resource –the resource itself; exports a set of predicates Resource interface –defines the resource as a datatype T, i.e. in terms of relations, functions and operations Resource interface variable –a TIS variable whose value is an object of the type T By changing the value of the variable, resources can be switched dynamically –change laguage –change domain

53. Goteborg University Dialogue Systems Lab Sample resource variable type declarations (incl. resource interface) resource_type( lexiconT ). resource_variable_of_type( lexicon, lexiconT ). of_type( lexicon_travel_english, lexiconT ). of_type( lexicon_autoroute_english, lexiconT ). of_type( lexicon_travel_svenska, lexiconT ). of_type( lexicon_cellphone_svenska, lexiconT ). resource_condition(lexiconT,input_form(Phrase,Move),Lexicon) :- Lexicon : input_form( Phrase, Move ). resource_condition(lexiconT,output_form(Phrase,Move),Lexicon):- Lexicon : output_form( Phrase, Move ).

54. Goteborg University Dialogue Systems Lab Conclusion

55. Goteborg University Dialogue Systems Lab explicit information state datastructure –makes systems more transparent –enable e.g. context sensitive interpretation, distributed decision making, asynchronous interaction update rules –provide an intuitive way of formalising theories in a way which can be used by a system –represent domain-independent dialogue management strategies TrindiKit features

56. Goteborg University Dialogue Systems Lab TrindiKit features cont’d resources –represent domain-specific knowledge –can be switched dynamically e.g. switching language on-line in GoDiS modular architecture promotes reuse –basic system -> genre-specific systems –genre-specific system -> applications

57. Goteborg University Dialogue Systems Lab Theoretical advantages of TrindiKit theory-independent –allows implementation and comparison of competing theories –promotes exploration of middle ground between simplistic and very complex theories of dialogue intuitive formalisation and implementation of dialogue theories –the implementation is close to the theory

58. Goteborg University Dialogue Systems Lab Practical advantages of TrindiKit promotes reuse and reconfigurability on multiple levels general solutions to general phenomena enables rapid prototyping of applications allows dealing with more complex dialogue phenomena not handled by current commercial systems

59. Goteborg University Dialogue Systems Lab Technical features interfaces to OAA (but can also run without it) –allows connecting systems to external software system modules can run either serially or in parallell wrappers for off-the-shelf recognizers and synthesizers runs on UNIX, Windows, Linux currently uses SICStus Prolog –but considering moving to shareware Prolog –possibly reimplement in other language –or make it independent of programming language (compilers for several languages)

60. Goteborg University Dialogue Systems Lab Availability Version 3.2 avaliable TrindiKit website –www.ling.gu.se/projects/trindi/trindikit SourceForge project –development versions available –developer community? licensed under GPL more info in –Larsson & Traum: NLE Special Issue on Best Practice in Dialogue Systems Design, 2000 –TrindiKit manual (available from website)

61. Goteborg University Dialogue Systems Lab GoDiS – information state based on Questions Under Discussion (Larsson et al 2000) –currently being reimplemented for thesis MIDAS – DRS information state, first-order reasoning (Bos & Gabsdil, 2000) EDIS – information state based on PTT (Matheson et al 2000) –extended to handle tutorial dialogue by Moore, Zinn, Core et al SRI Autoroute – information state based on Conversational Game Theory (Lewin 2000); robust interpretation (Milward 2000) Systems developed using TrindiKit

62. Goteborg University Dialogue Systems Lab Recent work D’Homme (EU 2001) –Dialogues in the Home Environment –GoDiS, SRI system Instruction Based Learning for mobile robots (U Edinburgh) –MIDAS Tutoring Dialogue (U Edinburgh) –BEETLE (based on EDIS) Student projects (Gothenburg) adapting GoDiS to various domains

63. Goteborg University Dialogue Systems Lab TrindiKit in SIRIDUS added modules for connecting speech improved update rule language GUI (in 3.0) –monitoring dialogue –generate dialogue printouts incl. infostat improved debugging facilities (in 3.0) –tracing (other than Prolog trace) –typechecking extending coverage of individual systems to action-oriented and negotiative dialogue

64. Goteborg University Dialogue Systems Lab Conclusions: TrindiKit & Information State Approach a toolkit for dialogue systems R&D freely available to researchers close the gap between theory and practice of dialogue systems theory-independent promotes reuse and reconfigurability on multiple levels

65. Goteborg University Dialogue Systems Lab TrindiKit and VoiceXML VoiceXML –industry standard –form-based dialogue manager –web/telephony infrastructure –requires scripting dialogues in detail Theory-specific? –VoiceXML implements a specific theory of dialogue –TrindiKit allows implementation of several different theories of dialogue –More complex dialogue phenomena hard to deal with in VoiceXML

66. Goteborg University Dialogue Systems Lab TrindiKit and VoiceXML, cont’d Combine VoiceXML with TrindiKit? –future research area –support connecting TrindiKit to VoiceXML infrastructure –use TrindiKit system as VoiceXML server, dynamically building VoiceXML scripts –convert VoiceXML specifications to e.g. GoDiS dialogue plans Implement VoiceXML in TrindiKit –Information state? forms? set of forms? as records? –Dialogue moves? Just one? Or two: block, prompt? –Update rules? implements FIA –Is asynchronous TrindiKit needed? events filler actions –How deal with grammars on different levels? probably need to load SR grammars dynamically Implement TrindiKit system (e.g. GoDiS) in VoiceXML –use JavaScript

67. Goteborg University Dialogue Systems Lab Possible future modifications regler i klasser väljs indeterministiskt test C -> if C then halt / break (gå ur loop) ta bort möjlighet att ha flera regler med samma namn