1. Introduction to the TrindiKit Dialogue Systems 2 GSLT spring 2003 Staffan Larsson sl@ling.gu.se

2. What is TrindiKit? a toolkit for – building and experimenting with dialogue move engines and systems, – based on the information state approach not a dialogue system!

3. architecture & concepts what’s in TrindiKit? extending TrindiKit building a system This lecture

4. Architecture & concepts

5. module 1 module … Total Information State (TIS) Information state proper (IS) Module Interface Variables Resource Interface Variables resource 1 control module i module j module … module n resource … resource m DME

6. an abstract data structure (record, DRS, set, stack etc.) accessed by modules using conditions and operations the Total Information State (TIS) includes –Information State proper (IS) –Module Interface variables –Resource Interface variables Information State (IS)

7. module or group of modules responsible for –updating the IS based on observed moves –selecting moves to be performed dialogue moves are associated with IS updates using IS update rules –there are also update rules no directly associated with any move (e.g. for reasoning and planning) update rules: rules for updating the TIS –rule name and class –preconditon list: conditions on TIS –effect list: operations on TIS update rules are coordinated by update algorithms Dialogue Move Engine (DME)

8. Modules (dialogue move engine, input, interpretation, generation, output etc.) –access the information state –no direct communication between modules only via module interface variables in TIS modules don’t have to know anything about other modules increases modularity, reusability, reconfigurability –may interact with user or external processes Resources (device interface, lexicons, domain knowledge etc.) –hooked up to the information state (TIS) –accessed by modules –defined as object of some type (e.g. ”lexicon”) Modules and resources

9. What’s in TrindiKit?

10. What does TrindiKit provide? High-level formalism and interpreter for implementing dialogue systems –promotes transparency, reusability, plug- and-play, etc. –allows implementation and comparison of dialogue theories –hides low-level software engineering issues GUI, WWW-demo Ready-made modules and resources –speech –interfaces to databases, devices, etc. –reasoning, planning

11. a library of datatype definitions (records, DRSs, sets, stacks etc.) –user extendible a language for writing information state update rules GUI: methods and tools for visualising the information state debugging facilities –typechecking –logs of communication modules-TIS –etc. TrindiKit contents (1)

12. A language for defining update algorithms used by TrindiKit modules to coordinate update rule application A language for defining basic control structure, to coordinate modules A library of basic ready-made modules for input/output, interpretation, generation etc.; A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc. TrindiKit contents (2)

13. Special modules and resources included with TrindiKit OAA interface resource –enables interaction with existing software and languages other than Prolog Speech recognition and synthesis modules –TrindiKit shells for off-the-shelf recognisers –currently only ViaVoice, but more on the way Possible future modules: –planning and reasoning modules –multimodal input and output

14. Asynchronous TrindiKit Internal communication uses either –OAA (Open Agent Architecture) from SRI, or –AE (Agent Environment), a stripped-down version of OAA, implemented for TrindiKit enables asynchronous dialogue management –e.g.: system can listen and interpret, plan the dialogue, and talk at the same time

15. How to build a system

16. TrindiKit information state approach How to use TrindiKit We start from TrindiKit –Implements the information state approach –Takes care of low-level programming: dataflow, datastructures etc.

17. TrindiKit basic dialogue theory basic system information state approach How to build a basic system Formulate a basic dialogue theory –Information state –Dialogue moves –Update rules Add appropriate modules (speech recognition etc)

18. TrindiKit basic dialogue theory basic system information state approach genre-specific theory additions genre-specific system How to build a genre-specific system Add genre-dependent IS components, moves and rules

19. TrindiKit basic dialogue theory domain & language resources basic system application information state approach genre-specific theory additions genre-specific system How to build an application Add application-specific resources

20. Come up with a nice theory of dialogue Formalise the theory, i.e. decide on –Type of information state (DRS, record, set of propositions, frame,...) –A set of dialogue moves –Information state update rules, including rules for integrating and selecting moves –DME Module algorithm(s) and basic control algorithm –any extra datatypes (e.g. for semantics: proposition, question, etc.) Building a domain-independent Dialogue Move Engine

21. Domain independence of the Dialogue Move Engine The DME is domain independent, given a certain type of dialogue –information-seeking –instructional –negotiative –... Domain independence of DME is not enforced by TrindiKit, but is good practice –promotes reuse of components –forces abstraction from domain-specific details, resulting in a more general theory of dialogue

22. Specifying Infostate type the Total Information State contains a number of Information State Variables –IS, the Information State ”proper” –Interface Variables used for communication between modules –Resource Variables used for hooking up resources to the TIS, thus making them accessible from to modules use prespecified or new datatypes

23. sample infostate type declaration infostate_variable_of_type( is, IS ) :- IS = record( [ private : Private, shared : Shared ] ), Shared = record( [ com : set( proposition ), qud : stack( question ), lm : set( move ) ] ), Private = record( [ agenda: stack( action ), plan : stackset( action ), bel : set( proposition ), tmp : Shared ] ) ] ).

24. resulting infostate type PRIVATE : PLAN : stackset( Action ) AGENDA : stack( Action ) SHARED : BEL : set( Prop ) TMP : (same type as SHARED) COM : set( Prop ) QUD : stack( Question ) LU: [SPEAKER:dp, MOVES:set( Move )]

25. Sample interface variable type declarations interface_variable_of_type( input, string ). interface_variable_of_type( output, string ). interface_variable_of_type( latest_speaker, speaker ). interface_variable_of_type( latest_moves, set(move) ). interface_variable_of_type( next_moves, set(move) ).

26. Specifying a set of moves amounts to specifying objects of type move (a reserved type) –there may be type constraints on the arguments of moves preconditions and effects of moves –formalised in update rules, not in the move definition itself –a move may have different effects on the IS depending e.g. on who performed it

27. sample move specifications % Social of_type( quit, move ). of_type( greet, move ). of_type( thank, move ). % Q&A of_type( ask(Q), move ) <- of_type( Q, question ). of_type(inform(P), move ) <- of_type( P, proposition). of_type( answer(R), move ) <- of_type( R, proposition) or of_type( R, ellipsis ).

28. Writing rules rule = conditions + updates –if the rule is applied to the IS and its conditions are true, the operations will be applied to the IS –conditions may bind variables with scope over the rule (prolog variables, with unification and backtracking)

29. A sample rule rule( integrateUsrAnswer, [ $/shared/lu/speaker = usr, assoc( $/shared/lu/moves, answer(R), false ), fst( $/shared/qud, Q ), $domain : relevant_answer( Q, R ), $domain : reduce(Q, R, P) ], [ set_assoc( /shared/lu/moves, answer(R),true), shared/qud := $$pop( $/shared/qud ), add( /shared/com, P ) ] ).

30. A sample rule (old syntax) rule( integrateUsrAnswer, [ –val#rec( shared^lu^speaker, usr ), –assoc#rec( shared^lu^moves, answer( R ), false ), –fst#rec( shared^qud, Q ), –domain :: relevant_answer( Q, R ), –domain :: reduce(Q, R, P) –], [ –set_assoc#rec( shared^lu^moves, answer(R),true), –pop#rec( shared^qud ), –add#rec( shared^com, P ) ] ).

31. Writing rules available conditions and operations depend on the infostate type –the infostate is declared to be of a certain (usually complex) type datatype definitions provide –selectors: Sel(InObj,SelObj) –relations: Rel(Arg1, …, ArgN) –functions: Fun(Arg1, …, ArgN,Result) –operations: Op(ObjIn,Arg1, …, ArgN,ObjOut) New datatypes may be added

32. Writing rules: locations in TIS objects may be specified by giving a path to a location in the infostate; –paths are specified using selectors, which are similar to functions $$Sel2($$Sel1) ~ $Sel1/Sel2 $$fst($/shared/qud) ~ $/shared/qud/fst –”$” evaluates a path and gives the object at the location specified –”$$” evaluates a function –$$fst($/shared/qud) = $/shared/qud/fst example: –is/shared/com is a path, pointing to a location in the TIS –$is/shared/com is the object in that location –the is can be left out, giving $/shared/com

33. Writing rules: conditions (1) conditions do not change the information state if a condition fails, backtracking ensues condition syntax (incomplete) –Rel(Arg1, …, ArgN), e.g. fst($/shared/qud,Q) –Arg1:Rel(Arg2,…,ArgN), e.g. $/shared/qud:fst(Q) $domain:relevant_answer(Q,A) –Arg1 = Arg2 Q = $$fst($/shared/qud) –Cond1 and Cond2 –Cond1 or Cond2 –not Cond1 –forall(Cond1, Cond2) –(Arg is object or prolog variable)

34. Writing rules: conditions (2) quantification, binding and backtracking –if an instantiation a of a variable V in a condition C is found that makes condition C true, V is bound to a –backtracking occurs until a successful instantiation of all variables in the list of conditions has been found example list of conditions fst($/shared/qud,Q), in($/shared/com,P), $domain:relevant_answer(P,Q) Explicit quantification  Q.  P. fst($/shared/qud,Q) & in($/shared/com,P) & $domain:relevant_answer(P,Q)

35. Writing rules: updates operations change the information state if an operation fails, an error is reported variable bindings survive from conditions to operations operation syntax (incomplete) –Op(Path,Arg1,…,ArgN) push(/shared/qud, Q) –Path : Op(Arg1, …,ArgN) /shared/qud : push(Q) –Store := Fun(Obj,Arg1,…,ArgN) /private/tmp/qud := $$push($/shared/qud,Q)

36. Specifying update algorithms uses rule classes constructs include –Rule –RuleClass –if Cond then S else T –repeat R until C –repeat R –try R –R orelse S –test C –SubAlgorithm

37. Sample update algorithm grounding, if $latest_speaker == sys then try integrate, try database, repeat downdate_agenda, store else repeat integrate orelse accommodate orelse find_plan orelse if (empty ( $/private/agenda ) then manage_plan else downdate_agenda repeat downdate_agenda if empty($/private/agenda)) then repeat manage_plan repeat refill_agenda repeat store_nim try downdate_qud

38. Specifying serial control algorithms serial constructs include –Module{:Algorithm} –if Cond then S else T –repeat R until C –repeat R –try R –R orelse S –test C –SubAlgorithm

39. Specifying concurrent control algorithms Agent 1 | Agent 2 | … | Agent N where Agent i is AgentName : { –import Module 1, – … –import Module p, –Trigger 1 => SerialAlgoritm 1, –… –Trigger m => SerialAlgoritm m } triggers: –condition(C) (C is a subset of the full condition set) –init –new_data(Stream)

40. Sample control algorithm (1) repeat ( [ select, generate, output, update, test( $program_state == run ), input, interpret, update ] )

41. Sample control algorithm (2) input: { init => input:display_prompt, new_data(user_input) => input } | interpretation: { import interpret, condition(is_set(input)) => [ interpret, print_state ] } | dme: { import update, import select, init => [ select ], condition(not empty(latest_moves)) => [ update, if $latest_speaker == usr then select ] } | generation: { condition(is_set(next_moves)) => generate } | output: { condition(is_set(output)) => output } )).

42. From DME to dialogue system Build or select from existing components: Modules, e.g. –input –interpretation –generation –output Still domain independent the choice of modules determines e.g. the format of the grammar and lexicon

43. Domain-specific system Build or select from existing components: Resources, e.g. –domain (device/database) interface –dialog-related domain knowledge, e.g. plan libraries etc. –grammars, lexicons

44. Extending TrindiKit

45. You can add Datatypes –Whatever you need Modules –e.g. General interfaces to speech recognizers and synthesizers Resources –E.g. General interfaces to (passive) devices Important that all things added are reasonably general, so they can be reused in other systsems

46. Datatype definitions relations –relations between objects; true or false –format: relation(Rel,Args). –Example definition: relation(fst,[stack([E|S]),E]). condition: fst($/shared/qud,Q)

47. Datatype definitions functions –functions from arguments to result –format: function(Fun,Args,Result). –Example definition: function(fst,[stack([E|S])],E). in condition: –Q = $$fst($/shared/qud) –Q = $/shared/qud/fst in effect: –next_move/content := $$fst($/shared/qud) –every function corresponds to a relation relation(Fun,[Args@[Result]]).

48. Datatype definitions (2) selectors –selects an object ( Obj ) embedded in another object ( Arg ) –selector(Sel,Arg,Obj,ArgWithHole,Hole ). –e.g. selector(fst,stack([E|S]),E,stack([H| S]),H). –Every selector corresponds to a function function(Sel,[Arg],Object).

49. Datatype definitions (3) operations –operation(Op,InObj,Args,OutObj). –e.g. operation(push,stack(S),[E],stack([E| S])). –every operation corresponds to a relation relation(Op,[InObj|Args]@[OutObj]). –push($/shared/qud,Q,$/shared/qud).

50. Building modules DME modules –Specific to a certain theory of dialogue management –Best implemented using rules and algorithms Other modules –Should be more general, less specific to certain theory of dialogue management –May be easier to implement directly in prolog or other language DME-ADL only covers checking and updating the infostate These modules may also need to interact with other programs or devices

51. Building resources Resource –the resource itself; exports a set of predicates Resource interface –defines the resource as a datatype T, i.e. in terms of relations, functions and operations Resource interface variable –a TIS variable whose value is an object of the type T By changing the value of the variable, resources can be switched dynamically –change laguage –change domain

52. sample resource variable type declarations (incl. resource interface) resource_type( lexiconT ). resource_variable_of_type( lexicon, lexiconT ). of_type( lexicon_travel_english, lexiconT ). of_type( lexicon_autoroute_english, lexiconT ). of_type( lexicon_travel_svenska, lexiconT ). of_type( lexicon_cellphone_svenska, lexiconT ). resource_condition(lexiconT,input_form(Phrase,Move),Lexicon) :- Lexicon : input_form( Phrase, Move ). resource_condition(lexiconT,output_form(Phrase,Move),Lexicon):- Lexicon : output_form( Phrase, Move ).

53. Conclusion

54. explicit information state datastructure –makes systems more transparent –enable e.g. context sensitive interpretation, distributed decision making, asynchronous interaction update rules –provide an intuitive way of formalising theories in a way which can be used by a system –represent domain-independent dialogue management strategies TrindiKit features

55. TrindiKit features cont’d resources –represent domain-specific knowledge –can be switched dynamically e.g. switching language on-line in GoDiS modular architecture promotes reuse –basic system -> genre-specific systems –genre-specific system -> applications

56. Theoretical advantages of TrindiKit theory-independent –allows implementation and comparison of competing theories –promotes exploration of middle ground between simplistic and very complex theories of dialogue intuitive formalisation and implementation of dialogue theories –the implementation is close to the theory

57. Practical advantages of TrindiKit promotes reuse and reconfigurability on multiple levels general solutions to general phenomena enables rapid prototyping of applications allows dealing with more complex dialogue phenomena not handled by current commercial systems

58. technical features interfaces to OAA (but can also run without it) –allows connecting systems to external software system modules can run either serially or in parallell wrappers for off-the-shelf recognizers and synthesizers runs on UNIX, Windows, Linux currently uses SICStus Prolog –but considering moving to shareware Prolog –possibly reimplement in other language –or make it independent of programming language (compilers for several languages)

59. availability version 3.0a avaliable! –SIRIDUS deliverable D6.4 TrindiKit website –www.ling.gu.se/projects/trindi/trindikit SourceForge project –development versions available –developer community? licensed under GPL more info in –Larsson & Traum: NLE Special Issue on Best Practice in Dialogue Systems Design, 2000 –TrindiKit manual (available from website)

60. GoDiS – information state based on Questions Under Discussion (Larsson et al 2000) –currently being reimplemented for thesis MIDAS – DRS information state, first-order reasoning (Bos & Gabsdil, 2000) EDIS – information state based on PTT (Matheson et al 2000) –extended to handle tutorial dialogue by Moore, Zinn, Core et al SRI Autoroute – information state based on Conversational Game Theory (Lewin 2000); robust interpretation (Milward 2000) Systems developed using TrindiKit

61. Recent work D’Homme (EU 2001) –Dialogues in the Home Environment –GoDiS, SRI system Instruction Based Learning for mobile robots (U Edinburgh) –MIDAS Tutoring Dialogue (U Edinburgh) –BEETLE (based on EDIS) Student projects (Gothenburg) adapting GoDiS to various domains

62. TrindiKit in SIRIDUS added modules for connecting speech improved update rule language GUI (in 3.0) –monitoring dialogue –generate dialogue printouts incl. infostat improved debugging facilities (in 3.0) –tracing (other than Prolog trace) –typechecking extending coverage of individual systems to action-oriented and negotiative dialogue

63. Conclusions: TrindiKit & Information State Approach a toolkit for dialogue systems R&D freely available to researchers close the gap between theory and practive of dialogue systems theory-independent promotes reuse and reconfigurability on multiple levels

64. TrindiKit and VoiceXML VoiceXML –industry standard –form-based dialogue manager –web/telephony infrastructure –requires scripting dialogues in detail Theory-specific? –VoiceXML implements a specific theory of dialogue –TrindiKit allows implementation of several different theories of dialogue –More complex dialogue phenomena hard to deal with in VoiceXML

65. TrindiKit and VoiceXML, cont’d Combine VoiceXML with TrindiKit? –future research area –support connecting TrindiKit to VoiceXML infrastructure –use TrindiKit system as VoiceXML server, dynamically building VoiceXML scripts –convert VoiceXML specifications to e.g. GoDiS dialogue plans

66. Possible future modifications regler i klasser väljs indeterministiskt test C -> if C then halt / break (gå ur loop) ta bort möjlighet att ha flera regler med samma namn