1. Goteborg University Dialogue Systems Lab Using TrindiKit and GoDiS as OAA resources TALK Edinburgh 7/10 2004

2. Goteborg University Dialogue Systems Lab Some of our WP5 work Improved GUI –logging & monitoring dialogue, incl. recorded speech Further integration with OAA –OAA API for TrindiKit –GoDiS DME as OAA agent –can be used as ”black box”

3. Goteborg University Dialogue Systems Lab TrindiKit 3.2 GUI Example from GoDIS-VCR corpus TrindiGui displaying a GoDiS-VCR session Audio files linked with utterances Access to information state at different stages of the dialogue (see demo)

4. Goteborg University Dialogue Systems Lab TrindiKit 4, GoDiS, and OAA TrindiKit 4 can be seen as a tool for building ISU dialogue systems as collectives of OAA agents TrindiKit 4 enables building OAA agents containing one or more of –information state (currently only one agent can have IS) –modules (parser etc.) –resources (databases etc.) A simple example OAA-based system: –TrindiKit used to build DME –other OAA agents used for SR, TTS, etc –control handled by control agent (either TrindiKit-style, or ”home- made”) TrindiKit control agent supports asynchronous control via IS triggers The DME can still be run without OAA, as a single Prolog process (useful for debugging)

5. Goteborg University Dialogue Systems Lab interpr. agent IS MIVs RIVs rules device VCR system control updateselect GF agent domain VCR OAA fac. Trindi GUI GoDiS core Graphical input Nuance ASR + TTS module GF module Log VCR text I/O module

6. Goteborg University Dialogue Systems Lab OAA API for TrindiKit 4 Specifies solvables for –TIS agent –Module agent –Resource agent –Control agent

7. Goteborg University Dialogue Systems Lab TIS agents TIS agents include the Total Information State; currently written in prolog Solvables: –tkit_check(+C) : C is a condition or a list of conditions; returns true if conditions hold –tkit_apply(+O) : O is a operation or a list of operations; returns true if successful; failure leads to error message –tkit_reset : resets TIS –tkit_print_state : makes TIS agent print the contents of current TIS –tkit_add_triggers(Ts) : Ts is a list of triggering conditions. Whenever a triggering condition T becomes true, TIS agent calls solvable tkit_trigger(T). –tkit_clear_triggers : Clears all triggers –tkit_load_resource(ResourceFileName, ResourceType) –tkit_connect_resource_agent(ResourceAgentName, ResourceType)

8. Goteborg University Dialogue Systems Lab Module agents Trindikit module agents can be written in any programming language supported by OAA. A module agent defines one or several Trindikit modules, and their algorithm. A module agent can check contents of TIS (handled by TIS agent) by calling TIS solvable tkit_check/1 and also alter the TIS by calling solvables tkit_apply/1 and tkit_apply_rule/3 Solvable: –tkit_call_module(+Module,+Algorithm): make module Module start executing module algorithm Algorithm.

9. Goteborg University Dialogue Systems Lab Resource agents Resources may be part of TIS, or separate agents –if part of TIS, handled by TIS agent Solvables for resource agents include: –tkit_resource_check (+R, +C) : C is a condition or a list of conditions; returns true if conditions hold of resource agent R –tkit_resource_apply(+R, +U) : U is a operation or a list of updates to resource R; returns true if successful; failure leads to error message

10. Goteborg University Dialogue Systems Lab Control agent Standalone SICStus OAA agent used for controlling the execution of Trindikit module algorithms. Reads in a asynchronous control algorithm from a file (either at specified at startup or by calling it's OAA solvable tkit_load_control_algorithm(+File) ). The asynchronous control is algorithm consists of a prolog list of the form [ Trigger1 => SerialAlgorithm1,..., TriggerN => SerialAlgorithmN ].

11. Goteborg University Dialogue Systems Lab Should be possible to read in a serial algo too!? Should also allow calling arbitrary (non-TK) agents? –oaa_Solve(nuance_agent, getspeech(String,Score)), –input:=String, –score:=Score, –gf_agent:parse, –godis_dme:update, –godis_dme:select, –gf_agent:linearize, –$output = Output, –oaa_Solve(nuance_agent, sayspeech(Output))

12. Goteborg University Dialogue Systems Lab When a trigger fires, the control agent will execute the associated (serial) algorithm, possibly in parallel with other serial algorithms which have been triggered before or at the same time. Two types of triggers are currently implemented: –init-triggers which are fired at initialization –condition-triggers which are fired when a TIS variable changes value.

13. Goteborg University Dialogue Systems Lab Solvables –tkit_control_load_algorithm(+File) :loads parallel control algorithm defined in File –tkit_control_run : starts trindikit by sending triggers to TIS-agent and starting to execute the algorithm(s) triggered by init. –tkit_trigger(+Trigger) : causes control agent to start executing all algorithms triggered by Trigger. Called by TIS agent, when a triggering condition becomes true. –tkit_control_stop : stops all serial algorithms and causes control agent to stop responding to tkit_trigger/1 calls.

14. Goteborg University Dialogue Systems Lab Sample GoDiS control algorithm % input runs as a separate (active) agent, not controlled by algorithm, % except for when initializing and quitting % initialisation init => [ input:init, output:init, generate:init, interpret:init, select,generate]. %quitting condition( $program_state == quit) => [ input:quit,output:quit, generate:quit, interpret:quit]. % interpret, update,select and generate run controlled by a single % algorithm which triggers on setting of input variable. condition( is_set(input)) => [ interpret, update, select, if not is_empty($latest_moves) then [generate] ]. % output triggers on setting of output variable condition(not empty($output)) => [ output, update].

15. Goteborg University Dialogue Systems Lab Asynchronicity in TrindiKit4 TrindiKit threaded control agent can be used to set up triggers –if X becomes true of IS, tell agent Y to do Z Other OAA Agents may be –passive, waiting for calls from TrindiKit, or –active, writing to the information state

16. Goteborg University Dialogue Systems Lab TrindiKit itself runs as a single OAA agent; no internal asychronicity Other OAA agents may run asynchronously

17. Goteborg University Dialogue Systems Lab Constraints on system architectures (to avoid e.g. deadlock problems in asynchronous systems) Control agent calls modules –Overall agent control handled outside DME Modules –cannot call other modules –can only be called by control agent –exchange information only via TIS Resources (incl. OAA agents) accessed by TIS update rules, but cannot modify TIS

18. Goteborg University Dialogue Systems Lab Benefits of OAA API TrindiKit and GoDiS become more accessible from OAA TrindiKit becomes less monolithic, more comprehensible, easier to use & learn A TrindiKit system such as GoDiS can be used as a ”black box” in an OAA-based system Allows agents in other languages –connected via ”shell module” in DME (already possible), or –implemented as TrindiKit agent, i.e. able to communicate using the TrindiKit/OAA API (in TrindiKit4) TrindiKit4 will provide Java classes for easy TIS access Potential for rewriting TrindiKit in other programming language, without affecting implemented systems –This would be further strengthened by a Prolog- independent (but equally powerful) TrindiKit language

19. Goteborg University Dialogue Systems Lab GoDiS with GF multimodal grammar and graphical I/O agent (also WP1) Straightforward to connect multiple input and output agents –Active speech input writes to input_speech:string in TIS –Active GUI writes to input_gui:string –GF parser reads from both, writes to latest_moves:queue(dmove) Alternatively –Both agents write to the same queue input:queue(string)

20. Goteborg University Dialogue Systems Lab TrindiKit 4 time plan Preliminary release projected Dec 04 Collect user feedback; maintainence & successive improvements Final version at end of project Contributions from other sites welcome (new agents etc.)

21. Goteborg University Dialogue Systems Lab

22. Differences between TrindiKit 4 and DIPPER Staffan Larsson 8/10 2004 Edinburgh TALK meeting (draft only, not presented)

23. Goteborg University Dialogue Systems Lab Overview Rule language Rule coordination Overall control Accessing other OAA agents Programming language

24. Goteborg University Dialogue Systems Lab Rule language Prolog variables? –TKIT: yes, more expressive language; easier to write rules –DIPPER: no, Prolog independent OAA calls in rule effects? –TKIT: no This should be handled by overall control strategy rather than in rules Still possible to connect OAA agents as resources –but resources cannot affect TIS –need to define them as object of a certiain datatype specifying relations, functions, operations –DIPPER: yes; convenient & simple

25. Goteborg University Dialogue Systems Lab Rule language cont’d Prolog calls? –TKIT: no takes update rule outside scope of TKIT decreases utility of TKIT –e.g. the GUI cannot be used to see what’s happening however, Prolog predicates can be defined in resources –need to define resource as object of a certiain datatype specifying relations, functions, operations –each predicate mus be defined as relation, function,... in the resource interface (one for each resource type) –DIPPER: yes convenient & simple needed since rule lang does not allow variables (except in prolog calls, presumably?)

26. Goteborg University Dialogue Systems Lab Rule language cont’d Explicit evaluation? –TKIT: yes –DIPPER: no

27. Goteborg University Dialogue Systems Lab Rule coordination Rule coordination algorithms? –TKIT: yes When number of rules is large, having an algorithm makes things more manageable, and the rules shorter –DIPPER: no Always possible to determine which rule will trigger next, by inspecting state and rules In TKIT, coordination algorithm execution state will also influence which rules are triggered Multiple DME modules in DME agent? –TKIT: yes DME might not be monolithic; e.g. selection rules only useful after user utterances –DIPPER: no(?)

28. Goteborg University Dialogue Systems Lab Overall control TKIT: –control agent –strict separation of agent types modules: –cannot be called by rules –can access TIS resources: –can be called by rules –cannot access TIS –pros: clean, object-oriented, modular, decrease risk of asynchronicity problems, easier to debug(?) –cons: extra work DIPPER: –control handled by DME –OAA calls encoded in update rules –pros: simple to set up quickly (?) –cons: less clean, higher risk of asychronicity problems, harder to debug(?)

29. Goteborg University Dialogue Systems Lab Accessing other agents DIPPER: –update rules may call other agent using whatever solvables the agent offers –also specifies what to do with the result, e.g. write it to the TIS –so arbitrary OAA agents can be connected without needing modification –passive agents only TKIT –each other agent must be defined as module or resource –module agents need to specify a set of algorithms called by control agent, specifying only algorithm name agent needs to know what to do with the result, i.e. needs to be adapted to TKIT, and perhaps to a particular DME (e.g. GoDiS) TKIT offers active module agents, which are initiated and then run on their own, accessing the TIS –resource agents If connected via TIS, and thus accessible to update rules, need to be specified as object of type, which will provide a resource interface If accessed directly (not going via TIS), accessed using whatever solvables it defines; not accessible to update rules but stil accessible to OAA community and thus to module agents

30. Goteborg University Dialogue Systems Lab Programming language TKIT: SICStus prolog –con: expensive DIPPER: SICStus prolog, Java