1. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Towards Embedding VoiceXML Applications Through Compilation dirk.buehler@e-technik.uni-ulm.de stefan.hamerich@temic-sds.com (Harman/Becker AG)

2. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Outline Introduction Compilation procedure Preliminary results Conclusions, Future work

3. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Outline Introduction -Motivation -Problems with VXML interpretation Compilation procedure Preliminary results Conclusions, Future work

4. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Application Scenario Automotive Environment Closed Environment - without needs to access dynamic data via web Tasks: control onboard devices such as -Navigation -Radio, CD, MP3 entertainment -Telephone (dialing)  Regarding VoiceXML, a static subset based on forms is sufficient for modeling the needed functionality.

5. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Typical Integration / Automotive Integration into cars currently done as -Hardware Integration -Software Integration Compilation: e.g. GDML (TEMIC dialog description language)  C code  native executable/hardware MTBF (mean time between failures) is an issue: typically > 10 years required

6. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Vielleicht zum erzählen??? Grundsätzlich werden Flash-Speicher verwendet zusätzlich gibt es RAM Consumer-Produkte nicht ausreichend, da Lebensdauer, MTBF „mean time between failure“ und Umgebungsbedingungen nicht ausreichend sind für Automotive

7. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Problems with Traditional VXIs HTTP/file serving for documents VXML/ECMA parsing (SRGS?) ECMA interpretation  Resource consumption and control are problematic VoiceXML Interpreter Document Server Implementation Platform requestdocument

8. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Why the VoiceXML Standard is Relevant Standard for dialogue development Need developers, e.g. external application providers “Buzzword” for clients

9. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 EcmaScript and its Role in VoiceXML ECMA-262: Standardized version of JavaScript (1999) Dynamically typed imperative language Expressive: records + 1 st order functions = objects Compact Profile: ECMA 327 In VoiceXML, it may be used on the client-side for event handling, cf. HTML, simple calculations (e.g. check credit card numbers) and accessing platform objects  VoiceXML specification based on EcmaScript

10. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 „Lines like these“

11. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Outline Introduction Compilation procedure -Approach -VoiceXML example -Form interpretation algorithm Preliminary results Conclusions, Future work

12. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Compilation Approach Off-line document retrieval: VoiceXML dialogs, grammars, scripts No dynamically created documents Compilation to EcmaScript -Copy script code in verbatim mode -Include VoiceXML elements and external scripts Adherence to Compact Profile (Try to) Platform integration through simple APIs for ASR and TTS

13. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Advantages At run-time: No HTTP serving No XML parsing (VoiceXML and grammars) No EcmaScript parsing  Resources, control, optimization, integration  Intellectual property(?) VoiceXML Application Document Server Implementation Platform requestdocument

14. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Translating a VoiceXML Primary components: -items:,,... - handlers - elements VoiceXML specifies a “dialog” data structure, corresponding to the current form. Idea: Generate this data structure to be processed by a separate Form Interpretation Algorithm (FIA).

15. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Translating a item Spec: Item value is represented as an EcmaScript variable in dialog data structure Shadow variables, for storing visit count, events, etc. Important components: -@cond, specifying arbitrary guard conditions -, for ASR (incl. semantics) -, for TTS and audio -event handlers Executable parts may be translated as methods (function objects)

16. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Example: VoiceXML  EcmaScript // EcmaScript document$.pizza_order = { grammars: [ loadGrammarFromURI("pizza.grxml") ], init: function(params$) { [...] dialog.initial1 = (undefined); dialog.initial1$ = { cond: function () { [...] return (document.doInit()); }, grammars: [], prompt: function (count$) { [...] if (true) { prompt ("May I take your order?"); } if (count$ == 2) { prompt("You can say something like: [...]"); } }, handle: function (_event, count$) { [...] if (count$ == 1 && (_event == "noinput")) { $doprompt = true; throw 'continue'; } else if (count$ == 2 && (_event == "noinput")) { initial1 = (’nothing’); $doprompt = true; throw 'continue'; } else throw _event; } }; [...] <initial name="initial1" cond=” document.doInit()”> May I take your order? You can say something like: [...] [...] Entering form “pizza_order” Activate grammar “pizza.grxml” Evaluate “initial1” Evaluate “doInit” Update “doInit$.count” Initiate prompts System: May I take your order? User: Handle exception “noinput” S: You can say something like: [...] U: Handle exception “noinput” S: Would you like...... Entering form “pizza_order” Activate grammar “pizza.grxml” Evaluate “initial1” Evaluate “doInit” Update “doInit$.count” Initiate prompts System: May I take your order? User: Handle exception “noinput” S: You can say something like: [...] U: Handle exception “noinput” S: Would you like...... Entering form pizza_order Activate grammar pizza.grxml Evaluate initial1 Evaluate document.doInit() Update doInit$.count Initiate prompts System: May I take your order? User: Handle exception, reprompt S: You can say something like: [...] U: Handle exception, assign initial1 S: Would you like...... Entering form pizza_order Activate grammar pizza.grxml Evaluate initial1 Evaluate document.doInit() Update doInit$.count Initiate prompts System: May I take your order? User: Handle exception, reprompt S: You can say something like: [...] U: Handle exception, assign initial1 S: Would you like......

17. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Form Interpretation Algorithm Coded completely in EcmaScript While processing a form: Select field to execute, or quit Collect item: -Play (execute) item’s prompts -Pass grammars to recognizer -Determine type of result -and handle exceptions Process result and execute fillers Goto 10 dialog data structure item$, item$.cond() item$.count item$.prompt() item$.grammars application.lastresult$ item$.handle() dialog.filled()

18. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Outline Introduction Compilation procedure Preliminary results -VXML to Java -VXML on PocketPC -Limitations Conclusions, Future Work

19. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Compiler Written in Haskell (GHC) OS independent: Linux, Windows, Solaris,... VoiceXML parsing and recursive document retrieval (about 300 Lines of code) Compilation into text (about 300 Lines of code) ComponentFormatSize [KB] FIA.JAR32 Front-end.JAR57 Rhino.JAR600 Demo (Pizza).VXML6.JS15.JAR42

20. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Experiment 1: VoiceXML to Java Compilation to.class files using RHINO JavaScript interpreter (Mozilla Project) OS independent: Windows, Linux,... Requires: JRE / Java Web Start Integration with FreeTTS, Typed input, SRGS processor written in Java  Shows VXML ECMA is feasible.

21. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Demo: http://it.e-technik.uni-ulm.de/~buehler

22. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Experiment 2: VoiceXML goes PocketPC Compilation to EcmaScript Inclusion in HTML (“Embedding”) Viewing with Pocket Internet Explorer  VoiceXML interpretation on PocketPC  EcmaScript interpretation (without Java)

23. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Demo: http://it.e-technik.uni-ulm.de/~buehler

24. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Experiment 2: Some glitches (Very) Limited grammar support FIA has to yield control, or else page will be in “Loading” state all the time (no interaction) Small modifications in FIA: 1.Save state (form, item) and interrupt execution. 2.On user input, resume with saved state. Still, VoiceXML application on limited device.

25. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Limitations VoiceXML 2.0 subset, similar to X+V: Mixed-initiative forms In-line and external grammars Links, exception handling Limited support for subdialogs Important missing features: Application / document / session handling Glitch: application/document scoping for.

26. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Outline Introduction Compilation procedure Preliminary results Conclusions, Future Work

27. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Conclusions Practical compilation of a subset of VoiceXML No dynamically created documents Easy to use demos, also for off-line use VoiceXML’s dependency on EcmaScript is actually advantageous OS-independent compilation feasible and sensible in certain environments, i.e. towards embedded systems

28. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Future Work Automotive environment: -Compiler for ECMA to Assembler for integration in existing technology (TEMIC) -Embedded Java as environment for SW integration in future cars (BMW Car IT) Platform integration for ASR Extend coverage 1: documents, applications Extend coverage 2: VoiceXML 2.1 goodies, e.g. for obtaining XML data without Simplify compilation: Perhaps XSLT? Useful for “traditional VXIs”: Cache? Go SourceForge?

29. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Thank you!

30. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 FIA in ECMA – SELECT phase function fia(dialog,$params) { //... while(true) { // SELECT var next = undefined; for (var i in dialog.items) { var name = dialog.items[i]; if (dialog[name] == undefined) if (dialog[name+"$"].cond(dialog)) { next = id; break; } }...

31. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 FIA in ECMA – COLLECT phase for (var i in prompts) { var p = prompts[i]; if (p.count == count) p.exec(); } activateLinks(item.links); activateGrammars(item.grammars); try { application.lastresult$ = recognize(); var g = application.lastresult$.grammar; if (checkGrammars(g,item.grammars)) { matchedLocal = true; dialog[next] = assignLocal(next,...); dialog[next+'$'].justfilled = true; }...

32. Universität Ulm Informationstechnik - Dialogsysteme Dirk Bühler, Universität Ulm13.10.2004 Link and Exception Handling Links are grammars that initiate a transition (like ). Exceptions are handled by ECMA try-catch-finally constructs. -However, document-form-field hierarchy of handlers works differently (has to be respected).