March 20, 2006 © 2005 IBM Corporation Distributed Multimodal Synchronization Protocol (DMSP) Chris Cross IETF 65 March 21, 2006 With Contribution from Gerald McCobb and Les Wilson

March 20, 2006 © 2005 IBM Corporation DMSP Background Result of 4 year IBM and Motorola R&D effort ID to IETF July 8, 2005 by IBM & Motorola Reason for contribution –A standard is needed to synchronize network based services implementing distributed modalities in multimodal applications –Other protocols may have overlap but do not address all multimodal interaction requirements –Other IETF IDs and RFCs: Media Server Control Protocol (MSCP) LRDP: The Lightweight Remote Display Protocol (Remote UI BoF) Media Resource Control Protocol Version 2 (MRCPv2) Widex RFC 1056 Distributed Mail System for Personal Computers (also DMSP  )

March 20, 2006 © 2005 IBM Corporation Why do you need a distributed system, i.e., a Thin Client? Grammar Size and Complexity G R Resources: memory and MIPS on the client device G Size and Complexity of application grammars R/G = 1Resources are adequate to perform “real time” recognition and synthesis. A thick client has speech recognition and synthesis on the device. As resources available on a device shrink or the application requirements increase (larger application grammars) then the performance of the system becomes unacceptable. When that threshold is reached then it is economically feasible to distribute the speech over the network. Client Resources R R/G = 1 Thick Client Thin Client

March 20, 2006 © 2005 IBM Corporation DMSP Architecture 1.Modalities 2.Model-View-Controller (MVC) design pattern 3.View Independent Model 4.Event-based modality synchronization There are 4 DMSP building blocks:

March 20, 2006 © 2005 IBM Corporation DMSP Architecture Building Blocks 1.Modalities are Views in the MVC Pattern GUI, Speech, Pen Individual browsers for each modality Compound browsers for multiple modalities Compound Browser

March 20, 2006 © 2005 IBM Corporation DMSP Architecture Building Blocks Multimodal system can be modeled in terms of the MVC pattern Each modality can be decomposed and implemented in its own MVC pattern A modality can implement a view independent model and controller locally or use one in the network (e.g., an IM) 2.Model-View-Controller (MVC) design pattern

March 20, 2006 © 2005 IBM Corporation DMSP Architecture Building Blocks View Independent Model Enables a centralized model Modality interaction updates view and model Local event filters reflect “important” events to view independent model A modality listens to view independent model for only the events it cares about Compound clients, centralized control (Interaction Manager) as well as distributed modalities all enabled with a single protocol

March 20, 2006 © 2005 IBM Corporation DMSP Architecture Building Blocks 4.Event-based synchronization Compound Client: All modalities rendered in client Interactions in one modality reflected in others thru event based changes to one or more model GUI DOM serves as View Independent model

March 20, 2006 © 2005 IBM Corporation DMSP Architecture Building Blocks 4.Event-based synchronization (CONT’D) Distributed Modality: A modality is handled in the infrastructure Requires the DMSP for distributing modality Event based synchronization via View Independent Model gives a modality independent distribution mechanism Enables multiple topographies –Compound Client w/ Distributed Modality –Interaction Manager Distributed Modality

March 20, 2006 © 2005 IBM Corporation DMSP Design There are 4 abstract interfaces 1.Command 2.Response 3.Event 4.Signal Each interface defines a set of methods and related data structures exchanged between user agents Specified as a set of messages XML and Binary message encodings

March 20, 2006 © 2005 IBM Corporation DMSP Message Types 1.Signals One-way asynchronous messages used to negotiate internal processing states Initialization (SIG_INIT) VXML Start (SIG_VXML_START) Close (SIG_CLOSE)

March 20, 2006 © 2005 IBM Corporation DMSP Message Types 2.Command and control messages Add and remove event listener (CMD_ADD/REMOVE_EVT_LISTENER) Can dispatch (CMD_CAN_DISPATCH) Dispatch event (CMD_DISPATCH_EVT) Load URL (CMD_LOAD_URL) Load Source (CMD_LOAD_SRC) Get and Set Focus (CMD_GET/SET_FOCUS) Get and Set Fields (CMD_GET/SET_FIELDS) Cancel (CMD_CANCEL) Execute Form (CMD_EXEC_FORM) Get and Set Cookies (CMD_GET/SET_COOKIES)

March 20, 2006 © 2005 IBM Corporation DMSP Message Types 3.Responses Response messages to commands OK (RESP_OK) Boolean (RESP_BOOL) String (RESP_STRING) Fields (RESP_FIELDS) Contains 1 or more Field data structures Error (RESP_ERROR)

March 20, 2006 © 2005 IBM Corporation DMSP Message Types 4.Events Asynchronous notifications between user agents with a common data structure Events correlated with event listeners DOM events DOMActivate, DOMFocusIn, and DOMFocusOut HTML 4 events Click, Mouse, Key, submit, reset, etc Error and abort VXML Done (e.g., VoiceXML form complete)

March 20, 2006 © 2005 IBM Corporation DMSP Message Types 4.Events (CONT’D) Form Data One or more Field data structures (GUI or Voice) Recognition Results One or more Result data structures with raw utterance, score, and one or more Field data structures Recognition Results EX One or more Result EX data structures with raw utterance, score, grammar, and semantics Start and stop play back Play back of audio or TTS prompts has started or stopped Start and stop play back mark TTS encounters a mark in the play text Custom (i.e., application-defined)

March 20, 2006 © 2005 IBM Corporation DMSP Conclusion A protocol dedicated to distributed multimodal interaction Based on the Model-View-Controller design pattern Enables both Interaction Manager and Client based View Independent Model topographies Asynchronous signals and events Command-response messages Can be generalized for other modalities besides GUI and Voice Supports application specific result protocols (e.g. EMMA) through extension TBD Interested in getting more participation