1. A Transformational Approach for Developing Multimodal Web User Interfaces Adrian Stanciulescu

2. Plan Context Motivations Thesis statement Requirements for multimodal web UIs Proposal Reference framework for multimodal web UIs Design space for multimodal web UIs Transformational method for producing multimodal UIs Tool support Case studies Conclusion & Future work

3. Context Graphical user interfaces – classical human-computer interface paradigm Users with limited literacy and typing skills Handheld devices Need for hands and eyes free Difficult to support mouse and keyboard interaction Web access anyplace, anywhere, anytime Need for newer, better interface paradigm Multimodal user interface is the answer… Adaptation to the context of use (user, platform, environment) More natural and efficient human-computer interaction

4. Motivations Modality interaction flexibility Faster interaction Lower incidence of errors and esier error recovery True device mobility Usability improvement Robust systems Enhancement of device effectiveness Improved experience Lack of multimodal applications

5. Thesis statement Development of multimodal web user interfaces Three types of user interfaces: Graphical Vocal Multimodal Conceptual and methodological aspects for a model-to-model transformational approach Development life cycle based on a design space composed of design options

6. Requirements Support for multimodal input and output CARE properties support for input and output modalities Approach based on design space Openness to new modalities Separation of modalities

7. Requirements (cont’d) Ontological independence of modalities Transformation-based development Machine processable Human readable Ontological homogeneity Reuse of specification Methodological explicitness Methodological extendibility Support for tool interoperability

8. Reference framework Camelon reference framework UsiXML – selected language: Structured according to four levels of abstraction Progressive transformation from Task & Concept level to Final UI Support for the extension of the transformational methodology (graph transformation) Underlying unique formalism (graph-based syntax) Support for reusability of specification and transformations thanks to the XML structure Support for modality independence Allows introduction of new interaction modality Cross-toolkit development

9. Vocal Concrete Interaction Objects Shortcomings of UsiXML previous version: No specialized containers for user-system dialog No specialized container for multiple selection No differentiations between the system’s output types No object dedicated to system interruptions

10. Vocal Concrete Interaction Objects (cont’d) VocalContainers: VocalGroup VocalForm VocalMenu VocalConfirmation VocalIndividualComponents: VocalOutput: VocalFeedback VocalPrompt VocalMenuItem VocalInput VocalNavigation Break Exit EventTypes: error, help, noInput, noMatch

11. Vocal Concrete Interaction Objects (cont’d) Example: virtual ice cream shop

12. Multimodal Concrete Interaction Objects MultimodalCIO: combination of graphical CIO and vocalCIO outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance groupBox + radioButtons vocalMenu + vocalItems + vocalInput groupBox + radioButtons + vocalMenu + vocalItems + vocalInput + guidance Graphical Vocal Multimodal

13. Concrete UI Relationship GraphicalRelationships GraphicalTransition GraphicalAdjacency GraphicalContainment VocalRelationships VocalTransition VocalAdjacency VocalContainment Synchronization: between vocalCIOs and graphicalCIOs

14. Multimodal Concrete Interaction Objects Syntax example: multimodal text field with equivalent input and redundant output

15. Specification of transformation Graph-based patterns Syntactically typed patterns Textual concrete syntax of patterns: UsiXML syntax

16. How transformation rules work… LHS: Find an occurrence of LHS into G NAC: Check that NAC does not match into G. If there is a match then skip to another occurrence of LHS RHS: Replace LHS by RHS Transformation rule Initial Model Resultant Model

17. UsiXML syntax of transformation rules NAC RHS mappings

18. The four steps of the transformational approach Step 1: Construct the Task&Domain Models Step 2: From Task & Domain to AUI Step 3: From AUI to CUI Step 4: From CUI to FUI

19. Sub-steps of the transformational approach

20. Design space for web UIs Support for the progressive development life cycle towards a final system Composed of a set of design options Each design option has a finite set of design values Advantages: Clarifies the development process in terms of options Facilitates the incorporation in the development life cycle of design options as an abstraction covered by a software Supports the tractability of more complex design problems

21. Design options for graphical web UIs

23. Design options for multimodal web UIs Prompt: vocal (A), graphical (A), multimodal (E, C, R) Grouping/distinction of items: Input: vocal (A), graphical (A), multimodal (E) Output: vocal (A), graphical (A), multimodal (E, C, R) Immediate feedback: vocal (A), graphical (A), multimodal (E, C, R) Guidance: Input: textual (A), Iconic (A), Acoustic (A), Speech (A), Multimodal (E, C, R) Immediate feedback: textual (A), Iconic (A), Acoustic (A), Speech (A), Multimodal (E, C, R)

24. Step 2: From Task & Domain to AUI Separated sub-task presentation Combined sub-task presentation T11T12T13 T1 AC11 AC12 AC13 AC1 T11T12T13 T1 AC11 AC12 AC13 NACLHSRHS NACLHSRHS

25. Step 3: From AUI to CUI Multimodal text field Prompt: multimodal (R) Input: multimodal (E) Immediate feedback: multimodal (R) Guidance: Input: iconic (A) Output: iconic (A) Prompt: multimodal Please say your name (vocal +graphical) 1 Input: multimodal Juan Gonzale z 2 (vocal +graphical) Juan Gonzalez Guidance for input: iconic Guidance for feedback: iconic Feedback:multimodal Your answer was: Juan Gonzalez 3 (graphical +vocal)

26. Transformation rule NAC: LHS: RHS: Multimodal prompt Guidance for input Multimodal input Guidance for output Vocal feedback

27. Case studies 2 case studies: Virtual Polling system and Car rental system

28. Tool support IdealXML TransformiXML GrafiXML CFB Generator Communication Flow Builder XHTML+Voice Generator tool

29. Conclusions Model-drive approach for the development of multimodal web user interfaces Requirements fulfillment: Support for multimodal input/output CARE properties support for input/output modalities Approach based on design options Methodology covers a subset of the UIs which can be developed with X+V

30. Conclusions Advantages of combining design options into a design space: Descriptive Generative Comparative

31. Conclusions Work positioned at the intersection of SE and HCI Focus on the feasibility of code generation (2 case studies) Usability point of view: Inexistent published studies on multimodal UI design in SE and HCI Few ongoing works caused by the reduced number of multimodal applications Internal validation of the Ph.D. (study based on qualitative principles derived from the cognitive psychology)

32. Future work Investigate the implementation of a new transformation engine: MOF QVT Reinforce existing vocal components Extend the ontology with new vocal components Design space improvement Extend transformation catalogs Adaptation of multimodal web user interface to the context of use