6 Complementary expertise Signal Processing UCL MULT Multimodal Interaction ALL Hardware UGLA SWIN Software Engineering UJF UGLA FIT ARC PCLI TXT Application domain: Large information space FT UJF UGLA Application domain: Game ARC FT UJF UGLA Ergonomics Usability FIT FT
7 OpenInterface: objectives Scientific understanding of multimodality Testbeds and Validators Open source platform Extension of standards
8 Multiple modalities Interaction Continuity The 2 action lines Two complementary goals: Extend the sensory-motor capabilities of mobile devices and environments
9 Modalities + The 2 action lines Two complementary goals: Extend the sensory-motor capabilities of mobile devices and environments
10 Standards Validation R&D Ideas Concept Pilots Multimodal Interaction within a pervasive environment Multimodal Interaction on a mobile device Market Concept OpenInterface platform for multimodality Existing standards for multimodality Why 2 action lines?
11 Standards Validation R&D Ideas Concept Pilots Multimodal Interaction within a pervasive environment Multimodal Interaction on a mobile device Market Concept VALIDATORS Why 2 action lines? TESTBEDS
16 OI Platform Open source platform To make it easier to develop effective interactive systems that exploit multimodal interaction techniques in modern application settings (mobile, ubiquitous, context sensitive)
17 OI Platform Open source platform for developing multimodal interfaces that enables quick replication, that enables iterative user-centred design, that contains a rich and extensible set of modalities, that enables a focus on innovation (new modalities or forms of multimodality), that supports dynamic selection and combination of modalities to fit the ongoing context of use.
18 OI Platform - Repository of modalities (components) - Repository of Fusion/Fission Mechanisms (components) - Runtime Kernel Tool for the application designer/developer Using/Assembling the components Tool for the provider of components Adding new components
19 OI platform Who are the users? Three types of users 1. Programmers Providing new components (populating the platform) New device New modality New fusion/fission mechanism 2. Multimodal Application Designers Assembling components to define multimodal interaction 3. End-users of multimodal applications Interacting with the multimodal application Adaptation at run-time Run- time Design time
21 Outlines Kernel Overview OI Components CIDL, PDCL Current Features Next Features
22 Overview Useful high level programming language support: C/C++, Java, Matlab, Python Plug n Play behavior. Lightweight skeleton platform.
23 Components Platform does not specify/define a Component Model (!!) Platform sees the external software as components components are assumed to behave like objects. Software/Component delivered as reusable independent unit with exported/imported I/O interfaces. Platform Kernel aims for the interaction of heterogeneous components
24 Component: OI point of view Facets can be seen as services provided by a component A Component is a set of facets This is how any external software is seen by the Kernel
25 Reusable Independent Unit Software With API With installation/configuration doc Without explicit dependencies with other components Tested and validated Much likely a library (!!) Impossible to test new modalities with an exe, for instance.
26 Exported/Imported IO Components only declare communication interface Enforce the requirement for independence Export IO (functions, etc.) when providing a functionality Display an image Import IO for requesting a functionality An event handler
27 OpenInterface CIDL Designed to describe Independent and Reusable component. (Intentionally) Not as expressive as WSDL (etc.) Restricted on purpose to cover common attributes of various programming languages Describes component interface using a programming language independent description (XML). Not intended to be written by human Generation tools are not yet available though (being developed)
28 I/O Interface Functions provided by a component facet 3 types of functions Sink With no return argument Source With return argument Callback With no return argument, for importing external functions
29 I/O interface: types Only supported languages common types Common primitives types Bool, integer(8,16,32,64bit), float, double, String Array of primitives types Complex types Objects
30 Runtime, Pipeline Pipeline description language to setup a running application initial connections.
31 Pipeline configuration Components Remote or local Connections Simple direct link Multicast (Publisher/Subscriber) (Temporal) Fusion Execution Pin initialization values Threaded execution
32 Current Features Supported Platform Linux and Windows Supported Languages (with minor restrictions) Java C/C++ Matlab Description languages CIDL, interfaces description PDCL, static pipeline description Temporal Fusion components Simple components distribution Specification on how to write a language plug-in Specification on how to write custom connectors
33 Next Features Installation Packages Windows installer, debian package Pipeline Kernel API component for controlling application dynamic connections Description languages tools Semi-Automatic CIDL generator Graphical tool for pipeline composition Supported Platform Other OS Supported Languages.NET framework Connectors More connectors plugins
46 Introduction What? : Graphically create an assembly of (OI) components At first: Technical (PDCL) Description Objective: OI Interface Not only a graphical editor but an IDE Actually: static assembly of components Next version: dynamic assembly of components Why?: Fast prototyping Runtime Testing and Monitoring
47 Demo Fast prototyping Create assembly Run the assembly Change Multimodal Interaction
48 Roadmap Introspection Runtime connection Monitoring of the components execution Usage data capture Link with Conceptual Model Multimodal interaction Multiple OS support