Break-out Group # D Research Issues in Multimodal Interaction

What are the different types Speech Haptics Gesture Deictic Head and eye movement EEG Electrocephalograms physiological measurements

What has been done so far? Semantic fusion of information – Speech and Gesture Preliminary efforts as to what types of modalities to intergrate. This is application dependent. Need standardization at the level of devices and types of information to be fused

Open Research Problems: Should we stay with current paradigms or invent new methodologies? There is no unifying framework for interaction in terms of devices/semantic integration. This is due to the lack of general purpose application. We see specific applications eg, simulation, medical training

Open Research Questions How to deal with specific tasks in terms of fusing channels. How should channels be fused. How to do transitions between tasks, e.g., manipulation vs loccomotion Need more experimentation and a theory as to where VR is needed?

Open Research Questions Formal study of tasks within applications(e.g., manipulation, selection, navigation, changing of attributes, numerical input) Need more research on output. So far mostly visual and oral.

First breakout group Taxonomy Semantics Cross-modal Representations (actions/perceptions)

Applications/Output Group Second Meeting New Issues we Discussed in the afternoon

DM: Third Breakout Group: Applications/Output Human Perception of the environment Integration with Input Relationships to basic principles

Human Perceptive abilities Vision Technology: Limitation in terms or lighting or real time rendering Limitations for other channels: Haptics, audio, olfaction, taste The type/mix of output depends on the application This is related to the internal representation

Continued Issue of using many modalities to offset the limitations of each modality. –Right now we do not have enough research data to support that. Do we or not need to represent exactly the environment? –Application dependent

Continued Abstraction vs exact representation –Application dependent Exact physical simulation vs fake physics. Ok or not to fool the user? –Probably application and technology dependent.

Other Human Perceptive Modalities Olfaction and taste: very little research Some modalities are better understood than others (e.g., visual vs haptic or olfaction)

Continued-Summary Big issues: –Sensory substitution –Level of detail (variable resolution) –Sampled vs synthetic generation –Online or offline computation –Preservation (or not) of individuality e.g two people with different sense of taste or heat etc –Higher-level emotional augmentation

Integration with Input Haptics is the most widely used output sense that is also used for input –Head orientation, whole-body position, eye gaze also Some output must be tightly coupled to input (it’s at the physical level) –Head motion to view changes, 3D audio

Integration with Input (cont.) Eye gaze-based control requires some interpretation Intentional vs unintentional movement –When is a gesture a gesture?

Relationship to Basic Principles Mapping semantics to output –One or multiple representations for all modalities eg. Language and visual output where we have a common representation but gets translated differently for output –Spatio-temporal synchronization –Cross-modal representation (actions/perceptions) Account for individual differences

Future Paper Topics All the previously mentioned open problems Short Term –Update of the NRC report on modalities Medium-Term –Modeling, Coupling and Output of modalities –In particular model smell and taste

Future Paper Topics Long Term –Further modeling and coupling –Advanced display technology –Personalization of output