THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels

Fels: Multimodal Design and Technologies 2 Overview Introduction Human I/O: Interface Examples Bringing Modalities Together Summary

Fels: Multimodal Design and Technologies 3 Introduction Two messages 1.knowledge of modalities motivates design 2.modalities can complement each other – challenges and pitfalls Applications in virtual environments –communication of experience –intimacy and embodiment

Fels: Multimodal Design and Technologies 4 Human Information Processing Input 5 usual senses –vision, hearing, touch, taste, smell position and motion sensing systems Output intentional –neuromuscular, movable, verbal non-intentional / biopotentials –galvanic skin response (GSR), heart rate, brain, muscle activation Cognitive –memory, decision making, tracking, learning

Fels: Multimodal Design and Technologies 5 Visual Display Technologies examples of designs exploiting human capabilities –Virtual Retinal Display (VRD) –Cubby –CAVE Automatic Virtual Environment

Fels: Multimodal Design and Technologies 6 Virtual Retinal Delay (VRD) From HIT lab, U. Of Washington (Furness et al., 1991) Microvision continuing work

Fels: Multimodal Design and Technologies 7 Cubby (Djajadiningrat and Gribnau, 2000)

Fels: Multimodal Design and Technologies 8 CAVE Illustration (U. of Illinois, 1992)

Fels: Multimodal Design and Technologies 9 Haptics Display Technologies examples of touch and force feedback –Pantograph –Phantom –CyberForce

Fels: Multimodal Design and Technologies 10 Pantograph (Hayward, McGill) 2 dof translation - 10cm X 10cm FF/B mouse acceleration best for shock and hard contacts

Fels: Multimodal Design and Technologies 11 Phantom (Massey, SensAble Devices) thimble on finger –single point force feedback –3 degrees of freedom

Fels: Multimodal Design and Technologies 12 CyberForce (Kramer, Immersion Corp.) Exoskelton –arm forces Tendons –grasp forces

Fels: Multimodal Design and Technologies 13 tricks: visual / aural illusions examples of metacognitive gap visual: never show the point penetrating the surface, even if it is aural: play a crisp contact sound on contact this makes the surface appear stiffer/harder (very robust effect) actual: displayed: BUT: if time offset too great, opposite effect

Fels: Multimodal Design and Technologies 14 Other Haptic Devices Tactile feedback Temperature Sheer forces

Fels: Multimodal Design and Technologies 15 challenges for haptic interaction design continuous vs. discrete manual control displaying interaction potential embedding haptic interfaces ensuring tight sensory coupling interacting with other modalities

Fels: Multimodal Design and Technologies 16 Olfactory Devices How do we smell? –different theories: i.e. chemical, infrared absorption –different perceptual mappings: smell prism four odors: fragrant, acrid, burnt and caprylic domain specific: wine, beer, etc. many, many others... –Acuity is great - 10,000 times more sensitive than taste –negative adaptation occurs you get used to the smell

Fels: Multimodal Design and Technologies 17 Olfactory Interfaces Smell camera –Susnick & Raknow, Nature, 2000, 406, Smell display –liquids, gels, microencapsulation Application Research: –Joe Kaye: inStink, Dollars & Scents, Scent Reminder, and more... –Fels, Gauthier, Smith: Interactive Yoga system Smell Display Research and Products: –DigiScents (bankrupt), TriSenx (bankrupt) –DIVEpak (Southwest Research Institute, 1993) –see notes

Fels: Multimodal Design and Technologies 18 Olfaction Display Challenges smell synthesis –many smells vs. mixing base smells control breathing space –sealed room with air filtration –air control in front of and behind user –sealed pod –tethered mask –tubes into an HMD from pack –built into HMD

Fels: Multimodal Design and Technologies 19 Taste taste buds for –sensations of sour, salty, bitter, sweet and umami –receptors not completely resolved umami receptor (Zucker et al., 2002) –extremely complex and poorly understood interacts with olfaction display = food?

Fels: Multimodal Design and Technologies 20 Biopotentials: Examples Galvanic Skin Response (GSR) –Affective Computing, i.e. Galvactinator, Scheirer and Picard Heart Rate (HR) –2 Hearts Musical System, (McCaig and Fels, 2002) Brain activity (EEG) –Brain Computer Interface (BCI) Muscle activity (EMG)

Fels: Multimodal Design and Technologies 21 Multitude of input/output systems –all active at once I/O mechanisms usually depend upon –cognitive context –emotional contexts All these systems available for applications –complement each other Multimodal design looks at: –integration –substitution –complement Summary of Human I/O

Fels: Multimodal Design and Technologies 22 Multimodal Design: Bringing it Together User centred and non-user centred Intimacy and Embodiment –automatic behaviour –sources of aesthetics

Fels: Multimodal Design and Technologies 23 Intimacy and Embodiment Want interfaces that feel “good” to use Humans and machines intimately linked –degree of intimacy supported may determine success Types of relationships: –human to human –human to machine

Fels: Multimodal Design and Technologies 24 Intimacy Intimacy is a measure of subjective match between the behaviour of an object and the control of that object. –extension of “control intimacy” from electronic musical instruments analysis (Moore, 1997) High intimacy implies: –object feels like an extension of self –satisfaction derives from interacting with object –emotional expression flows requires cognitive effort to prevent

Fels: Multimodal Design and Technologies 25 Intimacy, Embodiment and Aesthetics Case 1: Object disembodied from Self Case 2: Self embodies Object Case 3: Self disembodied from Object Case 4: Object embodies Self selfobject self object self objectself Aesthetic Control Result Reflection Belonging

Fels: Multimodal Design and Technologies 26 Intimacy and Embodiment Design Examples Many excellent examples out there of interesting designs –Many at Siggraph: Jam-o-Drum (Blaine et al.), Wooden Mirror (Rozin), etc. –Tangible Bits work (Ishii et al.) –Ubicomp (Weiser and more) –Wearable computing (Mann and others) –Art, Entertainment –Virtual reality

Fels: Multimodal Design and Technologies 27 Application: Iamascope (Fels and Mase, 1997)

Fels: Multimodal Design and Technologies 28 Iamascope Overview Video

Fels: Multimodal Design and Technologies 29 FlowField: Semantics of Caress ( Chen, Fels and Min, 2003) Whole hand interaction in VR –use multi-touch sensitive pad –immersive display Idea: –allow users direct manipulation of fluid use particle simulation for fluid –aesthetics was important hand manipulation on hard surface mapped to obstructions in the flow field

Fels: Multimodal Design and Technologies 30 FlowField: Semantics of Caress

Fels: Multimodal Design and Technologies 31 Multimodal interfaces need to consider –human information processing –matching interface to task use complementary modes where appropriate –intimacy and embodiment Plenty of research opportunities Summary

Fels: Multimodal Design and Technologies 32

THE UNIVERSITY OF BRITISH COLUMBIA Thank-you! Please fill out your comments forms.