1. THE UNIVERSITY OF BRITISH COLUMBIA Multimodal Design & Technologies Sidney Fels

2. Fels: Multimodal Design and Technologies 2 Overview Introduction Human I/O: Interface Examples Bringing Modalities Together Summary

3. Fels: Multimodal Design and Technologies 3 Introduction Two messages 1.knowledge of modalities motivates design 2.modalities can complement each other – challenges and pitfalls Applications in virtual environments –communication of experience –intimacy and embodiment

4. Fels: Multimodal Design and Technologies 4 Human Information Processing Input 5 usual senses –vision, hearing, touch, taste, smell position and motion sensing systems Output intentional –neuromuscular, movable, verbal non-intentional / biopotentials –galvanic skin response (GSR), heart rate, brain, muscle activation Cognitive –memory, decision making, tracking, learning

5. Fels: Multimodal Design and Technologies 5 Visual Display Technologies examples of designs exploiting human capabilities –Virtual Retinal Display (VRD) –Cubby –CAVE Automatic Virtual Environment

6. Fels: Multimodal Design and Technologies 6 Virtual Retinal Delay (VRD) From HIT lab, U. Of Washington (Furness et al., 1991) Microvision continuing work

7. Fels: Multimodal Design and Technologies 7 Cubby (Djajadiningrat and Gribnau, 2000)

8. Fels: Multimodal Design and Technologies 8 CAVE Illustration (U. of Illinois, 1992)

9. Fels: Multimodal Design and Technologies 9 Haptics Display Technologies examples of touch and force feedback –Pantograph –Phantom –CyberForce

10. Fels: Multimodal Design and Technologies 10 Pantograph (Hayward, McGill) 2 dof translation - 10cm X 10cm FF/B mouse acceleration best for shock and hard contacts

11. Fels: Multimodal Design and Technologies 11 Phantom (Massey, SensAble Devices) thimble on finger –single point force feedback –3 degrees of freedom

12. Fels: Multimodal Design and Technologies 12 CyberForce (Kramer, Immersion Corp.) Exoskelton –arm forces Tendons –grasp forces

13. Fels: Multimodal Design and Technologies 13 tricks: visual / aural illusions examples of metacognitive gap visual: never show the point penetrating the surface, even if it is aural: play a crisp contact sound on contact this makes the surface appear stiffer/harder (very robust effect) actual: displayed: BUT: if time offset too great, opposite effect

14. Fels: Multimodal Design and Technologies 14 Other Haptic Devices Tactile feedback Temperature Sheer forces

15. Fels: Multimodal Design and Technologies 15 challenges for haptic interaction design continuous vs. discrete manual control displaying interaction potential embedding haptic interfaces ensuring tight sensory coupling interacting with other modalities

16. Fels: Multimodal Design and Technologies 16 Olfactory Devices How do we smell? –different theories: i.e. chemical, infrared absorption –different perceptual mappings: smell prism four odors: fragrant, acrid, burnt and caprylic domain specific: wine, beer, etc. many, many others... –Acuity is great - 10,000 times more sensitive than taste –negative adaptation occurs you get used to the smell

17. Fels: Multimodal Design and Technologies 17 Olfactory Interfaces Smell camera –Susnick & Raknow, Nature, 2000, 406, 710- 714. Smell display –liquids, gels, microencapsulation Application Research: –Joe Kaye: inStink, Dollars & Scents, Scent Reminder, and more... –Fels, Gauthier, Smith: Interactive Yoga system Smell Display Research and Products: –DigiScents (bankrupt), TriSenx (bankrupt) –DIVEpak (Southwest Research Institute, 1993) –see notes

18. Fels: Multimodal Design and Technologies 18 Olfaction Display Challenges smell synthesis –many smells vs. mixing base smells control breathing space –sealed room with air filtration –air control in front of and behind user –sealed pod –tethered mask –tubes into an HMD from pack –built into HMD

19. Fels: Multimodal Design and Technologies 19 Taste taste buds for –sensations of sour, salty, bitter, sweet and umami –receptors not completely resolved umami receptor (Zucker et al., 2002) –extremely complex and poorly understood interacts with olfaction display = food?

20. Fels: Multimodal Design and Technologies 20 Biopotentials: Examples Galvanic Skin Response (GSR) –Affective Computing, i.e. Galvactinator, Scheirer and Picard Heart Rate (HR) –2 Hearts Musical System, (McCaig and Fels, 2002) Brain activity (EEG) –Brain Computer Interface (BCI) Muscle activity (EMG)

21. Fels: Multimodal Design and Technologies 21 Multitude of input/output systems –all active at once I/O mechanisms usually depend upon –cognitive context –emotional contexts All these systems available for applications –complement each other Multimodal design looks at: –integration –substitution –complement Summary of Human I/O

22. Fels: Multimodal Design and Technologies 22 Multimodal Design: Bringing it Together User centred and non-user centred Intimacy and Embodiment –automatic behaviour –sources of aesthetics

23. Fels: Multimodal Design and Technologies 23 Intimacy and Embodiment Want interfaces that feel “good” to use Humans and machines intimately linked –degree of intimacy supported may determine success Types of relationships: –human to human –human to machine

24. Fels: Multimodal Design and Technologies 24 Intimacy Intimacy is a measure of subjective match between the behaviour of an object and the control of that object. –extension of “control intimacy” from electronic musical instruments analysis (Moore, 1997) High intimacy implies: –object feels like an extension of self –satisfaction derives from interacting with object –emotional expression flows requires cognitive effort to prevent

25. Fels: Multimodal Design and Technologies 25 Intimacy, Embodiment and Aesthetics Case 1: Object disembodied from Self Case 2: Self embodies Object Case 3: Self disembodied from Object Case 4: Object embodies Self selfobject self object self objectself Aesthetic Control Result Reflection Belonging

26. Fels: Multimodal Design and Technologies 26 Intimacy and Embodiment Design Examples Many excellent examples out there of interesting designs –Many at Siggraph: Jam-o-Drum (Blaine et al.), Wooden Mirror (Rozin), etc. –Tangible Bits work (Ishii et al.) –Ubicomp (Weiser and more) –Wearable computing (Mann and others) –Art, Entertainment –Virtual reality

27. Fels: Multimodal Design and Technologies 27 Application: Iamascope (Fels and Mase, 1997)

28. Fels: Multimodal Design and Technologies 28 Iamascope Overview Video

29. Fels: Multimodal Design and Technologies 29 FlowField: Semantics of Caress ( Chen, Fels and Min, 2003) Whole hand interaction in VR –use multi-touch sensitive pad –immersive display Idea: –allow users direct manipulation of fluid use particle simulation for fluid –aesthetics was important hand manipulation on hard surface mapped to obstructions in the flow field

30. Fels: Multimodal Design and Technologies 30 FlowField: Semantics of Caress

31. Fels: Multimodal Design and Technologies 31 Multimodal interfaces need to consider –human information processing –matching interface to task use complementary modes where appropriate –intimacy and embodiment Plenty of research opportunities Summary

32. Fels: Multimodal Design and Technologies 32

33. THE UNIVERSITY OF BRITISH COLUMBIA Thank-you! Please fill out your comments forms. www.icics.ubc.ca/hci/multimodal/