MULTIMODAL AND NATURAL COMPUTER INTERACTION Domas Jonaitis

CONCEPTION Multimodal human-computer interaction refers to the “interaction with the virtual and physical environment through natural modes of communication”, i. e. the modes involving the five human senses. This implies that multimodal interaction enables a more free and natural communication, interfacing users with automated systems in both input and output. Specifically, multimodal systems can offer a flexible, efficient and usable environment allowing users to interact through input modalities, such as speech, handwriting, hand gesture and gaze, and to receive information by the system through output modalities, such as speech synthesis, smart graphics and others modalities, opportunely combined.

AIM Multimodal interaction systems aim to support the recognition of naturally occurring forms of human language and behavior through the use of recognition-based technologies (Oviatt, 2003; Waibel et al., 1996)

HUMAN – COMPUTER INTERACTION A PROCESS WHEN INFORMATION TRANSFERRED 1 )User to machine (Human computer interaction (HCI)) 2) Machine to user(Computer human interaction (CHI))

MODALITIES Inputs: sensory organs Outputs: effectors

HUMAN - COMPUTER INTERACTION

ADVANTAGES OF MULTIMODELS Natural/realism making: making use of more (appropriate) senses New ways of interacting Flexible: different modalities excel at different tasks Wearable computers and small devices Helps the visually/physically impaired Faster, more efficient, higher information processing bandwidth Robust: mutual disambiguation of recognition errors Multimodal interfaces are more engaging Use more of users senses User perceive multiple things at once

EXAMPLES

MULTIMODAL INPUT

MULTIMODAL OUTPUT

LEVELS OF MULTIMODAL FUSION Data level Feature level Decision level

HUMAN COMPUTER INTERACTION ARCHITECTURE Multimodal HCI Systems The system based on combination of multiple modalities of interaction by simultaneous use of different channels Unimodal HCI Systems The system based on single channel of input

MODALITIES VISUAL-BASED HCI Facial Expression Analysis Body Movement Tracking (Large-scale) Gesture Recognition Gaze Detection (Eyes Movement Tracking)

MODALITIES AUDIO-BASED HCI Speech Recognition Speaker Recognition(fig 6). Auditory Emotion Analysis Human-Made Noise/Sign Detections (Gasp, Sigh, Laugh, Cry, etc.) Musical Interaction

MODALITIES SENSOR-BASED HCI 1. Pen-Based Interaction 2. Mouse & Keyboard 3. Joysticks 4. Motion Tracking Sensors and Digitizers 5. Haptic Sensors 6. Pressure Sensors 7. Taste/Smell Sensors

ARCHITECTURE OF MULTIMODAL USER INTERFACE

APPLIED COMPUTER VISION TECHNOLOGIES COMPUTER VISION GESTURE, MOTION RECOGNITION GOOGLE GLASS OCULUS, MICROSOFT HOLOLENS

APPLIED SOUND TECHNOLOGIES Speech recognition technologies Machine translation Siri, Google

APPLIED TOUCH TECHNOLOGIES Haptic technologies recreate the sense of touch by applying force, vibrations or motions to the user

BRAIN COMPUTER INTERFACE (BCI) Tries to understand human brain waves

SMELL/TASTE Electronic nose Electronic tongue (used to classify liquid contents)

PROBLEMS AND CHALENGES? Technological issues High price Portability Power consumption Lack of specialized software Intelligent agents require supercomputers Need powerful computers Virtual reality is tiring after long usage Human issues how do we handle conflicting information how to keep user attention what happens when everyone starts talking with their devices

POTENTIAL USAGE Can process and manipulating complex data Can help elderly or disabled people to communicate Can improve many routine things. Entertainment industries

CONCLUSIONS Although the literature on formal assessment of multimodal systems is still sparse, various studies have shown that multimodal interfaces maybe preferred by users over unimodal alternatives, can offer better flexibility and reliability, can offer interaction alternatives to better meet the needs of diverse users with a range of usage patterns and preferences. Multimodal interfaces can increase task efficiency. They permit the flexible use of input modes, including alternation and integrated use. They support improved efficiency, especially when manipulating graphical information. They can support shorter and simpler speech utterances than a speech-only interface, which results in fewer disfluencies and more robust speech recognition. They can support greater precision of spatial information than a speech-only interface, since pen input can be quite precise. They give users alternatives in their interaction techniques. They lead to enhanced error avoidance and ease of error resolution. They accommodate a wider range of users, tasks, and environmental situations. They are adaptable during continuously changing environmental conditions. They accommodate individual differences, such as permanent or temporary handicaps. They can help prevent overuse of any individual mode during extended computer usage.

FUTURE TECHNOLOGIES Self Driving car (Google)

NATURAL AND EVAPORATING INTERFACES

QUESTIONS? 1.What are the main human senses used for multimodal computer interaction ? 2.The conception of multimodal computer interaction? 3.The advantages of multimodal computer interaction? 4.Where can we implement this model? 5.The practical example of this model?