1. MULTIMODAL AND NATURAL COMPUTER INTERACTION Domas Jonaitis

2. CONCEPTION Multimodal human-computer interaction refers to the “interaction with the virtual and physical environment through natural modes of communication”, i. e. the modes involving the five human senses. This implies that multimodal interaction enables a more free and natural communication, interfacing users with automated systems in both input and output. Specifically, multimodal systems can offer a flexible, efficient and usable environment allowing users to interact through input modalities, such as speech, handwriting, hand gesture and gaze, and to receive information by the system through output modalities, such as speech synthesis, smart graphics and others modalities, opportunely combined.

3. AIM Multimodal interaction systems aim to support the recognition of naturally occurring forms of human language and behavior through the use of recognition-based technologies (Oviatt, 2003; Waibel et al., 1996)

4. HUMAN – COMPUTER INTERACTION A PROCESS WHEN INFORMATION TRANSFERRED 1 )User to machine (Human computer interaction (HCI)) 2) Machine to user(Computer human interaction (CHI))

5. MODALITIES Inputs: sensory organs Outputs: effectors

6. HUMAN - COMPUTER INTERACTION

7. ADVANTAGES OF MULTIMODELS Natural/realism making: making use of more (appropriate) senses New ways of interacting Flexible: different modalities excel at different tasks Wearable computers and small devices Helps the visually/physically impaired Faster, more efficient, higher information processing bandwidth Robust: mutual disambiguation of recognition errors Multimodal interfaces are more engaging Use more of users senses User perceive multiple things at once

8. EXAMPLES

9. MULTIMODAL INPUT

10. MULTIMODAL OUTPUT

11. LEVELS OF MULTIMODAL FUSION Data level Feature level Decision level

12. HUMAN COMPUTER INTERACTION ARCHITECTURE Multimodal HCI Systems The system based on combination of multiple modalities of interaction by simultaneous use of different channels Unimodal HCI Systems The system based on single channel of input

13. MODALITIES VISUAL-BASED HCI Facial Expression Analysis Body Movement Tracking (Large-scale) Gesture Recognition Gaze Detection (Eyes Movement Tracking)

14. MODALITIES AUDIO-BASED HCI Speech Recognition Speaker Recognition(fig 6). Auditory Emotion Analysis Human-Made Noise/Sign Detections (Gasp, Sigh, Laugh, Cry, etc.) Musical Interaction

15. MODALITIES SENSOR-BASED HCI 1. Pen-Based Interaction 2. Mouse & Keyboard 3. Joysticks 4. Motion Tracking Sensors and Digitizers 5. Haptic Sensors 6. Pressure Sensors 7. Taste/Smell Sensors

16. ARCHITECTURE OF MULTIMODAL USER INTERFACE

17. APPLIED COMPUTER VISION TECHNOLOGIES COMPUTER VISION GESTURE, MOTION RECOGNITION GOOGLE GLASS OCULUS, MICROSOFT HOLOLENS

18. APPLIED SOUND TECHNOLOGIES Speech recognition technologies Machine translation Siri, Google

19. APPLIED TOUCH TECHNOLOGIES Haptic technologies recreate the sense of touch by applying force, vibrations or motions to the user

20. BRAIN COMPUTER INTERFACE (BCI) Tries to understand human brain waves

21. SMELL/TASTE Electronic nose Electronic tongue (used to classify liquid contents)

22. PROBLEMS AND CHALENGES? Technological issues High price Portability Power consumption Lack of specialized software Intelligent agents require supercomputers Need powerful computers Virtual reality is tiring after long usage Human issues how do we handle conflicting information how to keep user attention what happens when everyone starts talking with their devices

23. POTENTIAL USAGE Can process and manipulating complex data Can help elderly or disabled people to communicate Can improve many routine things. Entertainment industries

24. CONCLUSIONS Although the literature on formal assessment of multimodal systems is still sparse, various studies have shown that multimodal interfaces maybe preferred by users over unimodal alternatives, can offer better flexibility and reliability, can offer interaction alternatives to better meet the needs of diverse users with a range of usage patterns and preferences. Multimodal interfaces can increase task efficiency. They permit the flexible use of input modes, including alternation and integrated use. They support improved efficiency, especially when manipulating graphical information. They can support shorter and simpler speech utterances than a speech-only interface, which results in fewer disfluencies and more robust speech recognition. They can support greater precision of spatial information than a speech-only interface, since pen input can be quite precise. They give users alternatives in their interaction techniques. They lead to enhanced error avoidance and ease of error resolution. They accommodate a wider range of users, tasks, and environmental situations. They are adaptable during continuously changing environmental conditions. They accommodate individual differences, such as permanent or temporary handicaps. They can help prevent overuse of any individual mode during extended computer usage.

25. FUTURE TECHNOLOGIES Self Driving car (Google)

26. NATURAL AND EVAPORATING INTERFACES

27. QUESTIONS? 1.What are the main human senses used for multimodal computer interaction ? 2.The conception of multimodal computer interaction? 3.The advantages of multimodal computer interaction? 4.Where can we implement this model? 5.The practical example of this model?