Multimodal and Natural computer interaction Evelina Stanevičienė

Introduction People naturally interact with the world multimodally. Multimodal human–computer interaction has sought for decades to endow computers with similar capabilities, in order to provide more natural, powerful, and compelling interactive experiences. Multimodal research that leverages speech, touch, vision, and gesture is on the rise. Multimodal and Natural computer interaction 2

Human-Computer Interaction The body of knowledge of Human- Computer Interaction can be perceived as resulting from the following five interrelated aspects, depicted in the diagram bellow: (U) the use and context of computers; (H) human characteristics; (C) computer system and interface architecture; (D) the development process. Multimodal and Natural computer interaction 3

A history of multimodal interaction ‘‘Put That There’’ was followed by numerous systems that sought to integrate various aspects of speech and gesture in a range of application areas. Multimodal and Natural computer interaction 4 „Shape-it-up“ system.

Advantages of multimodal interaction Humans may process information faster and better when it is presented in multiple modalities. Multimodal interfaces can increase task efficiency, although perhaps not significantly. Multimodal interfaces can support improved efficiency, especially when manipulating graphical information. Multimodal interfaces can permit the flexible use of input modes, including alternation and integrated use. Multimodal and Natural computer interaction 5

Input and output modalities Human sensory modalities relevant to multimodal human–computer interaction Humans primarily interact with the world through their five major senses of sight, hearing, touch, smell, and taste. Multimodal and Natural computer interaction 6

Designing and building multimodal interaction systems Multimodal systems should be designed for the broadest range of users and contexts of use. Designers should take care to address privacy and security issues in multimodal systems. Multimodal interfaces should adapt to the needs and abilities of different users, as well as different contexts of use. Be consistent – in system output, presentation and prompts, enabling shortcuts, state switching, etc. Provide good error prevention and error handling. Multimodal and Natural computer interaction 7

Conclusions Multimodal interaction is designed to provide enriched and challenging experience, such as the diversification of information: visual, touch, sound. This interaction allows a more flexible and efficient to operate. Speech recognition is useful when the user's hands are busy, do not have a keyboard, and can be used by people with certain disabilities. Touching can be used for both information input and output. Includes both touch sensation experienced by the skin, and a sense of motion and positioning. Sign recognition of communication is characterized by a very friendly user interface and requires no extra control facilities. Hands gestures use computer management has many advantages due to its simplicity, convenience and intuitive. Research and design is constantly being improved speech analysis, sound and sight recognition. It also analyzes the added value of combining different input and output. Multimodal and Natural computer interaction 8

Questions 1.Apibūdinkite multimodalinę sąveiką. 2.Multimodalinės sąveikos privalumai? 3.Kokie pagrindiniai žmogaus pojūčiai naudojami multimodalinėje sąveikoje? 4.Ką žinote apie kalbos atpažinimą, nekalbinius garsus? 5.Pateikite multimodalinės sistemos pavyzdį. Multimodal and Natural computer interaction 9