1. Multimodal Communication Jens Allwood Swecog National Swedish graduate school in cognitive science

2. Multimodal Communication 1. Why study multimodal communication? 2.What is multimodal communication? 3.Multimodality in Human-Human Communication –Production –Types of Representation –Perception, Understanding –Contextual Dependence 4.Some Theoretical issues 5.How study multimodal communication? 6.Multimodality and ICT (Information & Communication Technology)

3. Why study multimodal communication? Interesting in itself. What we are built for and are used to. -very much is still unknown -need for multimodal communication corpora -provides a background for what can be simulated or made use of in technical systems Source for design of computer supported applications -gives redundance, robustness, synergy -more natural and efficient HMI -more efficient and enjoyable Human- Human Communication

4. Examples -New interactive search machines (internet & local DB) -VR applications (games, education, trade) -Other educational support (Tutoring etc.) -Public information -Commercial systems Multimodality needs to be combined with multiculturality and multifunctionality

5. What is multimodal communication? Communication = def. Coactivation, sharing and coconstruction of information Mode/modality =manner, way Multimodal = Several alternatives of interpretation Main alternative 1. Modalities of perception and production Visual- gestures Auditory - sounds, speech Tactile- touch Olfactory- smell (molecules) Gustatory - taste (molecules) ? Submodalities color, shape & movement

6. Other alternatives 2. Medium =physical carrier Light waves, sound wave molecules, electromagnetic waves etc. 3. Type of representation = index, icon, symbol 4. Communication aids writing, radio, TV etc. Suggestion for definition: Multimodal Communication = coactivation, sharing and coconstruction of information simultaneously and sequentially through several modes of perception & production

7. Multimodal communication often becomes also Multimedial ( always) Multirepesentational Using several aids of communiction

8. Types of representation 3 ways of carrying, conveying and sharing information Index Icon Symbol All 3 are used simultaneously in ordinary human communication. E.g.Words (symbols) Gesture (icons) Voice quality (index) “ I ’ m small ” Variation in awareness and intentionality Bodily communication often more spontaneous, unaware, unintended (index, icon) than words (symbols).

9. Analysis of relation between Modalities necessary both for 1. production, planning of different simultaneous expressions and 2. perception, understanding of different simultaneous expressions

10. Theoretical Issues 1.How is (should) information (be) distributed (fission) in production across modalities, e g text, picture, voice, diagram, gesture 2.How is information from different modalities integrated (fusion) in reception? What function do the modalities have in relation to each other? 3. Which modes of production and perception are used in human-human communication (HHC)? 4.Are some forms of representation modality neutral and others modality specific? spoken (& written) language vs pictures

11. Theoretical issues continued 5. Is it possible to translate between modalities? –vision hearing, –Smell touch –touch vision Broad conception of translation and context to determine meaning 6. Is it possible to translate between types of representation? Speech (symbol) picture (icon) Speech (symbol) freq diagram (index) 7. What is synesthesia? - activated complementary? 8.How can cognitive development in general/in different tasks best be supported by multimodal resources?

12. Theoretical issues continued 9.How do we formally represent symbolic non-vocal and non-verbal types of representation? 10.Problem Tradition of neglect of multimodal issues 11.How can the modes of production and perception be made use of, emulated, simulated in HMI (human- machine interaction) and computer supported HHC?

13. How should multimodal communication be studied? 1. Descriptive and explanatory (audio, video recorded) studies of human-human communication in naturalistic settings –establishment of multimodal corpora and techniques for establishing and analyzing these 2. Experimental studies of human-human communication 3. Simulation of human-human communication (VR and robot) 4. Creation of multimodal application using Wizard of Oz techniques or “ Dialog distilling ”

14. Production Means of expression 1. PrimaryDirectly controllable Body movements, voice, speech, song, touch, smell?, taste? Can involve symbols, icons & indeces Can be regulated by social convention 2. SecondaryAids for supporting, augmenting and preserving primary means of expression Writing, telephone, radio, TV 3. TertiaryAids for supporting augmenting and preserving secondary means of expression Telegraph, fax, e-mail 4. QuarternaryAll human artifacts (collective communication) also unintended, undesired remains

15. Primary means Multimodal Face-to-face Communciation Production & perception

16. Communicative Body Movements 1)Facial gestures (nose, eyebrows, cheek, forehead, chin etc) 2)Head movements 3)Gaze direction, mutual gaze 4)Pupil size 5)Lip movements 6)Hand and arm movements 7)Leg and foot movements 8)Body posture 9)Distance btw communicators 10) Spatial orientation 11) Clothing and bracelets 12) Touch 13) Smell 14) Taste 15) Non-linguistic sounds

17. Multimodal Face-to-face Communication

18. Three levels of intentionality in communication Indicate Display Signal All three occur simultaneously in normal human communication indexiconsymbol indicatex displayx signalx All combinations are, in principle, possible, but the three x:s are more common.

19. One can increase awareness and intentionality through practice. indicate -> display -> signal What has been practiced can become a habit conventional signal -> automatic

20. Ways of processing information (reception) 1.Unaware reaction 2.Perception (apprehension) 3.Understanding 4.Other attitudes 5.Behavioral response 6.Other actions

21. Communication Coactivation, sharing and coconstruction between two or more agents SenderRecipientRepresenta- tion Indicate Display Signal Influence Perceive Understand Other attitudes Behavioral responses Index Icon Symbol

22. Concluding remarks Overview of multmodality in human communication