Affective Interfaces Present and Future Challenges Introductory statement by Antonio Camurri (Univ of Genoa) Marc Leman (Univ of Gent) MEGA IST Multisensory Expressive Gesture Applications

Introductory Statement (1/3) KANSEI, Affect, Emotion, Personality, Expressiveness, Gesture,... Psychology, AI, CHI, Art and Humanities Interfaces: –Enhance interfaces by means of non-verbal expressive data modeling and communication –Embed models of emotion and personality in natural language dialog (e.g., web agents,...)

...Introductory Statement… (2/3) (USA) Affective Computing: Bates/CMU, B.Hayes-Roth/Stanford, Picard/MIT,... O’Rorke-Ortony model –Games/entertainment, Web-agents with personality, wearables, mobile devices... (Japan) KANSEI Information Processing: S.Hashimoto/Waseda, K.Mase/ATR,... –KANSEI Industry (Art, Entertainment, Fashion, Cosmetics, Kansei-Bots,...); the HUMANOID Project (e.g., elderly assistance by “Kansei machines/interfaces”)

...Introductory Statement (3/3) An example - a spectator observes a “microdance”: –Which are the lower-level “cues” in movement which may explain “expressiveness”? (eg, rigid/fluid, direct/flexible, extrovert, energy, rhythm features…) –does the spectator understand the dancer’s intended emotional expression ? (eg, “sad”, “angry”) –“KANSEI” or “sensible” content in the microdance: how much intensely the spectator is “hit” by an inspired dancer performance ? (similar to the “arousal” problem in emotion psychology theories) Music, Sound and Voice Expressive spaces as a new perspective on multimodality

MEGA IST-40210: Toward an Original European Route MEGA (EU IST aims at consolidating a third original European route, gounded on: –art and humanistic theories of non-verbal communication –“expressive gestures”: expressive cues, emotion, “sensible”/KANSEI processing Focus: non-verbal expressive gestures communication in artistic scenarios, trying to inprove technology by arts and humanities

The notion of “Gesture” non-verbal epistemic unit: e.g., movements of the human body, particular occurrences in music defined by criteria related to human information processing, in particular in motor and perceptual actions has typical beginning and ending describes trajectories in a feature space may occur at different levels: e.g., in music, an individual note may be considered as a gesture, with a given attack, sustain and decay pattern. Also a sequence of notes may be considered a gesture, or a repeating rhythm pattern.

“Expressive Gesture” A gesture, in principle, may be neutral, hence operating as a self-contained unit, yet many gestures are called ‘expressive’ Expressive gestures refer to a semantics outside the gesture pattern. This semantic may relate to synaesthetic (e.g. sound-colour), kinaesthetic (sound-movement, cf. Laban), and cenaesthetic issues (sound-sensation-emotion) The notion of gesture as a fundamental category of human action and perception, that ultimately relates to affective and sensitive computing

“Gesture” in MEGA Definition of the notion of gesture in multimodal artistic and cultural scenarios Intermodality or mapping of gestures from one to another domain Recognition of features that are relevant for expressive gestures, in particular also for affective and sensitive computing

MEGA: Research Analysis of expressive gesture in –movement (incl.dance) WP3 –audio (incl. music) WP4 Synthesis of expressive gesture in –movement and visual media WP5 –sound (incl. music) WP6 Multimodal Integration Mapping strategies from input modalities to output media Open software platform (EyesWeb) to support research and to enable commercial exploitations

MEGA: open architecture Based on EyesWeb (DIST), –open platform supporting MEGA research and apps –the platform integrates “modules” for expressive analysis, synthesis, and mapping strategies. –visual language, Wizard for plug-in’s devel. –based on industrial standards (MS COM/DCOM) –engineered release, coming from previous versions used in concrete apps –EyesWeb user community –Integrated with other tools available in MEGA: IPEM ToolBox (IPEM), DOVRE (Telenor),...

MEGA: Applications Exploitation: –Art and theatre: MEGA consortium –Musical instruments: GeneralMusic –Visual “soundtrack”: Eidomedia –Distributed cooperative apps: Telenor Demos at IBC2001? Public artistic events planned: music theatre, interactive museum exhibits