WP6 Emotion in Interaction Embodied Conversational Agents WP6 core task: describe an interactive ECA system with capabilities beyond those of present day ECAs Advance ECA capabilities in perception, generation, adaptation and interaction

Agent capabilities ECA to be provided with capacities for: –perception: show behaviors associated with emotion-related attention shifts react in emotionally colored ways to environmental events –generation: produce dynamic expressive behaviors achieve coordination of signs of emotion in multiple modalities

Agent capabilities –adaptation: reactions to the user’s emotional state reactions to the situational context –creating affective bonds: converge or diverge with user’s behavior

WP6 Exemplar Exemplar will take the form of a theoretical framework –Description of overall ECA architecture –Sub-modules based on agent capabilities –Clear conceptual links between sub-modules –A number of conceptual sub-modules have been enumerated to provide the following capacities:

Agent Capacities 1. Co-ordination of signs of emotion in multiple modalities 2. Production of expressive behaviours 3. Behaviours associated with emotion-related attention shifts 4. React in emotionally coloured ways to environmental events 5. Adapt reactions to the users emotional state 6. Adapt reactions to the context 7. ‘Converge’ or ‘diverge’ with the users behaviour appropriately

Behaviour Expressivity of ECA Focus on how behaviours that accompany acts of speech are generated Expressivity is key: capture with small number of numerical attributes providing high level control Find good set of parameters that apply across modalities Establish connection between signal intensity and expressivity parameterisation Address multimodal integration issues of signals Investigate influence of multiple nested communicative goals on signal production Related WPs: WP3, WP4, WP9 exemplar sub-module 8.1

Attention-based ECA Discourse Control of flexible discourse between multiple ECA’s, focusing on a model of the listener ECA perception and interpretation of signals from other agents Combination of different attention modalities into a single focus of attention Agent memory for storage of discourse information, theories of mind etc. Generation of related behaviours Related WPs: WP3, WP4, WP7 exemplar sub-component 8.2

Emotionally Sensitive Reactive Agent Agents that can react emotionally to internal and external stimuli perceived through virtual sensors Spontaneous gestures modulated by an emotional layer Automatic reactions to unexpected sounds, objects or characters Related WPs: WP4 exemplar sub-component 8.3

Adaptation to User’s Affective State ECAs that can perceive and act accordingly to the emotional state of a user in a principled way GAMBLE – Gretas Affective MoBiLe dicing intEraction Emotional situation where the agent blames the user for deceit, or the user detects agents deceit and agent must react How emotions need to be conveyed in order to increase the user’s trust in an agent Study emotion interaction in multi-party situation with several human users Related WPs: WP4, WP8, WP10 exemplar sub-component 8.4

The Role of Affect in Imitation Study the decision making process during imitation – what, when and how Understand emotional contagion mechanisms and model underlying mimicry Study the use of synchronisation in social interaction Related WPs: WP4, WP7 exemplar sub-component 8.8

Multimodal Coordinated Signs of Emotion Synchronization scheme across modalities Temporal relation between modalities Selection of modalities Weighting modalities Context Related WPs: WP3, WP4, WP5 exemplar sub-component 8.5

Situational Context Factors that contribute to an experiment –Participants, instructions, task, modalities used Slight change in one can have great influence Determine the type of possible emotional states Select the type of behavior to look for in a video corpus or from the user in an interactive setting Adapt the agent’s behavior to the context Related WPs: WP4, WP5 exemplar sub-component 8.6

Interactive Artistic Performance Testbed New methods for natural interaction Analysis of expressive gesture Use dance (as artistic expression of human movement) as test bed Output expressive audio and visual content Not ECA oriented; but could be linked to expressive gesture for ECAs (exemplar sub-component 8.1) Related WPs: WP3, WP4 exemplar sub-component 8.7

Relation with WPs WP3: define proper set of expressivity parameters; relevant parameter; how intensity of emotional state is reflected by behavior expressive parameters WP4: perception and recognition of user’s emotional state; understanding and learning the dynamic properties of user’s behavior

Relation with WPs WP5: collecting and analysing the data; defining what to code and how to code it; build a lexicon; define a language to describe gesture. WP7: link between modelling of the mind and behavior externalization; decision making

Relation with WPs WP8: effects of gaze and facial expression on the persuasiveness of an agent on a human viewer, particularly in terms of influencing the users emotional state. WP9: use user-centered design method to design a given application WP10: manipulation of user’s emotional state, say ‘fake’ information with ‘believability’

SALAS Artificial listener designed to allow the user to move through various emotional states Consists of slides that offer a range of responses to a human speaker Responses depend on which of four personalities human has chosen to talk to –Poppy: cheerful and optimistic –Spike: aggressive and confrontational –Obadiah: gloomy and pessimistic –Prudence: matter-of-fact with a practical view on life

SALAS Additional slides for switches between personalities and ‘repairs’ for when something goes wrong Each personality tries to steer the user towards a target state Useful as a crystallisation point where modules can be plugged in Avoids high-level planning problems Different versions with varying qualities May act as a thread through WP4, WP5 and WP6 Will NOT be a system however