Laurence DEVILLERS & Jean-Claude MARTIN LIMSI-CNRS FP6 IST HUMAINE Network of Excellence / Association ( W3C Emotion Incubator Group.

 Definition (Scherer 2000)  An hypothetical construct denoting a process of an organism’s reaction to significant events  Cognitive theories  Focus on emotion as an cognitive evaluation of events  Sequential evaluation of appraisal variables Event Coping potential? Which kind of ressources he has to deal with the situation? Compatibility with external and internal standards? Is the event compatible with norms? Conduciveness to goals? Is this situation conductiveness goal ? Global pleasantness? Is this situation pleasant or not ? Novelty? Is the event new ? emotion

 Example of a clip from French TV news (EMOTV): someone saying that « I thank you for saying that I was innocent » => a mixture of negative and positive emotions

Naturalistic clips often feature several events eliciting the emotional behavior: Examples of events: -event #1: the person that is videotaped has been accused although she is innocent - event #2: she was recognized innocent - event #3: she is interviewed

 In this talk, « emotion » stands for « affective state » : emotions, moods, attitudes, …  Gap between emotions observed in artificial data (acted) and those observed with « real-life spontaneaous data in SITU » (Batliner, 2000)  Previous works (Ekman, Scherer, Campbell, Cowie, Devillers, Martin) have shown that there are many complex emotion mixtures in real-life audio or audio-visual data  Difference is mainly due to the context eliciting the emotion: the events that are at the origin of the emotions of a person

 Existing work  Scherer ‘appraisal variables’ are generally studied in emotion recall experiments and have been used for predicting emotions from events. Most studies consider acted data collected in-lab  Our research questions  How to annotate events in real life data?  Can we apply the appraisal model to perception of more general affective states from real life data?

 This study explores how to code emotional events  Are we able to code the events that triggered spontaneous expressions of real-life affective states in audiovisual data?  What are the relevant dimensions of events?  Are we able to infer them from short video clips?

 We already used appraisal dimensions in emotion perception investigations (Devillers et al., 2006):  Method: subjects annotate their perception of the videotaped person’s appraisal (28 clips (Belfast Naturalistic Data + French EmoTV), 5 coders, 16 appraisals dimensions)  Results: Several dimensions can be reliably annotated: conduciveness to goals, pleasantness, relation to expectations, controllability-conseqevt  Main difficulty: relations between simulatenous appraisals of multiple events and complex emotions mixtures

Complex Emotion mixtures Event 1 Controllability consequence events? Which kind of ressources he has to deal with the situation? Compatibility with external and internal beliefs? Is the event compatible with beliefs? Conductiveness goals? Is this situation conductiveness goal ? Global pleasantness ? Is this situation pkeasant or not ? Novelty? Is the event new ? Event 2 Controllability consequence events? Which kind of ressources he has to deal with the situation? Compatibility with external and internal beliefs? Is the event compatible with beliefs? Conductiveness goals? Is this situation conductiveness goal ? Global pleasantness? Is this situation pkeasant or not ? Novelty? Is the event new ?

We use two corpora collected at LIMSI-CNRS: (  EmoTV : « provocative corpora » (Abrilian et al., 2005)  Naturalistic corpus extracted from FrenchTV news  100 clips of few minutes (< 1 hour)  EmoTaboo (zara et al., 2007)  Record of a two-players game designed in order to induce emotions  10 clips of around 50 minutes each (8 hours)

 Definition of an event  An event is perceived in the video clips as triggering the affective state  Majority voting between 3 annotators  28 clips from EmoTV  Three steps 1. Identify and annotate the emotional events 2. Annotate the temporal dimensions of event 3. Annotate appraisal dimensions of events

Three groups of emotional events are defined in the OCC model (Ortony, Clore, & Collins, 1988) depending on what is evaluated: - the consequences of events for oneself or for others, - the actions of others - and the perception of objects. In our corpus, we only observed events that are being evaluated with respect to their consequence for oneself or others and actions of others. Our events scheme permits to annotate up to 3 emotional events for a given clip. Each event is annotated according to the temporal dimensions

 Affective state labels  Courage  Disappointment  Emotional events 1. Election campaign ▪ Temporality: past ▪ Duration : months and more 2. Election results ▪ Temporality: present(D-day) ▪ Duration : minutes

 Emotional labels  Serenity  Pride  Emotional events 1. Football cup ▪ Temporality: past ▪ Duration : months and more 2. Football match ▪ Temporality: close future ▪ Duration : hours

 Emotional labels  Anger  Despair  Disappointment  Disgust  Helplessness  Worry  Emotional events 1. Lawsuit (current) : ▪ Temporality: past ▪ Duration : months and more 2. Someone is accusing other people ▪ Temporality: present ▪ Duration : hour 3. The trial has just finished ▪ Temporality: present (D-day) ▪ Duration : minutes

 In-lab induced protocol EmoTaboo : adaptation of the game TABOO  It involves interaction between two players  One of them has to guess a word that the other player is describing using his own speech and gestures without uttering five forbidden words  We use strategies for eliciting emotions connected to:  the course of the game  the selection of the cards  The instructions given to the confederate

19  Dyadic interaction  Word guessing game  10 pairs  naïve subjects: 4 women, 6 men  confederates (close relations with research staff): 3 women, 4 men  4 cameras  8 hours

 Examples of events used for inducing emotions  the card contains the word "palimpsest"  the experimenter announces a penalty  the confederate proposes words with no relation at all with what is said by the naïve player  the confederate finds the mimed word/does not find the mimed word  the confederate is ironic  the confederate criticizes the naïve player  Ex: the card contains the word "palimpsest" is estimated by the player as:  recent  unexpected (the player did not expect a word that he did not know)  incompatible with his(her) purposes to win the game.

 Timescale of events is not the same as in EmoTV  EmoTABOO events are rather short term  One long-term event: the game session

 In this exploratory study, we tried to show the potential of a scheme of emotional events in order to better understand the emotional states  We proposed to code the temporal aspects of the events: date and duration (short-time, long-time)

 Annotation of more data  Annotate appraisal dimensions of events  Study the relations between  temporal and appraisal annotations  the events and the perceived complex emotions

 Goal: designing affective computing applications  Possible approach:  Collect small set of real life data  Annotate (events, emotion labels, …)  Inspire from these annotations to define a protocol close to the target application to elicit emotional behaviors in-lab  Collect large corpora of induced realistic emotional behavior relevant for the application  Opens a big question:  how close to the intended context does training material need to be?

 Thanks for attention

The motivation of this special issue is to report innovative work on the modelling and generation of affect in real- life speech and spoken interaction (including human- human or human-machine interaction, multi-party interaction) or in “realistic” interactions (including realistic fictions).Editors: Nick Campbell and Laurence Devillers

28  LIMSI – TV clips : Interviews from news - French Cause event: Unexpected bad Results in an Election Current event: Impact of election loss on political group Multimodal Cues: + speech content -Tense Smile (bad Control of their Facial muscles) -Emotion blends: - Disappointment masked by + courage

Cognitive appraisal theory (Scherer, 2000) argues that - an organism may possess many distributed processes for interpreting this relationship (e.g., planning, explanation, perception, etc.) -but that appraisal maps characteristics of these disparate processes into a common set of intermediate terms (intermediate between stimuli and response, between organism and environment) called ‘appraisal variables’ (Gratch and Marsella, 2004).