2. Feedback Elisabetta Bevacqua, Dirk Heylen,, Catherine Pelachaud, Isabella Poggi, Marc Schröder

3. Roddy + Ruth

4. Modeling

5. Listener system Listener’s modules: generator of listener’s behaviours Input: video and audio data from real world Player: 3D agent Greta Backchannel library: lexicon of backchannel signals Whiteboard Psyclone: communication protocol system

6. Dialogue Management Integrating the various components of a dialogue system capable of non-verbal expressivity. We connected –a visual renderer (Greta), –an audio renderer (MARY), and –a dummy dialogue system capable of generating non-verbal behaviour (Conversational Dialogue Engine / DFKI) Using OpenAIR.

7. Reactive Listener module The reactive module generates listener’s responses according to speaker’s head movement To detect head movements we integrated Watson (Gratch et al.) –At the moment Greta reacts with a head nod every time the speaker performs a nod or a shake –In the future Greta will be able to react with different backchannel signals and/or copy the speaker’s head movement

8. Analysis Head movements –tracking –classification

9. Television

10. Data Annotation (SAL)

11. Cognitive Listener module We use the SAL Wizard of Oz to trigger deliberative listener behaviours for Greta Pre-calculated FAP files are selected according to the wizard’s decision The Player displays the selected FAP files

12. Backchannel lexicon We aim at building a listener ECA able to display backchannel signals according to its style of behaviour: assertive/not assertive, believing/not believing, interested/not interested and so on We need to define a set of backchannel signals that users are able to interpret and understand To define such a library of recognizable signals we performed a perceptive test

13. Perception/Feedback Samples* for subjects to judge –questionnaires (semantic scales) –ask to label things Facial Expressions Affect Bursts

14. Perceptive Test Perceptive test: find a mapping between signals and meanings Questions: –it is possible to identify a signal (or a combination of signals) for each meaning, –a combination of signals can alter the meaning attached to each backchannel single signal.

15. Subjects and material Sixty French subjects (age mean 20.1) Tasks: select meanings for facial expressions and head signals displayed by Greta 21 different signals 12 meanings: agree, disagree, accept, refuse, interested, not interested, believe, disbelieve, understand, don't understand, like, dislike As the list of meanings was too long, subjects were divided into two groups: group1 and group2

16. Signals Signals can be simple (containing just a single action) or complex (containing several actions): nodtilt smiletilt and frown raise eyebrowstilt and sad eyebrows nod and smiletilt and raise eyebrows nod and raise eyebrowstilt and gaze right down shakegaze right down frowneyes roll up tension 1 raise left eyebrow shake and frownsad eyebrows frown and tension 1 eyes wide open shake, frown and tension 1 1 tension of the lips

17. Result

18. 3

19. 11

20. First question Q1 it is possible to identify a signal (or a combination of signals) for each meaning agree and accept: the signal nod proved to be very significant. All signals containing nods were interpreted as signals of agreement and acceptance like: the signal smile conveys this meaning understand: this meaning can be conveyed through the combination of smile and raise eyevrows

21. Second question Hyp2: a combination of signals can alter the meaning of backchannel single signals. Tension alone and frown alone do not mean dislike, but their combination does To convey the meaning disbelieve tilt and frown must be displayed together The signal frown means don’t understand but when a shake is added their combination loses this meaning Tilt alone and gaze right down alone do not mean not interested, but their combination does

22. Experiment: Affect bursts as listener feedback Research questions 1. Affect bursts used as listener feedback => same emotion? 2. How acceptable is such feedback? (3. Difference between German and Dutch listeners?)

23. Method Stimuli –select German affect bursts from Schröder (2003) –embed into neutral German / Dutch speaker utterance “Yeah, then I told myself, why don’t you try it and then I did it!” –10 emotions, 2 affect bursts each => 20 stimuli per language e.g. Dutch + admiration-wow e.g. Dutch + anger-growl … e.g. German + worry-ohoh e.g. German + startle-ah …

25. Results

26. maybe: social appropriate ness + admiration, elation, relief, worry - threat, startle, anger

27. Discussion “Acceptability” is very ambiguous –general appropriateness in the context (intended) –strange as reaction to speaker utterance –technical aspects mismatch between sound quality timing of feedback –social appropriateness: display rules social norms prescribed by one’s culture as to “who can show what emotion to whom, when” (Ekman, 1977)

28. Discussion (2) Tentative set of display rules for affect bursts –display gratifying emotions (admiration); –display empathy emotions (elation, worry, relief); –do not display negative evaluation (disgust, contempt) –do not display aggression (anger, threat) Can explain most observations –but not high acceptability of boredom

29. Summary and Questions For some emotions, highly recognisable affect bursts were judged to fit well with the context Perception of emotional feedback may depend on: –social acceptability (display rules); –semantic/pragmatic interaction between speaker utterance and affect burst; –timing of feedback; –relation between speaker/listener; –formality of the situation; –…

30. Future work In the future we aim at: –integrating perception of audio data –defining a set of rules to decide when a reactive backchannel signal must be triggered and which signal Greta should display –defining different styles to create variety of agents (assertive/not assertive, interested/not interested, believing/not believing, and so on) and evaluating their impression on users

31. CONTEXT first phase has ended succesfully move to the next… and put the findings in context