1. Backchannels Through Gaze as Indicators of Persuasive Success E. Bevacqua, M. Mancini, C. Peters, C. Pelachaud University of Paris 8 Isabella Poggi Università Roma Tre

2. GOAL Define speaker that alters gaze behaviour during discourse for the purposes of persuasive interaction Define speaker that alters gaze behaviour during discourse for the purposes of persuasive interaction Speaker monitors amount of interest listener has in what it is saying Speaker monitors amount of interest listener has in what it is saying Part of wider framework (WP6): Part of wider framework (WP6): An agent that can establish, maintain and close a conversation (Sidner, 2004)An agent that can establish, maintain and close a conversation (Sidner, 2004) Significance to WP8: Significance to WP8: consider reactive backchannels through attention and interestconsider reactive backchannels through attention and interest important prerequisites for persuasionimportant prerequisites for persuasion

3. Background Interaction is by no means a one way communication channel between parties Interaction is by no means a one way communication channel between parties Within an interaction, parties take turns in playing the roles of the speaker and of the listener Within an interaction, parties take turns in playing the roles of the speaker and of the listener Where persuasion fits in: Where persuasion fits in: Speaker to monitor listeners attention and interest in what they have to say Interest maintenance as prerequisite for persuasive interaction Listener to persuade the speaker that they are paying attention Vital for maintaining engagement

4. Role of Speaker Speaker: Speaker: Pointless for a speaker to engage in an act of communication if listener does not pay or intend to pay attentionPointless for a speaker to engage in an act of communication if listener does not pay or intend to pay attention Important for speaker to assess listener’s engagement at:Important for speaker to assess listener’s engagement at: when starting an interaction: assess the possibility of engagement in interaction (establish phase) when starting an interaction: assess the possibility of engagement in interaction (establish phase) when interaction is going on: check if engagement is lasting and sustaining conversation (maintain phase) when interaction is going on: check if engagement is lasting and sustaining conversation (maintain phase)

5. Role of Listener Listener Listener attention: pay attention to the signals produced by speaker to perceive, process and memorize themattention: pay attention to the signals produced by speaker to perceive, process and memorize them perception: of signalsperception: of signals comprehension: understand meaning attached to signalscomprehension: understand meaning attached to signals internal reaction: the comprehension of the meaning may create cognitive and emotional reactioninternal reaction: the comprehension of the meaning may create cognitive and emotional reaction decision: decision to communicate the internal reactiondecision: decision to communicate the internal reaction generation: display behaviorsgeneration: display behaviors

6. Application to Persuasive ECAs Persuasive ECAs ought to be able to: Persuasive ECAs ought to be able to: establish backchannels with the otherestablish backchannels with the other conduct eye contact properlyconduct eye contact properly Maintain mutual eye contact but… Maintain mutual eye contact but… Don’t stare or threaten Don’t stare or threaten interpret looking behaviours of the otherinterpret looking behaviours of the other Requires engagement capabilities Requires engagement capabilities Perception of gaze Perception of gaze Establish backchannels Establish backchannels Generate gaze Generate gaze

7. Key Capabilities in our Model Listener backchannel Listener backchannel Speaker interpreting listener behaviour Speaker interpreting listener behaviour Speaker generating or altering its own behaviour Speaker generating or altering its own behaviour

8. 1. Listener Backchannel During conversation… During conversation… two streams of information exchanged at the same time between participants (the speaker and listener).two streams of information exchanged at the same time between participants (the speaker and listener). first one concerned with the topic of the conversationfirst one concerned with the topic of the conversation second provides information on the successfulness of the communicationsecond provides information on the successfulness of the communication Listener can show level of engagement in conversation Listener can show level of engagement in conversation Through linguistic and gestural signals, called backchannel feedbacksThrough linguistic and gestural signals, called backchannel feedbacks

9. Backchannel Depending on the type of speech act they respond to, a signal will be interpreted as a backchannel or not. Depending on the type of speech act they respond to, a signal will be interpreted as a backchannel or not. backchannel: a signal of agreement / disagreement that follows the expression of opinions, evaluations, planningbackchannel: a signal of agreement / disagreement that follows the expression of opinions, evaluations, planning not a backchannel: a signal of comprehension / incomprehension after an explicit question « Did you understand? »not a backchannel: a signal of comprehension / incomprehension after an explicit question « Did you understand? »

10. Listener Backchannel based on taxonomy of Poggi (2005) based on taxonomy of Poggi (2005) attentionattention comprehensioncomprehension believabilitybelievability interestinterest agreementagreement positive/negative positive/negative any combination of the above any combination of the above pay attention but not understand; understand but not believepay attention but not understand; understand but not believe Our focus: interest and attention on a signal level (not on a cognitive level) Our focus: interest and attention on a signal level (not on a cognitive level) listener’s feedback is triggered by the speaker’s behaviour and not by what the speaker is sayinglistener’s feedback is triggered by the speaker’s behaviour and not by what the speaker is saying

11. Backchannel modelling Reactive model Reactive model generates an instinctive feedback without reasoninggenerates an instinctive feedback without reasoning Cognitive model Cognitive model conscious decision to provide backchannel to provoke a particular effect on the speaker or to reach a specific goalconscious decision to provide backchannel to provoke a particular effect on the speaker or to reach a specific goal Currently, our model is reactive in nature Currently, our model is reactive in nature Dependant on perceptionDependant on perception Extendable to include limited language reasoning by inclusion of APML tagsExtendable to include limited language reasoning by inclusion of APML tags

12. Listener Backchannel Listener’s behaviour produced to show feedback can be categorized as follows: Listener’s behaviour produced to show feedback can be categorized as follows: gaze,gaze, paraverbals,paraverbals, head movements, like nods, jerks, shakes, tilts,head movements, like nods, jerks, shakes, tilts, facial expressions,facial expressions, other gestures such as hand and body movements.other gestures such as hand and body movements.

13. Backchannel signals of gaze Gaze Gaze shows direction of attentionshows direction of attention informs on level of engagement or on intention to maintain engagementinforms on level of engagement or on intention to maintain engagement indicates degree of intimacyindicates degree of intimacy but also monitors the gaze behavior of others to establish their intention to engage or maintain engagedmonitors the gaze behavior of others to establish their intention to engage or maintain engaged Shared attention Shared attention involves mutual gaze at other partner or mutual gaze at a same objectinvolves mutual gaze at other partner or mutual gaze at a same object By analysing the listener’s gaze behaviour By analysing the listener’s gaze behaviour can compute his level of attention and interestcan compute his level of attention and interest E.g. if the listener starts to frequently avert his eyes from the speaker, it means that he is probably losing interest and that he is not paying attention anymore. E.g. if the listener starts to frequently avert his eyes from the speaker, it means that he is probably losing interest and that he is not paying attention anymore.

14. 2. Speaker interpreting listener Speaker perceives and interprets behaviour of listener Speaker perceives and interprets behaviour of listener determine effectiveness and progression of persuasion attemptsdetermine effectiveness and progression of persuasion attempts perceives eye, head and body directions of the listenerperceives eye, head and body directions of the listener ‘Snapshot’ perceptual updates ‘Snapshot’ perceptual updates Weighted sum forms an attention level… Weighted sum forms an attention level… …the amount of attention that the other is paying at this instant of time …the amount of attention that the other is paying at this instant of time

15. Metrics Based on virtual visual sensing of the others looking behaviour Based on virtual visual sensing of the others looking behaviour Used to interpret and generate ECA gaze behaviours. Used to interpret and generate ECA gaze behaviours. Attention Level: amount of visual attention one is perceived to be paying at an instant of time, based on their eye, head and body directions.Attention Level: amount of visual attention one is perceived to be paying at an instant of time, based on their eye, head and body directions. Interest Level: amount of attention perceived to be paying over a period of time and is a measurement of the others interest in what the speaker has to say.Interest Level: amount of attention perceived to be paying over a period of time and is a measurement of the others interest in what the speaker has to say. Effectiveness: indication of how well communication is progressing – that is, how the listener is perceived to be behaving (e.g. attending, looking elsewhere, not interested) when the speaker is saying something important.Effectiveness: indication of how well communication is progressing – that is, how the listener is perceived to be behaving (e.g. attending, looking elsewhere, not interested) when the speaker is saying something important.

16. Interpreting An agent’s attention at one instant of time can be misleading… An agent’s attention at one instant of time can be misleading… … so a profile of attention over time is considered… so a profile of attention over time is considered Attention levels over time form an attention profileAttention levels over time form an attention profile Integration produces interest level metricIntegration produces interest level metric Effectiveness of speakers communication used to alter speaker’s behaviour towards the listenerEffectiveness of speakers communication used to alter speaker’s behaviour towards the listener frames the actions of the listener in the context of what the speaker is saying. frames the actions of the listener in the context of what the speaker is saying.

17. 3. Speaker generating / altering its behaviour Speaker Speaker Alters its behaviour to try to be more persuasive and obtain listener's attentionAlters its behaviour to try to be more persuasive and obtain listener's attention Variation in gaze parameters Variation in gaze parameters Abides by timing constraints for gazeAbides by timing constraints for gaze maximum amount of time speaker can look at listener without interruption maximum amount of time speaker can look at listener without interruption maximum amount of time speaker and listener can engage in mutual attention maximum amount of time speaker and listener can engage in mutual attention

18. Speaker generating / altering its behaviour During behaviour generation: During behaviour generation: level of effectiveness is constantly calculatedlevel of effectiveness is constantly calculated compared to a desired levelcompared to a desired level if it is below this threshold, the speaker increases its tendency to look at the listenerif it is below this threshold, the speaker increases its tendency to look at the listener temporal constraints are appliedtemporal constraints are applied final behaviour is decided by a probabilistic modelfinal behaviour is decided by a probabilistic model

19. Algorithm input: APML tagged text, effectiveness threshold and maximum duration of gaze direction for speaker S and listener L input: APML tagged text, effectiveness threshold and maximum duration of gaze direction for speaker S and listener L compute behaviors for S and L based on APML tags specification compute behaviors for S and L based on APML tags specification compute metrics compute metrics attention profileattention profile level of interestlevel of interest level of effectiveness of conversationlevel of effectiveness of conversation

20. Videos Low interest Low interest High interest High interest

21. Future Work Interpretation module based on perception Interpretation module based on perception listener’s gazelistener’s gaze Simplified language-based planning scheme Simplified language-based planning scheme use of hard-coded tags in the input APML providing semantic information related to that utteranceuse of hard-coded tags in the input APML providing semantic information related to that utterance Multiple choice selection for the speaker Multiple choice selection for the speaker at each stage in the dialogue choose between predefined utterances associated with slightly different verbal or behavioural characteristics depending on the goal of the speaker e.g. increase in volume, increased expressivity of hand gestures.at each stage in the dialogue choose between predefined utterances associated with slightly different verbal or behavioural characteristics depending on the goal of the speaker e.g. increase in volume, increased expressivity of hand gestures.

22. Future Work Planner uses interpretation of Planner uses interpretation of perceived behaviour of the listener during the previous utteranceperceived behaviour of the listener during the previous utterance backchannelling, in particular head nods and facial expressions backchannelling, in particular head nods and facial expressions Gaze behaviour Gaze behaviour comparison with the importance of the utterancecomparison with the importance of the utterance APML tags corresponding to importance of speech segments APML tags corresponding to importance of speech segments effectiveness of conversation effectiveness of conversation theory of the interaction goal of the listener (e.g. maintain or finish conversation). theory of the interaction goal of the listener (e.g. maintain or finish conversation). Enhance listener backchannel model Enhance listener backchannel model new backchannel signals (head nods, facial expressions)new backchannel signals (head nods, facial expressions) feedbacks triggered also by the semantic of Speaker’s wordsfeedbacks triggered also by the semantic of Speaker’s words reactive model -> reactive+cognitive modelreactive model -> reactive+cognitive model