Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23 Marieke Agterbos | Tim van Bremen | Carel Jansen | Lisa Oud | Justin Post

Research Question To what extent do compliments have an impact on the user experience of embodied conversational agents?

Questions 1.Are the ECA’s compliments recognized as compliments? (Part I) 2.Experience with vs. without compliments (Part II) A.Perceived liking of the agent B.Perceived level of comfort C.Perceived information quality

Background theory  Making machines more human (Picard, 2000)  Minimize gap between real life contact and digital contact  Lifelike ECA (Louwerse et al., 2009)  Is sensitive to moods and sentiments  Politeness and variation in language (Behnam & A. Niloufar, 2011)  Add value and believability  Function of Compliments (Manes en Wolfson, 1981)  Goodwill and solidarity  Three types of personal compliments (Jucker, 2009)  Implicit, explicit or indirect  Behavioral realism might be more important than photorealism (Groom, 2009)  Generating positive response  More productive conversation

Pre-test  Are the ECA’s compliments recognized as compliments?  Participants read a transcript of an ECA conversation  Questionnaire on different parts of the conversation: 1. Bot: ‘Hello, how are you?’ 2. User: ‘Great, and you?’ 3. Bot: ‘Good, so am I.’ 4. User: ‘Good.’ 5. Bot: ‘What was the name of the lead guitar player of the band called “A band of Gypsies?”’ 6. User: ‘Jimi Hendrix.’ 7. Bot: ‘That is correct.’ 8. Bot: ‘Someone told me you are very bright.’

Pre-test results ‘Yes, this is a significant difference (p<0.01)’  84 participants (53, 31)

Pre-test results

Experiment Design  Measured variables  Liking  Comfort  Information Quality

Experiment  53 Participants  Conversation with bot  Quiz setting  Questionnaire  Perceived liking  Perceived level of comfort  Information quality “How many strings does a guitar have?” “Who is the prime minister of the Netherlands?”

Results

Effects

Results  H1. Contextual compliments have a positive effect on the user experience of an ECA.  H1-a. perceived liking of the agent.  H1-b. perceived level of comfort.  H1-c. perceived quality of information.  H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.  H2-a. perceived liking of the agent.  H2-b. perceived level of comfort.  H2-c. perceived quality of information.  H3. Contextual compliments have a more positive effect than non-contextual compliments.  H3-a. perceived liking of the agent.  H3-b. perceived quality of information.  H3-c. perceived level of comfort.

Results  H1. Contextual compliments have a positive effect on non-contextual compliments on the user experience of an ECA.  H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.  H3. Contextual compliments have a more positive effect than non- contextual compliments.  H4. Compliments have a positive effect on the user experience of an ECA.  H4-a. perceived liking of the agent.  H4-b. perceived quality of information.  H4-c. perceived level of comfort.

Conclusion  Compliments have a positive effect on the user experience of an ECA.  However, not on likingness.  The type of compliment matters  Non-contextual > contextual

Discussion Pros:  Pretest, kind of compliments accounted for.  Pretest in line with experiment outcome.  All experiments under supervision of a test leader. Improvements:  Sample too small  Language, not native language of all subjects  Variable mount of compliments given  Compliments when correctly answered  Definition of comfort, persuasiveness, likeliness might differ individually

Questions?

Measuring - Liking  ‘I would like to be friends with her’  ‘If I was upset, she would be able to cheer me up.’  ‘She is a bad listener’  Rated on Likert scale 1-7

Measuring - Level of comfort  During the conversation, I felt:  Comfortable  Involved  Open  Relaxed  Warm  Rated on Likert scale 1-7

Measuring - Information quality  The extent to which the information was:  Correct  Helpful  Persuasive  Insightful  Relevant  Rated on Likert scale 1-7

Context  Controlled environment  Participants from same population