1. 1 Chap. 13 Expressive Behaviors for Virtual Worlds Stacy Marsella, Jonathan Gratch, and Jeff Rickel November 22, 2004

2. 2 Outline Introduction Steve Jack and Steve Carmen’s Bright IDEAS (CBI) Mission Rehearsal Exercise (MRE) Conclusion

3. 3 Introduction Goal of this chapter –To create virtual humans that convey much information to humans while interacting with them in virtual worlds –Information includes non-verbal behaviors concerning emotional state Requirements for virtual human design –Believable: they must provide human-like behaviors –Responsive: they must respond to the events surrounding them –Interpretable: the user must be able to interpret their response to situation This chapter describes –The progress toward a model of outward manifestations of an agent’s cognitive and emotional state –The review of three prior systems that influenced model of this chapter –The model of this chapter is an integration of prior systems

4. 4 Contribution Previous systems provides impressive capabilities in its area of research focus, but they had some limitations Limitation of previous systems –Agents are modeled only for conversation between two of them –Collaboration of users and agents on one task is not allowed –Agent’s presence was partially limited to a 2D –Agent’s movement and relationships are limited Steve’s contribution –Only Steve can interleave task-related behaviors and face-to-face dialogue with humans and virtual humans in dynamic virtual worlds –Interleaving 은 성능을 높이기 위해 데이터가 서로 인접하지 않도록 배열하는 방법 Steve

5. 5 The Architecture Steve Steve consists of three modules: perception, cognition and motor control

6. 6 Primitives The cognition module generates Steve’s communicative behavior by dynamically selecting next action from a list of primitives Primitives –Speak –Move to an object –Manipulate an object –Visually check an object –Point at an object –Give tutorial feedback –Offer turn –Listen to student –Wait for someone Steve –Acknowledge an utterance –Drop hands –Attend to Action

7. 7 React to Interruption & Context (1) To react to interruption –In dynamic virtual worlds, to react to the unexpected events is important for task-oriented collaboration –It is important to maintain coherence of an agent’s behavior Steve considers 2 separate but complementary types of context 1. Task context –Continually monitors the virtual world –Uses the task model to plan to complete the task –Uses variant of partial-order planning techniques 2. Dialogue context –Represents the state of interaction between a student and Steve –Uses focus stack: represents the hierarchy of tasks, subtasks, and currently engaged actions –Maintains a record, Steve’s answer etc. Steve

8. 8 React to Interruption & Context (2) Based on the current task and dialogue contexts, Steve can choose next action Steve can choose an action of following three roles –Steve can respond to the student to know a student’s request –Steve can choose for himself how to advance the collaborative dialogue –Steve can choose a turn-taking or grounding act that helps regulate the dialogue without advancing the task In case several actions are appropriate at one moment, –Steve choose next action considering priorities –Actions with low priority will be performed later, not be deferred Summary –Steve does not include an emotion model, but –It serves a valuable foundation for MRE model Steve

9. 9 Jack and Steve vs. Steve The Jack and Steve system was predicated on two basic claims: –Plans and plan reasoning can mediate expressive behavior –Small biases in how plans are evaluated and generated can result in large systematic differences in agent behavior Jack and Steve vs. Steve Jack and Steve Steve To support biases in plan evaluation, J-S include a richer plan representation To translate small biases into large external variations, J-S focused on plan generation Steve focused more on plan execution and repair J-S explored how biases in the plan generation process could support a variety of non-collaborative interactions as well, but as the focus was exploring systematic differences in joint behavior, agents only interact with other computational agents and not with human users Steve focused on collaborative interactions between agents and human users

10. 10 Motivating Example This example shows two separate runs of system where the only difference is a change in the personality of Steve (rude / fair) –Jack: I want to make some big money –Steve: I want to catch some waves Jack and Steve Mental state of each agent at specific point in the interaction Mental state shows individual action, plan, emotion etc.

11. 11 Plan-Based Social Appraisal (1) Jack and Steve supports expressive and flexible interactions by implementing social reasoning as a layer atop a general-purpose planning system Planning system provides –Domain-independent representations of world actions –General reasoning mechanisms that construct partial plans, repair interactions among them, and oversee plan execution The social layer –Manages communication –Biases plan generation and execution according to social context Jack and Steve General purpose planning system Social reasoning layer

12. 12 Plan-Based Social Appraisal (2) To support various social interactions, the social reasoning layer must provide a rich model of the social context The social situation is described in terms of many static and dynamic features Static features –include innate properties of the character –social role and small set of personality variables Dynamic features –Are derived from inference procedure that operate on the current mental state of agent –Include - current communication obligation - various relations between plans in memory (e.g. your plans threaten my plans) - a model of the emotional state of the agent Jack and Steve

13. 13 Plan-Based Social Appraisal (3) One novel aspects of this system is that the social layer alters the planning process fundamentally In terms of planning, the social layer can bias planning to be more or less considerate to the goals of other participants In terms of communication, agents can vary from bossy agents that try to tell others what to do to passive agents that avoid interactions or social conflicts Jack and Steve

14. 14 Social Context, Operator, Rule Jack and Steve system maintains a rich representation of the social context The social context of Jack and Steve is divided into 3 categories of plan context, emotional context, and communicative context Social operators are actions that occur at the social level Social operators are subdivided into meta-planning operators and communicative operators. Distinct personalities are implemented via a set of social rules Social rules execute sequences of social operators based on appraised features of the social context Jack and Steve system include 30 social rules Jack and Steve

15. 15 Introduction Carmen’s Bright IDEAS (CBI) –An agent-based system designed to realize an Interactive Pedagogical Drama –The pedagogical goal is to help mothers of pediatric cancer patients –The drama mirrors the mother’s own problems –It allows the learner to interactively influence how Carmen copes with problems In CBI –to model the causes of emotions is important –agents needed effective ways to convey the impact of emotion on both the agent’s dialogue and physical behavior Carmen’s Bright IDEAS

16. 16 The Drama of Carmen’s Bright IDEAS CBI is an interactive pedagogical drama Carmen discusses her problems with a clinical counselor, Gina Gina suggests a problem solving technique called Bright IDEAS Bright IDEAS means –Bright: positive attitude –I: Identify a solvable problem –D: Develop possible solution –E: Evaluate your options –A: Act on your plan –S: See if it worked User, the human mother, interacts with the drama by making choice for Carmen Both Gina’s dialogue and the user’s choices influence the cognitive and emotional state of Carmen (in conflicting ways) Carmen’s Bright IDEAS

17. 17 Interaction Model Carmen’s Bright IDEAS In this interaction model, the user gets a vivid demonstration Learner influences Carmen by selecting thought balloons

18. 18 CBI Agent Model Carmen’s Bright IDEAS IPD have 5 main components, but the discussion will focus on the on-screen character agents Each on-screen character is realized by an agent architecture that has modules for problem solving, dialogue, emotional appraisal, and behavior generation

19. 19 Emotional Model The pedagogical goal was for the mothers to learn how to choose and carry out the right coping strategy for a given situation Cognitive appraisal theory by Richard Lazarus influences CBI –This organizes human behavior around appraisal and coping –Appraisal leads to emotion by assessing the person-environment relationship –Coping and appraisal interact and unfold over time, supporting the temporal character of emotion evident in human behavior Coping is the process of dealing with emotion –Problem-focused coping –Emotion-focused coping Carmen’s Bright IDEAS

20. 20 Remarks The design of dialogue, emotions, and expressive behavior systems was driven by a need to convey deep inner conflicts and how those conflicts play out expressively Carmen’s Bright IDEAS

21. 21 Introduction The Mission Rehearsal Exercise (MRE) system –Brings ideas from each of the preceding systems –Creates a broader and more flexible array of expressive behaviors –The goal is to teach leadership skills in high-stakes social situation To model such dramatic and interactive scenario, the MRE –Combines Steve’s ability to flexibly interact with a human user –Augment it with the richer social and emotional behaviors of CBI and Jack and Steve The MRE combines a variety of capabilities in service of realistic and natural collaboration with virtual humans Example Mission Rehearsal Exercise

22. 22 Cognition and Emotions Our virtual humans must –Provide realistic expressive behaviors –Require cognitive machinery to recognize which behaviors are appropriate in the course of an unscripted interaction with a human user The Steve system is selected as a starting point since it supports flexible face-to-face interactions with a human user Integrate plan-based appraisal model of the Jack and Steve system into the Steve system Integrate CBI’s coping model –Coping strategies were recast explicitly into procedures that updated Steve’s task representations and reasoning processes Mission Rehearsal Exercise

23. 23 Cognition and Emotions Mission Rehearsal Exercise Steve Plan-based appraisal model of Jack and Steve Coping model of Carmen’s Bright IDEAS Steve supports flexible face-to-face interaction with human users This allows agents to negotiate over tasks and express emotions, i.e. task model is extended in a number of ways to represent the socio-emotional context Integrating CBI’s coping model results in a tight integration between appraisal, coping and task reasoning that closely follows the cognitive appraisal theory of Richard Lazarus + +

24. 24 Physical Behavior Virtual humans in the MRE attempt to manifest the rich dynamics of this cognitive and emotional inner state through each character’s external behavior The key challenge is the range of behaviors that must be integrated: Each character’s body movements must reflect –Its awareness of events in the virtual world –Its physical actions –The number of non-verbal signals that accompany speech during social interaction –Its emotional reactions Expressive physical behavior in the MRE agents integrates the task-related non-verbal behaviors of the Steve system and the coping behaviors of CBI Mission Rehearsal Exercise

25. 25 Conclusion Steve serves as the foundation for the virtual humans in MRE system as his ability to interleave task-related behaviors and face-to-face dialogue in dynamic virtual worlds The Jack and Steve system contributed a model of task-oriented emotional appraisal and a model of socially situated planning The CBI system contributed a complementary model of emotional appraisal focusing on social relationships MRE virtual humans integrate many of the ideas from these three prior systems