1. 1 Modeling Emotions in Game Characters: Theoretical Foundations & Practical Guidelines Joost Broekens SEP 10, 2009 ACII 2009 Amsterdam Eva Hudlicka

2. 2 Outline Emotions & Games: Affective Gaming Theoretical Foundations Practical Guidelines Conclusions

3. 3

4. 4

5. 5 Where We Are Now Tremendous advances in gaming technologies –From dots to cinematic characters –From single screen to virtual world Focused primarily on: –Physical realism of game characters & environmentsgame characters –Complexity & performance of simulations & networking –Rich gameplay/interaction Today’s games still limited in: –Affective realism for game character (NPC) –Social complexity & realism of interactions between and with NPC –Ability to adapt to player’s state: personalization of the experience (see Georgios’ talk)

6. 6 We are about here… In terms of the full potential of gaming..

7. 7 To achieve the “next big leap”..in engagement & effectiveness Games would benefit from: Adapting to players’ affective states Enhancing social & affective complexity & realism of: –Game characters –Their interaction with each other & the players –Game narrative as a whole (interactive storytelling)

8. 8 Current focus of Affective Gaming AC Methods & Techniques Relevant for Affective Gaming Sensing & recognition of players’ emotions –Adaptive gaming –Game control Expression of emotions by game characters –More realistic visuals Models of emotion in game characters –To support complex, autonomous behavior –To support adaptive behavior Models of players’ emotions –Affective user models to support game personalization (Yannakakis) Affective game evaluation –Use affective feedback to develop games with desired affective profiles

9. 9 Emotion Models in Affective Gaming Game characters’ emotions –To generate realistic & affectively-complex character behavior & believable affective expressions –You need Emotion generation & emotion effects on cognition & behavior … responsive to dynamically-evolving game scenario & player behavior … in real-time (practical issues such as scalability and performance) Players’ emotions –Help recognize player emotions in real-time –Develop gameplay adaptations –Generate more realistic player avatar behavior

10. 10 Outline Emotions & Games: Affective Gaming Theoretical Foundations Practical Guidelines Conclusions

11. 11 Emotion and affect in human behavior Basic emotions: fear, anger, happiness, sadness, surprise, disgust Short episode of multimodal activity triggered by event: –subjective feelings (the emotion we normally refer to), –tendency to do something (action preparation), –facial expressions, –evaluation of the situation (cognitive evaluation, thinking), –physiological arousal (heartbeat, alertness). Affect = related to emotion, mood and attitudes: –emotion: object directed, short term, high intensity, action oriented, differentiated. –mood: usually unattributed and undifferentiated, longer term, low intensity. –attitude: affect permanently associated with an object/person –affect: abstraction of emotion/mood in terms of, positiveness/negativeness and activation/deactivation (e.g., Russell, Rolls).

12. 12 Emotion and affect in human behavior Situational evaluation (intrapsychic) and communication (interpersonal). Heuristic relating events to actions through an evaluation of personal relevance (e.g., goals, needs, drives, motivations): –Evaluation of personal relevance of event –Speeds-up decision-making –fast reactions and action preparation –influence information processing Learning & adaptation, attention, mental search/planning, creativity, etc.. Communication medium: –communicate internal state and intention –alert others –show empathy (understanding of situation of others)

13. 13 Emotion: dimensions Set of underlying common factors of emotion –E.g. Russell, Mehrabian, Wundt (Breazeal, 2003) adapted from Russell (1997)

14. 14 Emotion: categories Sadness: –Low arousal –Face: sad –Avoid –Bad feeling Anger: – High arousal – Face: angry – Approach – Bad feeling Joy: – High arousal – Face: happy – Play – Good feeling A small number of hardwired basic emotions exist – E.g. Tomkins, Izard, Ekman, Panskepp Category is a typical “emotion syndrome” – A complex of physiology, expression, behavior, and feeling

15. 15 Emotion: components Parallel evaluation in terms of appraisal dimensions –(E.g. Scherer: stimulus checks) NoveltyPleasantnessGoal/Need conduciveness Coping potential Sensory- Motor level Sudden, intense stimulation Innate preferences/ aversions Basic needsAvailable energy Schematic level Familiarity: schema matching. Learned preferences or aversions Acquired needs motives Body schema Conceptual level Expectations: cause/effect, probability Recalled, anticipated, or derived positive- negative estimates Conscious goals, plans Problem-solving ability.

16. 16 Emotion: summary

17. 17 Outline Emotions & Games: Affective Gaming Theoretical Foundations Practical Guidelines Conclusions

18. 18 Main questions I want to build an emotional agent, now –Where do I start? –Where do I end? –When did I succeed? A tree has no emotions A mosquito has “emotional behavior” in the eye of the emotional beholder (fear, frustration). A lizard has emotions, although probably only a couple of them (fight, flight, satisfaction) A grown buffalo has emotions, and probably quite a lot of them (fear, excitement, joy, attraction, satisfaction). Agent? Emotion? Let’s dive into the details…

19. 19 Agent related issues Really understand the environment & agent / NPC / Virtual Character / etc. What is the observable behavior of my agent? –What are its potential actions? What is the environment of the agent? –What are the observable events for that agent? What are the beliefs and motivations for the agent? –What does the agent want, like, dislike etc… Do emotions make sense, what are they supposed to add? –HCI aspects: Believability and effectiveness (serious games), fun (games), interaction –Agent autonomy aspects: intelligence, planning, learning and adaptation, etc.?

20. 20 Emotion related issues How do I select an emotion theory that best matches the: –Internal workings of my agent (motivations, beliefs, etc.) –Observable behavior of my agent (actions) –Environment (events) How do I value events in relation to the emotion theory? –Direct versus indirect mapping. How do I represent emotion internally? –States, traits, moods, attitudes –Factors, categories, components –Level of complexity of the “emotion object” (level of detail) How do I represent emotion dynamics? –Onset, decay, mixed emotions (similar versus opposing), etc. How do I link emotion to emotion effects, expression and behavior? –Direct vs Indirect, feature vs expression –Facial expression, body posture, movement, behavior. –Effects on problem solving, planning, decision making, learning, etc.

21. 21 Emotion architecture for an NPC Guideline to make your choices Not everything is needed in your model! EnvironmentInterpretationBehavior Internal Events External Events Psychological Biological Social Physiological Affect Type Trait Emotion Mood Attitude Face Posture Actions Behaviors Emotion generation Emotion representation Emotion effects Effects Psychological Biological Social Physiological Thoughts

22. 22 Outline Emotions & Games: Affective Gaming Theoretical Foundations Practical Guidelines: example integrating generation and effects Conclusions

23. 23 First Person Shooter NPC The agent –Monster (zombie) –Needs: Health (hit-points) Hunger (eat flesh, drink) –Potential events see_human, see_animal, see_water, health_high, health_low energy_high, energy_low Hunger_high, hunger_low attacked –Potential behaviors wander, chase_human, chase_animal, move_to_water, search_X, flee eat_human, eat_animal, drink, sleep Source: resident evil

24. 24 First Person Shooter NPC The emotions –Why? Add realism Influence action choice Influence attention –Expression: Fear, aggression, happiness, panic. –Behavior: Trigger search, chase and flee –Attention Field of view (broad-narrow) Object fixation Self, other oriented Source: resident evil

25. 25 First Person Shooter NPC: Select theory

26. 26 Homeostasis: why? Drives & needs –Drive: motivator for particular behavioral “programs” –Needs: biological drives Sleep Hunger Thirst Etc. When drives not met (or met too much) this influences: –Emotion, behavior and attention to regulate drives Process is called homeostasis: –Behavior is aimed at satisfying needs and regulates emotion –Emotion is a signal that biases organisms towards interaction that satisfies current needs.

27. 27 First Person Shooter NPC: Select affect types Affect typeCharacteristics EmotionQuick onset and decay, intensive, targeted, expressed, direct link to behavior MoodSlow onset and decay, moderate, indirect link to behavior TraitBias to mood, emotion (propensity to be in a certain emotion or mood) and nature of affective dynamics. AttitudeEmotional counterpart of cognitive event, influence emotion and mood Complex  affect dynamics  Simple Updates to affect type values are based on changes (deltas) resulting from interpretation Result of interpretation is an absolute value that “sets” the affect type values

28. 28 First Person Shooter NPC: expression Direct mappingIndirect mapping Complete Expression based Emotion representation  Emotion expression (I am sad  I express sadness) Emotion representation  Expression representation  Emotion expression (Sad  Sad face  Express) Feature based e.g. FACS Emotion representation  expressive features (I am sad  I express eyes, lips) Emotion representation  Feature representation  Expressive features (Sad  Sad eyes, lips  Express)

29. 29 First Person Shooter NPC Architecture EnvironmentInterpretationBehavior Health_low Health_high Hunger_low Hunger_high See_human See_animal See_water Biological Needs: Health Hunger Affect Type Emotion (categories) Fear Aggression Happiness Panic Fear Anger Happiness Panic Search_X chase_X Flee, etc. Emotion elicitation Emotion representation Emotion effects Effects attention

30. 30 First Person Shooter NPC Instrumentation –Direct mapping based on drives/needs and anticipated effect. Event/behaviorDrive effectAnticipation effect See_humanHunger -0.5 Health -0.1 See_animalHunger -0.25 Energy -0.2 Chase_humanHealth -0.1Health –0.3 Hunger -0.5 Chase_animalHealth –0.1Hunger –0.25 attackedHealth -0.3 Eat_humanHunger -0.5 Eat_animalHunger –0.25 WanderHealth -0.1 SearchHealth -0.2Hunger -0.1 SleepHealth +0.2 Etc.

31. 31 First Person Shooter NPC Affect dynamics –FeltDrive=drive+1/2*anticipation Expression –Based on mapping of FeltDrive combination to emotion category HungerHealthEmotion LowhighHappiness HighhighAnger HighlowPanic Low Fear

32. 32 Emotion effects Fewer theories exist than for emotion generation Specific mechanisms of emotion effects not as well developed Some available theories: –Distinct modes of processing associated with different emotions (Oatley & Johnson-Laird, 1987) –Spreading activation & priming (Bower, 1984; Derryberry, 1988) –Emotions as patterns of parameters modulating processing (Fellous, Matthews, Ortony et al., Hudlicka, Ritter) –Componential model of effects (Scherer et al., Lerner & Tiedens)

33. 33 Emotion effects Many relationships between affect and cognition: Mood influences information processing style –Top-down (positive) versus bottom-up (negative) –Heuristic/generic/assuming/creative processing (positive) versus detail/feature/critical/procedural processing (negative) Mood influences learning –Flow, boredom, frustration, etc. Emotion influences information processing, e.g., –anxiety  threat bias (Mineka et al., 2003) –aggression  higher risk tolerance (Lerner & Tiedens, 2006) –arousal is related to attention capacity

34. 34 Emotion Effects in NPC Context Zombie tasks: –Maintain health –Maintain energy –Stay fed NPC behavior changes depending on emotion –homeostatic interpretation: specific emotions favor specific behaviors to satisfy needs. –Happiness: increase tendency to…wander, sleep. –Panic: increase tendency to…search_animal, chase_animal, flee_human –Aggression: increase tendency to… search_X, chase_X, eat_X –Fear: increase tendency to… wander, flee_X.

35. 35 Emotion Effects in NPC Context (2) Zombie attention: –Nr of humans/animals tracked (parameter for field of view: detail vs. global) –Object fixation (parameter for stability & capacity working memory) –Focus on self-related needs vs. other (parameter for self-other directedness) Effects (provided as example!): –Pos. valence: global attention, distractible (e.g., Dreisbach & Goshke, 2004) –Neg. valence: focus on details, tunnel vision (e.g., Clore, Isen) –Anger: Attribution of hostility in others (Lerner & Tiedens, 2006) –Fear: Threat-directed attention (Mineka et al., 2003) Emotion influences attention parameters (parameter-based effect) –Happy: Large field of view, weak object fixation, balanced needs –Panic: Small field of view, weak object fixation, balanced needs –Aggression: Small field of view, strong object fixation, hunger –Fear: Small field of view., strong object fixation, health

36. 36 Outline Emotions & Games: Affective Gaming Theoretical Foundations Practical Guidelines Conclusions

37. 37 Conclusions Affect-focused game design –emotion plays a central role in: Gameplay design - adapt to player emotions Game character design - more affective realism “Assist Me, Challenge Me, Emote Me” (Gilleade, Dix & Allanson 2005) Affective game engines (Hudlicka, 2009): –Game development tools that support: Sensing & recognition of player emotions Gameplay adaptation to player affect Design of affective game characters

38. 38 Modeling Emotions in Game Characters Affective gaming Emotion recognition for adaptive game control Emotion in NPC for adaptation, realistic behavior & expression Models of players’ emotions for game personalization Affective feedback for game evaluation and development Affective gaming Emotion recognition for adaptive game control Emotion in NPC for adaptation, realistic behavior & expression Models of players’ emotions for game personalization Affective feedback for game evaluation and development NPC emotionsNPC emotions Generate realistic affective character behavior & expressions Emotion effects on cognition & behavior … responsive to dynamic scenario & player behavior … in real-time NPC emotionsNPC emotions Generate realistic affective character behavior & expressions Emotion effects on cognition & behavior … responsive to dynamic scenario & player behavior … in real-time What does my NPC need? Agent issues observable events, actions, architecture complexity, motivation Game issues Computational complexity available for emotion, Intended player experience Emotion issues Theory selection, Emotion representation and dynamics, Emotion effects on cognition, expression, behavior What does my NPC need? Agent issues observable events, actions, architecture complexity, motivation Game issues Computational complexity available for emotion, Intended player experience Emotion issues Theory selection, Emotion representation and dynamics, Emotion effects on cognition, expression, behavior Direct mappingIndirect mapping Complete Expression based Emotion representation  Emotion expression (I am sad  I express sadness) Emotion representation  Expression representation  Emotion expression (Sad  Sad face  Express) Feature based e.g. FACS Emotion representation  expressive features (I am sad  I express eyes, lips) Emotion representation  Feature representation  Expressive features (Sad  Sad eyes, lips  Express) Requirements for Affective NPCs - Affect-focused game design perspective - Tools for analysis of affective requirements within game - Tools for design and development of affective NPCs - Affective Game Engines Requirements for Affective NPCs - Affect-focused game design perspective - Tools for analysis of affective requirements within game - Tools for design and development of affective NPCs - Affective Game Engines Emotion Architecture Overview Affect Type Alternatives Theoretical Perspectives Implementation Alternatives