Constructing Complex NPC Behavior via Multi- Objective Neuroevolution Jacob Schrum – Risto Miikkulainen – University of Texas at Austin Department of Computer Sciences

Introduction Goal: discover NPC behavior automatically Goal: discover NPC behavior automatically Benefits Benefits Save production time/effort Save production time/effort Learn counterintuitive behaviors Learn counterintuitive behaviors Find weaknesses in static scripts Find weaknesses in static scripts Tailor behavior to human players Tailor behavior to human players Challenges Challenges Games are complex Games are complex Multi-modal behavior is required Multi-modal behavior is required Success often depends on multiple objectives Success often depends on multiple objectives RL & Evolution popular approaches RL & Evolution popular approaches Current methods do not explicitly consider multiple objectives! Current methods do not explicitly consider multiple objectives!

Multi-Objective Optimization Exceptional performance in any objective is important Exceptional performance in any objective is important Imagine game with two objectives: Imagine game with two objectives: Damage Dealt Damage Dealt Health Remaining Health Remaining A dominates B iff A is strictly better in one objective and at least as good in others A dominates B iff A is strictly better in one objective and at least as good in others Population of points not dominated are best: Pareto Front Population of points not dominated are best: Pareto Front High health but did not deal much damage Dealt lot of damage, but lost lots of health Tradeoff between objectives

NSGA-II Evolution: natural approach for finding optimal population Evolution: natural approach for finding optimal population Non-Dominated Sorting Genetic Algorithm II* Non-Dominated Sorting Genetic Algorithm II* Population P with size N; Evaluate P Population P with size N; Evaluate P Use mutation to get P´ size N; Evaluate P´ Use mutation to get P´ size N; Evaluate P´ Calculate non-dominated fronts of {P  P´} size 2N Calculate non-dominated fronts of {P  P´} size 2N New population size N from highest fronts of {P  P´} New population size N from highest fronts of {P  P´} *K. Deb et al. 2000

Neuroevolution Genetic Algorithms + Artificial Neural Networks Genetic Algorithms + Artificial Neural Networks NNs good at generating behavior NNs good at generating behavior GA creates new nets, evaluates them GA creates new nets, evaluates them Three types of mutations (no crossover used) Three types of mutations (no crossover used) Perturb Weight Add Connection Add Node

Battle Domain 15 NPCs attack player Player Bat Infinite Plane

NPC Fitness Objectives NPCs have several goals NPCs have several goals Inflict damage as a group → Attack Assist Bonus Inflict damage as a group → Attack Assist Bonus Avoid damage individually → Damage Received Avoid damage individually → Damage Received Stay alive individually → Time Alive Stay alive individually → Time Alive Objectives are contradictory and distinct Objectives are contradictory and distinct NPCs take damage from batPlayer is knocked back by NPC

Experiment Multi-objective vs. Single-objective Multi-objective vs. Single-objective Single-objective uses weighted sum of z-scores Single-objective uses weighted sum of z-scores Evolve each method against static player strategies Evolve each method against static player strategies Spinning strategy Spinning strategy Alternating strategy Alternating strategy Chasing strategy Chasing strategy NPCs must beat previous player strategy to progress NPCs must beat previous player strategy to progress

Results Differences for Alternating and Chasing significant with p <.05

Evolved NPC Behaviors Single-Objective Single-Objective Cannot find balance between attack and defense Cannot find balance between attack and defense Behavior is ineffectual Behavior is ineffectual Multi-Objective Multi-Objective Baiting Behavior Baiting Behavior One agent acts as bait One agent acts as bait Others attack player on blind side Others attack player on blind side Charging Behavior Charging Behavior Try to dodge bat while rushing in Try to dodge bat while rushing in Maximizes damage dealt Maximizes damage dealt

Discussion Benefits Benefits Multi-modal behavior Multi-modal behavior Exploits dynamics of environment Exploits dynamics of environment Individuals can have different behaviors Individuals can have different behaviors Future Work Future Work Evolve against humans Evolve against humans Learn many behaviors in advance Learn many behaviors in advance More complex games: Unreal Tournament More complex games: Unreal Tournament

Conclusion Multi-objective encourages multi-modal behavior Multi-objective encourages multi-modal behavior Multi-objective better than single-objective Multi-objective better than single-objective Deals with contradictory objectives Deals with contradictory objectives Leads to several different but effective solutions Leads to several different but effective solutions Lead to multi-modal behavior in future games Lead to multi-modal behavior in future games

Questions? See movies at: See movies at: