Evolving the goal priorities of autonomous agents Adam Campbell* Advisor: Dr. Annie S. Wu* Collaborator: Dr. Randall Shumaker** School of Electrical Engineering and Computer Science* Institute for Simulation and Training**

Goal Develop a controller for a team of collaborating autonomous vehicles Simple implementation Allows for new goals (behaviors) to easily be added Would like to add social interactions between the agents in the future Evolve the parameters of this controller to determine how the goal weights correlate with different environments These simple tests will allow us to get a better idea of how the goals interact with one another Having the goal priorities evolve will allow us to more easily hand code the parameters for future experiments

Motivation Prioritizing conflicting, parallel goals in a robot controller is a difficult, and open problem in artificial intelligence Action selection This research examines an evolutionary approach to the Action Selection problem Imagine an insect with two goals Get food Avoid predator When should the “get food” goal priority be higher than the “avoid predator” goal? Two general methods Take one action Combine actions Used in this research

Genetic algorithm Survival of fittest amongst problem solutions General algorithm… 1) Initialize random population 2) Evaluate population 3) Select individuals 4) Recombine/mutate selected individuals 5) If stopping condition not satisfied 6) GOTO 2

Genetic algorithm example Problem: find all black squares Random population Fitness

GA example continued Selected populationCrossover& Mutation 5 Fitness Legend:  Crossover point

How is the GA used? Immediate goal functions Produce a vector indicating where the agent should move in order to best satisfy the goal Each immediate goal has a weight associated to it Five immediate goal functions Avoid agent Avoid obstacle Momentum Go to area of interest (AOI) Follow obstacle

Additional parameters Randomness Comfort Allows obstacle following to occur

Parameters Simulation parameters Test cases2 Simulation ticks10000 Agents25 Runs per test case30 Weight range[0.0, 1.0] Genetic algorithm parameters Population size50 Generations50 Crossover rate0.9 Mutation rate (per weight)0.005

Two scenarios Environment 1Environment 2

Average fitness Agents must survive and see as many AOIs as possible Not much difference in fitness between two scenarios

Evolved parameters

Evolved agents in action

Summary and conclusion Discussed action selection problem in artificial intelligence and showed an evolutionary approach to solving Method combines actions of goals Tested approach on simple problem scenarios Performed well on both scenarios New behaviors (goals) can easily be added to the system The parameters evolved are specific to the environment they were learned in

Future work Social interactions between agents Allow communication of data between agents New immediate goal functions needed Allow agents to have more than one set of goal weights Depending on the agent’s state (hungry, low on fuel, in danger, etc.) use a different set of goal weights Other ways to combine vectors from immediate goal functions Non-linear combination of vectors Genetic programming Currently being worked on at George Mason University Better test scenarios Evolve parameters that generalize well to unseen environments

Related work Action selection M. Humphrys. Action selection in a hypothetical house robot: Using those RL numbers. In Proceedings of the First International ICSC Symposia on Intelligent Industrial Automation (IIA-96) and Soft Computing (SOCO- 96), M. Humphrys. Action selection methods using reinforcement learning. In From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, Cambridge, MA, pages MIT Press, Bradford Books, Robot control R. C. Arkin. Motor schema based navigation for a mobile robot. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Raleigh, NC, pages , May O. Buffet, A. Dutech, and F. Charpillet. Automatic generation of an agent's basic behaviors. In Proceedings of the 2nd International Joint Conference on Autonomous Agents and MultiAgent Systems (AAMAS'03), J. Casper, M. Micire, and R. R. Murphy. Issues in intelligent robots for search and rescue. In Proceedings SPIE Volume 4024, Unmanned Ground Vehicle Technology II, pages , July S. Koenig and M. Likhachev. Improved fast replanning for robot navigation in unknown terrain. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages , J. Rosenblatt. DAMN: A distributed architecture for mobile navigation. In Proceedings of the 1995 AAAI Spring Symposium on Lessons Learned from Implemented Software Architectures for Physical Agents. AAAI Press, March S. P. Singh, T. Jaakkola, and M. I. Jordan. Learning without state-estimation in partially observable Markovian decision processes. In International Conference on Machine Learning, pages , 1994.