1. Artificial Life/Agents Creatures: Artificial Life Autonomous Software Agents for Home Entertainment Stephen Grand, 1997 Learning Human-like Opponent Behaviour for Interactive Computer Games Christian Bauckhage, Christian Thurau, and Gerhard Sagerer, 2003 Evolving Neural Network Agents in the NERO Video Game Kenneth O. Stanley, Bobby D. Bryant, Risto Miikkulainen, 2005

2. Creatures 1997 state of the art Artificial Life game Creatures with neural network brains and an evolving genome Learn by punishment/reward reinforcement Can learn rudimentary ‘verb-object’ language Sense of sight, sound, touch Complex biochemistry (metabolism, immune system, genetically encoded morphology)

3. Creatures’ Brains Hebbian learning ~1000 neurons, ~5000 synapses Organised into ‘lobes’:

4. Characteristics Designed for efficiency (runs on 1997 commodity hardware) Limited number of neurons Brain model is also limited, restricts potential functions

5. Learning Human-like Opponent Behaviour

6. Neural-network control system for a Quake II bot Offline, supervised learning Feed-forward, back-propagation learning, multilayer perceptron network One network for moving, one for aiming Trained to learn one path, then multiple paths, then moving and aiming

7. Advantages Potentially cheaper and faster than scripting bots Generalises to novel situations More efficient than on-line learning bots Good introduction to learning agents

8. Problems Paper is horribly structured and hard to read Assumption: only the agent’s current state/environmental influences matter! Experiments didn’t work very well Bots still static, can’t learn opponent tactics Maybe difficult to get training data

9. Evolving Neural Network Agents in NERO

10. Online, reinforcement learning Agent fitness increased by learning and evolution Player can train teams of bots to compete against each other in increasingly complex training scenarios Won Best Paper Award at the IEEE Symposium on Computer Intelligence and Games

11. The Network

12. Learning Method rtNeat: basically, a technique for evolving increasingly complex neural networks Benefits over traditional RL: – Diversity increased/maintained through speciation – Can keep a memory of past events Player provides customised fitness function NERO removes the worst agents, breeds the best ones

13. Currently NERO is quite simple Paper presents no quantitative results, but results seem promising