Presentation on theme: "Evolving Levels for Super Mario Bros Using Grammatical Evolution & Evolving a Ms. Pac-Man Controller Using Grammatical Evolution Presentation by Alex van."— Presentation transcript:
Evolving Levels for Super Mario Bros Using Grammatical Evolution & Evolving a Ms. Pac-Man Controller Using Grammatical Evolution Presentation by Alex van Poppelen, Boris Jakovljevic, and Nadia Boudewijn
The papers Written by several people (1 person worked on both papers) from: Natural Computing Research & Applications Group, University College Dublin, Ireland & Center for Computer Games Research, IT University of Copenhagen, Denmark Fairly recent: 2010 (Ms. PacMan) / 2012 (SMB)
Contents Grammatical Evolution (Nadia) Using grammatical evolution to evolve a Ms. Pac-Man controller (Alex) Using grammatical evolution to evolve levels for Super Mario Bros (Boris)
Grammatical Evolution (GE) Relatively new concept (1998) Related to the idea of genetic programming (GP): find an executable program that will achieve a good fitness value for the given objective function Main difference: GP: uses tree style structured expressions that are directly manipulated GE: manipulates integer strings, that are subsequently mapped to a program through the use of a grammar
Grammatical Evolution Integer strings -> grammar -> program fitness(program) -> integer string Inspired by nature: separate genotype from phenotype Genotype: integer string Phenotype: tree-like structure that is evaluated recursively (same as GP) Benefit of GE’s modular approach: no specific algorithm or method is required to perform search operations It is possible to structure a GE grammar that for a given function/terminal set is equivalent to genetic programming.
What is a grammar? “Grammar” can apply to: Natural Language (Linguistics): a set of structural rules governing the composition of clauses, phrases and words in any given natural language. Formal Language (mathematics, logic,and theoretical computer science): is a set of production rules for strings in a formal language. A grammar does not describe the meaning of the strings or what can be done with them in whatever context.
Only syntax – NO semantics 1. The ate bear the fish 2. The bear ate the fish 3. The fish ate the bear The grammar for the natural language English will accept sentences 2 & 3, but will reject sentence 1. What does it mean to say that a grammar accepts some string (or sentence)?
Example CFG and parse tree Language: a n b n Context Free Grammar: S :- a S b S :- Є Parse tree for “aabb”: Є
Context-Free grammar? Equivalent to Backus-Naur Form CF-grammar: lexicon of words and symbols + production rules Two classes of symbols: terminal + non-terminal Formal language defined by a CFG: set of strings derivable from the start symbol CF-grammar use: A device for generating sentences A device for assigning a structure to a given sentence
Formal CFG definition
Expressive Power Formal mechanisms (CFG’s, Markov Models, transducers etc.) can be described in terms of their power: = in terms of the complexity of the phenomena they can describe One grammar has greater generative power or complexity than another if it can define a language the other cannot define. Chomsky Hierarchy: hierarchy of grammars, where the set of languages describable by grammars of greater power subsumes the set of languages describable by grammars of less power.
Back to GE Population: a set of integer strings Applying a mapping rule, these integer strings are converted into problem instances following the rules of the Context-Free Grammar involved
Criticism and Variants Due to the fact that GE uses a mapping operation, GE’s genetic operators do not achieve high locality: small changes in the genotype always result in small changes in the phenotype This is a highly regarded property of genetic operators in evolutionary algorithms. One possibility for variants is to use particle swarm optimization to carry out the search instead of genetic algorithms
Evolving a Ms. Pac-Man Controller using GE Deterministic?
Ms. Pac-Man Competition Aims to provide best software controller for the game of Ms. Pac-Man Best human player score: 921,360 Best computer score: 30,010 Hand-coded agent Developed by Matsumoto et al from Kyoto, Japan Year 2009
Using an Evolutionary Approach Previous approach by Koza: Used Genetic Programming to combine pre-defined actions and conditional statements to evolve a simple Ms. Pac-Man player Goal: Achieve highest score Fitness function: Points earned per game Used reinforcement learning and the cross-entropy method to assist agent in learning appropriate decisions This paper: Attempts to successfully evolve rules in the form of “if then perform ” Uses Grammatical Evolution
Representation Grammatical evolution represents programs as a variable length linear genome Genome is an integer array of elements called codons Genotype mapped to phenotype using grammar in Backus- Naur Form Mapping function: Rule = c mod r c is the codon integer value r is the number of choices for the current symbol Codons may remain unused, or there may not be enough. In the latter case, may wrap back to the beginning up to a maximum number of times
Experimental Setup One level, one life Fitness function: Add scores for each pill, power pill, and ghost eaten Generation approach Population size 100 Ramped half and half initialization method (max tree depth 10) Tournament selection size 2 Int flip mutation (probability 0.1) One-point crossover (probability 0.7) Maximum of 3 wraps allowed to “fix”invalid individuals
Best Evolved Controller Very aggressive Heads for power pills and then tries to eat all edible ghosts without looking to see if there are inedible ghosts in the way
Benchmarking Performance Compared evolved agent to 4 other agents Hand-coded agent Random agent (chooses up, down, left, right, or neutral at every time step) Random Non-Reverse agent (same as random, but no back- tracking) Simple Pill Eater (heads for nearest pill, ignores all else)
Different Ghost Teams Three different ghost teams were used to test the agents Random team (Each ghost chooses a random direction each time step, no back-tracking) Legacy team (Three ghosts use different distance metrics: Manhattan, Euclidean, and shortest path distance. Last ghost makes random moves) Pincer team (Each ghost attempts to pick the closest junction to Ms. Pac-Man within a certain distance in order to trap her)
Conclusions Evolved controller beat their own hand-coded controller against all ghost teams Evolved controller did not match of exceed the score of Matsumoto’s hand-coded agent But: Matsumoto’s agent was given three lives, could earn more lives, and had more than one level to play Our question: Why didn’t they evolve their controller under the same circumstances?
Evolving Levels for Super Mario Bros Using Grammatical Evolution Boris Jakovljevic
The paper Authors: Noor Shaker Miguel Nicolau Georgios N. Yannakakis (Member, IEEE) Julian Togelius (Member, IEEE) Michael O’Neill
Table of Contents Introduction Background Testbed Platform Game Level Representation GE-based Level Generator Other Generators Expressivity Analysis Conclusions and Future Work
Framework extended through: more informative aesthetic measures of generators’ expressivity applying the above measures to analyze and compare expressivity ranges of 3 level generators
Testbed Platform Game A modified version of Markus “Notch” Persson’s Infinite Mario Bros (IMB)*. Super Mario Bros – a very rich Environment Representation. J. Togelius, S. Karakovskiy, J. Koutnik, and J. Schmidhuber, “Super Mario Evolution” in Proceedings of the 5 th international conference on Computational Intelligence and Games, ser. CIG’09. Piscataway, NJ, USA: IEEE Press, 2009, pp
Level Representation IMB: 2D array of Objects (brick blocks, coins, enemies…) Short levels 100 “blocks” wide app. 30 seconds to finish A set of “chunks”: platforms gaps tubes cannons boxes coins enemies
Level Representation More “terrain”: Obstruction Platforms Hills
GE-Based Level Generator Sample Level A sample generated level showing some of the grammar’s limitations
Implementation and Experimental Setup GEVA software – implement needed functionalities M. O’Neill, E. Hemberg, C. Gilligan, E. Bartley, J. McDermott, and A. Brabazon. “GEVA: -grammatical evolution in Java”, ACM SIGEVO-lution, vol. 3, no. 2, pp. 2, pp , Experimental parameters: 1000 runs for (generations): 10 population size (individuals): 100 Maximum derivation tree depth:100 Tournament selection size:2 int-flip mutation probability:0.1 (10%) one-point crossover probability:0.7 (70%)
Implementation and Experimental Setup
Notch Level Generator Incrementally places different chunks Difficulty: number of generated gaps, enemies and enemy types
Parametrized Level Generator
Expressivity Analysis 1000 levels 8 features 4 metrics: Linearity Density Leniency Compression Distance
Expressivity Analysis Linearity Variety of hills and platforms Normalized to [0, 1] Linearity = 0.99 Linearity = 0
Expressivity Analysis Density Density = 0 Density = 0.85 (Linearity = 0.4) Density = 1 (Linearity = 0.9)