Presentation on theme: "Alex van Poppelen, Boris Jakovljevic, and Nadia Boudewijn"— Presentation transcript:
1 Alex van Poppelen, Boris Jakovljevic, and Nadia Boudewijn Evolving Levels for Super Mario Bros Using Grammatical Evolution & Evolving a Ms. Pac-Man Controller Using Grammatical EvolutionPresentation byAlex van Poppelen, Boris Jakovljevic, and Nadia Boudewijn
2 The papersWritten by several people (1 person worked on both papers) from:Natural Computing Research & Applications Group, University College Dublin, Ireland&Center for Computer Games Research,IT University of Copenhagen, DenmarkFairly recent: 2010 (Ms. PacMan) / 2012 (SMB)
3 Contents Grammatical Evolution (Nadia) Using grammatical evolution to evolve a Ms. Pac-Man controller (Alex)Using grammatical evolution to evolve levels for Super Mario Bros (Boris)
4 Grammatical Evolution (GE) Relatively new concept (1998)Related to the idea of genetic programming (GP):find an executable program that will achieve a good fitness value for the given objective functionMain difference:GP: uses tree style structured expressions that are directly manipulatedGE: manipulates integer strings, that are subsequently mapped to a program through the use of a grammar
5 Grammatical Evolution Integer strings -> grammar -> programfitness(program) -> integer stringInspired by nature: separate genotype from phenotypeGenotype: integer stringPhenotype: tree-like structure that is evaluated recursively (same as GP)Benefit of GE’s modular approach: no specific algorithm or method is required to perform search operationsIt is possible to structure a GE grammar that for a given function/terminal set is equivalent to genetic programming.
6 What is a grammar?“Grammar” can apply to:Natural Language (Linguistics): a set of structural rules governing the composition of clauses, phrases and words in any given natural language.Formal Language (mathematics, logic ,and theoretical computer science): is a set of production rules for strings in a formal language.A grammar does not describe the meaning of the strings or what can be done with them in whatever context.
7 Only syntax – NO semantics 1. The ate bear the fish 2. The bear ate the fish 3. The fish ate the bear The grammar for the natural language English will accept sentences 2 & 3, but will reject sentence 1. What does it mean to say that a grammar accepts some string (or sentence)?
8 Example CFG and parse tree Language: anbnContext Free Grammar:S :- a S bS :- ЄParse tree for “aabb”:Є
9 Context-Free grammar? Equivalent to Backus-Naur Form CF-grammar: lexicon of words and symbols + production rulesTwo classes of symbols: terminal + non-terminalFormal language defined by a CFG: set of strings derivable from the start symbolCF-grammar use:A device for generating sentencesA device for assigning a structure to a given sentence
11 Expressive PowerFormal mechanisms (CFG’s, Markov Models, transducers etc.) can be described in terms of their power:= in terms of the complexity of the phenomena they can describeOne grammar has greater generative power or complexity than another if it can define a language the other cannot define.Chomsky Hierarchy: hierarchy of grammars, where the set of languages describable by grammars of greater power subsumes the set of languages describable by grammars of less power.
13 Back to GE Population: a set of integer strings Applying a mapping rule, these integer strings are converted into problem instances following the rules of the Context-Free Grammar involved
14 Criticism and Variants Due to the fact that GE uses a mapping operation, GE’s genetic operators do not achieve high locality:small changes in the genotype always result in small changes in the phenotypeThis is a highly regarded property of genetic operators in evolutionary algorithms.One possibility for variants is to use particle swarm optimization to carry out the search instead of genetic algorithms
15 Evolving a Ms. Pac-Man Controller using GE Deterministic?
16 Ms. Pac-Man Competition Aims to provide best software controller for the game of Ms. Pac-ManBest human player score: 921,360Best computer score: 30,010Hand-coded agentDeveloped by Matsumoto et al from Kyoto, JapanYear 2009
17 Using an Evolutionary Approach Previous approach by Koza:Used Genetic Programming to combine pre-defined actions and conditional statements to evolve a simple Ms. Pac-Man playerGoal: Achieve highest scoreFitness function: Points earned per gameUsed reinforcement learning and the cross-entropy method to assist agent in learning appropriate decisionsThis paper:Attempts to successfully evolve rules in the form of “if <condition> then perform <action>”Uses Grammatical Evolution
18 RepresentationGrammatical evolution represents programs as a variable length linear genomeGenome is an integer array of elements called codonsGenotype mapped to phenotype using grammar in Backus-Naur FormMapping function:Rule = c mod rc is the codon integer valuer is the number of choices for the current symbolCodons may remain unused, or there may not be enough. In the latter case, may wrap back to the beginning up to a maximum number of times
21 Experimental Setup One level, one life Fitness function: Add scores for each pill, power pill, and ghost eatenGeneration approachPopulation size 100Ramped half and half initialization method (max tree depth 10)Tournament selection size 2Int flip mutation (probability 0.1)One-point crossover (probability 0.7)Maximum of 3 wraps allowed to “fix”invalid individuals
22 Best Evolved Controller Very aggressiveHeads for power pills and then tries to eat all edible ghosts without looking to see if there are inedible ghosts in the way
23 Benchmarking Performance Compared evolved agent to 4 other agentsHand-coded agentRandom agent (chooses up, down, left, right, or neutral at every time step)Random Non-Reverse agent (same as random, but no back-tracking)Simple Pill Eater (heads for nearest pill, ignores all else)
24 Different Ghost TeamsThree different ghost teams were used to test the agentsRandom team (Each ghost chooses a random direction each time step, no back-tracking)Legacy team (Three ghosts use different distance metrics: Manhattan, Euclidean, and shortest path distance. Last ghost makes random moves)Pincer team (Each ghost attempts to pick the closest junction to Ms. Pac-Man within a certain distance in order to trap her)
26 ConclusionsEvolved controller beat their own hand-coded controller against all ghost teamsEvolved controller did not match of exceed the score of Matsumoto’s hand-coded agentBut: Matsumoto’s agent was given three lives, could earn more lives, and had more than one level to playOur question: Why didn’t they evolve their controller under the same circumstances?
27 Evolving Levels for Super Mario Bros Using Grammatical Evolution Boris Jakovljevic
28 The paper Authors: Noor Shaker Miguel Nicolau Georgios N. Yannakakis (Member, IEEE)Julian Togelius (Member, IEEE)Michael O’Neill
29 Table of Contents Introduction Background Testbed Platform Game Level RepresentationGE-based Level GeneratorOther GeneratorsExpressivity AnalysisConclusions and Future Work
30 IntroductionGame Development → Time and Money → Automatic Game Content GenerationCompare different techniques of content generationLarge amount of content → Automatic EvaluationGenetic Programming → Grammatical EvolutionGreater control of outputGeneralization to different types of gamesFramework → generators’ Expressivity Range analysis and comparison
31 Background Grammatical Evolution Framework → generators’ Expressivity Range analysis and comparisonSuggested by G. Smith and J. Whitehead in ‘Analyzing the expressive range of a level generator’ from “Proceedings of the 2010 Workshop on Procedural Content Generation in Games”; ACM, 2010, p. 4Defined Description MetricsVisualize the Generative SpaceAnalyze the (Parameters 𝑖𝑚𝑝𝑎𝑐𝑡 Expressivity) relation of Level Generators
32 Background Framework extended through: more informative aesthetic measures of generators’ expressivityapplying the above measures to analyze and compare expressivity ranges of 3 level generators
33 Testbed Platform GameA modified version of Markus “Notch” Persson’s Infinite Mario Bros (IMB)*.Super Mario Bros – a very rich Environment Representation.J. Togelius, S. Karakovskiy, J. Koutnik, and J. Schmidhuber, “Super Mario Evolution” in Proceedings of the 5th international conference on Computational Intelligence and Games, ser. CIG’09. Piscataway, NJ, USA: IEEE Press, 2009, pp
34 Level Representation IMB: 2D array of Objects (brick blocks, coins, enemies…)Short levels100 “blocks” wideapp. 30 seconds to finishA set of “chunks”:platformsgapstubescannonsboxescoinsenemies
36 GE-Based Level Generator Design Grammar Ease of Interaction for designersChunk positioning regardless of other chunksProblems?Chunk properties:x and y → starting point coordinates in a 2D Level Array → [ ], [3 … 5]𝑤 𝑔 → gap width𝑤 𝑏 → number of boxes𝑤 𝑒 → number of enemies𝑤 𝑐 → number of coinsℎ → height of flower tubes and cannons
38 GE-Based Level Generator Design Grammar First version of the grammarLimitationsGame experience𝑤 𝑏𝑒𝑓𝑜𝑟𝑒 → width of the platform before the chunk𝑤 𝑎𝑓𝑡𝑒𝑟 → width of the platform after the chunkPlacement of enemiesFirst version of grammar → in groups (always on platforms)<chunks> ::= <chunk> |<chunk> <chunks> <chunk> ::= gap(<x>,<y>,<wg>) | platform(<x>,<y>,<w>) | hill(<x>,<y>,<w>) | cannon_hill(<x>,<y>,<h>) | tube_hill(<x>,<y>,<h>) | coin(<x>,<y>,<wc>) | cannon(<x>,<y>,<h>) | tube(<x>,<y>,<h>) | boxes(<x>,<y>,<wb>) | enemy(<x>,<y>,<we>) <x> :: = [5..95] <y> ::= [3..5] <wg> ::= [2..5] <w> ::= [3..15] <h> ::= [2..3] <wb> ::= [2..7] <we> ::= [1..7]
39 GE-Based Level Generator Design Grammar First version of the grammarLimitationsMore variabilityMore enemy types / any platform with:constructing the physical structure of the levelcalculating the possible spawn positionsspawning enemies on one of possible positionsEnemy’s position → parameter in grammar (maintain deterministic genotype → phenotype mapping)<chunks> ::= <chunk> |<chunk> <chunks> <chunk> ::= gap(<x>,<y>,<wg>) | platform(<x>,<y>,<w>) | hill(<x>,<y>,<w>) | cannon_hill(<x>,<y>,<h>) | tube_hill(<x>,<y>,<h>) | coin(<x>,<y>,<wc>) | cannon(<x>,<y>,<h>) | tube(<x>,<y>,<h>) | boxes(<x>,<y>,<wb>) | enemy(<x>,<y>,<we>) <x> :: = [5..95] <y> ::= [3..5] <wg> ::= [2..5] <w> ::= [3..15] <h> ::= [2..3] <wb> ::= [2..7] <we> ::= [1..7]
40 GE-Based Level Generator Design Grammar <level> ::= <chunks> <enemy><chunks> ::= <chunk> |<chunk> <chunks><chunk> ::= gap(<x>,<y>, <wg>,<wbefore>,<wafter>)| platform(<x>,<y>,<w>)| hill(<x>,<y>,<w>)| cannon_hill(<x>,<y>,<h>,<wbefore>,<wafter>)| tube_hill(<x>,<y>,<h>,<wbefore>,<wafter>)| coin(<x>,<y>,<wc>)| cannon(<x>,<y>,<h>,<wbefore>,<wafter>)| tube(<x>,<y>,<h>,<wbefore>,<wafter>)| <boxes><boxes> ::= <box_type> (<x>,<y>) 2 | ...| <box_type> (<x>,<y>) 6<box_type> ::= blockcoin | blockpowerup| rockcoin | rockempty<enemy> ::= (koopa | goompa)(<x>) 2 | ...| (koopa | goompa)(<x>) 10<x> :: = [5..95] <y> ::= [3..5]Second version of the grammarA simplified version of the final grammar for level design specification;Superscripts → repetitions
41 GE-Based Level Generator Conflict Resolution x and y → any value from [ ], [3 … 5]High % of overlappingExample: 𝐡𝐢𝐥𝐥 𝟔𝟓, 𝟒, 𝟓 hill 25, 4, 4 𝐜𝐚𝐧𝐧𝐨𝐧_𝐡𝐢𝐥𝐥 𝟔𝟕, 𝟒, 𝟒, 𝟒, 𝟑 coin 22, 4, 6 platform 61, 4, 5Resolution:priority value → chunk
42 GE-Based Level Generator Sample Level A sample generated level showing some of thegrammar’s limitations
43 Implementation and Experimental Setup GEVA software – implement needed functionalitiesM. O’Neill, E. Hemberg, C. Gilligan, E. Bartley, J. McDermott, and A. Brabazon. “GEVA: -grammatical evolution in Java”, ACM SIGEVO-lution, vol. 3, no. 2, pp. 2, pp , 2008.Experimental parameters:1000 runsfor (generations): 10population size (individuals): 100Maximum derivation tree depth: 100Tournament selection size: 2int-flip mutation probability: 0.1 (10%)one-point crossover probability: 0.7 (70%)
44 Implementation and Experimental Setup Fitness function’s objective: levels with acceptable number of chunksFitness function → weighted sum of 2 normalized measures𝑓 𝑝 → maximum # of chunks – current # of chunks𝑓 𝑐 → # of conflicting chunks foundConflict:↑ 𝑓 𝑝 ⇒ ↑(𝑓 𝑐 )%
45 Notch Level Generator Incrementally places different chunks Difficulty:number of generated gaps, enemies and enemy types
46 Parametrized Level Generator Based on Notch’s level generator6 features:𝐺 → # of gaps in the level𝐺 𝑤 → average gap width𝐸 → # of enemies𝐸 𝑝 → placement of enemies𝑃 𝑥 → above or under horizontal blocks𝑃 𝑔 → close to gap edge 100%𝑃 𝑟 → random placement on ground𝑁 𝑤 → # of powerups𝐵 → # of boxes# of cannons and flower tubes → random
56 Expressivity Analysis Statistical Analysis All levels across all generatorslinearity + leniencylinearity − densityleniency − density
57 Conclusion and Future Work Potential use for game designersGenerator comparison within the same genreFuture work:incorporate player experiencedesign grammar personalizationmore detailed expressivity measureLimitation:GE unable to generate high density levels → constraint-free grammar + play-test