6/4/03Genetic Algorithm The Genetic Algorithm The Research of Robert Axelrod The Algorithm of John Holland Reviewed by Eyal Allweil and Ami Blonder

6/4/03Genetic Algorithm The Prisoner’s Dilemma CooperateDefect Player 1 Player 2 0 S(ucker) 5 1 P(unishment) 1 Defect: 3 R(eward) 3 5 T(emptation) 0 Cooperate:

6/4/03Genetic Algorithm The Iterated Prisoner’s Dilemma (IPD) Two tournaments held by Robert Axelrod The highest average score was Anatol Rapoport’s Tit-For-Tat (TFT) Eight representatives explain 98% of the variance in the tournament results. We will also hear criticism of these claims. ( Ken Binmore)

6/4/03Genetic Algorithm Introduction: Axelrod’s Motivation Axelrod wanted to prove a point: The success of Tit-For-Tat in his Iterated Prisoner’s Dilemma was not based on the preconceptions of those who submitted entries. The hammer and the nail.

6/4/03Genetic Algorithm More practical motivations We have a group aim, and we want agents that can fulfill it. More accurately, we want our agents to be as efficient as possible in achieving this goal. Example: The Mars element-gathering, subsumption architecture experiment

6/4/03Genetic Algorithm The Genetic Algorithm 1.Construct an initial population 2.Test and score population 3.Calculate number of offspring 4.Reproduction: Choose parental pairs 5.Reproduction: Crossover / Mutation

6/4/03Genetic Algorithm Encoding a Strategy (1) 6 ……… ……… Axelrod encoded strategies as sequences of 70 bits. How does this work? Note: 0 = C, 1 = D

6/4/03Genetic Algorithm Encoding a Strategy (2) What do I do now? Move -1 / me Move -1 / him Move -2 / me Move -2 / him Move -3 / me Move -3 / him 0 = C ………………… 1 = D111111

6/4/03Genetic Algorithm 1. Constructing an Initial Population Two possibilities for creating an initial population: 1.Initial strategies can be assembled (as in the first two tournaments) or - 2. They can be randomly created

6/4/03Genetic Algorithm Constructing an Initial Population (cont’d) Axelrod used normalized population of 20 A rule of thumb is that the product of the number in the population and the number of generations should exceed 100,000 In addition, the number of individuals in the population must considerably exceed the number of genes in each individual's chromosome.

6/4/03Genetic Algorithm 2. Testing the Population In the Iterated Prisoner’s Dilemma, the outcome of each interaction is added to produce a player’s score for that generation. In general, an ordering function is needed. This function must be efficient!

6/4/03Genetic Algorithm 3. The Number of Offspring (in Axelrod’s simulation) Strategies that were one standard deviation above the average score produced two matings. Strategies that were one standard deviation below the average were barren- no offspring. Other strategies produced a single mating Each generation is disjoint from its predecessor

6/4/03Genetic Algorithm The Number of Offspring (cont’d) There are other possibilities available when calculating offspring: Normalization / growth of population size Preservation of arbitrary amount of previous generation’s strategies (not done in Axelrod model)

6/4/03Genetic Algorithm 4. Choosing Parents In Axelrod’s simulations, pairs were chosen randomly to mate and produce, each, two offspring. Other possibilities exist: Mating by excellence (short term exploitation) Mating by geographic proximity

6/4/03Genetic Algorithm 5. Reproduction: Crossover “Crossover selects one or more places to break the parents’ chromosomes in order to construct two offspring each of whom has some genetic material from both parents.” Other forms of crossover are possible. Syntactic integrity must be preserved! What are the advantages of high/low crossover? (more on this later)

6/4/03Genetic Algorithm 5. Reproduction: Mutations In every offspring born, there is a small chance of bit reversal- a change in strategy. Don’t forget syntactic integrity! What are the advantages of high/low mutation rates?

6/4/03Genetic Algorithm Technical Details Population size: 20 Round length: 151 meetings Each population member met one of 8 “representatives” (not each other) Therefore meetings per generation Number of generations: parent (sexual) experiments, 40 asexual ones

6/4/03Genetic Algorithm Conclusions (1) : Effectiveness “the problem for evolution can be conceptualized as a search for relatively high points in a multidimensional field of gene combinations, where height corresponds to fitness.” Axelrod: TFT-like strategies are the big winner! The genetic algorithm produces results better than the second Axelrod tournament: 450 weighted score vs. 428 for TFT But these results were in 11 (out of 40) experiments which resulted in a strategy which tried to exploit “sucker” strategies! (more later)

6/4/03Genetic Algorithm Successful Alleles: TFT-like behavior 1.Don’t rock the boat ( C after RRR ) 2.Be provocable ( D after RRS ) 3.Accept apologies: ( C after TSR ) 4.Forget: ( C after SRR ) 5.Accept a rut: ( D after PPP )

6/4/03Genetic Algorithm Conclusions (2) : Sexual Reproduction In biology, sexual reproduction carries a stiff price – useless males Computationally, sexual reproduction is cheap. Asexual runs of IPD resulted in lower average scores (5 out of 11 had higher median scores than TFT) Parasite theory. (only if we have time)

6/4/03Genetic Algorithm Conclusions (3) : Arbitrariness Hitch-hiking genes- typically present, not typically employed. (TFT example – PPR) Premises and their results ( TFT example – the original six bit premise ) Two basin theory ( Ken Binmore ) : By choosing the correct premises, we can produce different (stable) evolutionary end-products

6/4/03Genetic Algorithm Conclusions (4) : Tradeoffs The trade off that exists is between flexibility and specialization. This can be translated into short and long term gains (exploitation vs. exploration) Varying mutation based on how dynamic the environment is. Invasion

6/4/03Genetic Algorithm Conclusions (5) : Irreversibility The possibility exists of getting stuck in a local maximum. This results from adaptation to a set of premises, which strengthens them in the future. There are those who claim that TFT is such a local maximum.

6/4/03Genetic Algorithm Conclusions and Criticism Kristain Lindgren (1991) – found cyclical history of stability / instability Lombard (1996) : more extensive simulations, copy-and-innovate instead of standard genetic algorithm Lombard: Noise is a crucial factor

6/4/03Genetic Algorithm More Criticism and Conclusions Binmore - TAT-FOR-TIT is a better strategy? Were more experiments called for? Probst (1996) found exploitive machines thriving after running longer experiments! His criticism is against the simulation results, not against the use of the algorithm in general. But this is a lesson that applies only in pairwise interactions. In multi-person interactions, it need not be the injured party who punishes a cheater.

6/4/03Genetic Algorithm References Axelrod, Robert. The Complexity of Cooperation Online: Review of The Complexity of Cooperation (Ken Binmore) Online: Simulation for the Social Scientist (Nigel Gilbert and Klaus G. Troitzsch)