1. Stabilization of Tag-Mediated Interaction by Sexual Reproduction in an Evolutionary Agent System F. Alkemade D.D.B. van Braget J.A. La Poutr é Copyright 2001 © AI-ECON Research Center

2. Outline 1. Introduction Introduction 2. Problem Description Problem Description 3. Experimental Setup Experimental Setup 4. Results and Discussion Results and Discussion 5. Tagging in the N-person IPD game Tagging in the N-person IPD game 6. Conclusions Conclusions 7. Discussion Discussion

3. Introduction Motivation Previous Studies Our Work

4. Motivation The “ Spontaneous ” Emergence of Cooperation in Large System of Selfish Agents. Visible Tags Might Be Useful in Promoting the Evolution of Cooperation? Sexual Reproduction (Recombination)

5. Nowak and Sigmund (1998) images Investigate the role of “ images ” in the emergence of indirect reciprocity. Similar to tags, images are parts of an agent that are visible to the other agents. However, the images have a fixed meaning and truthfully reveal information, while tags do not have an explicit and predetermined meaning.

6. Axtell, Epstein and Young (2000) dividing-the-dollar Investigate a simplified “ dividing-the-dollar ” game where agents adjust their strategy on the basis of the tag of their opponent. They investigate a Markov-chain model of two fixed- size tag groups where the agents can condition their strategy upon the tag of their opponent. This model does not allow for choice and refusal of partners and the emergence of tags.

7. Stanley, Ashlock and Tesfatsion (1994) They also study and IPD game with selective interaction. In their model however, partner selection is based on a continuously updated expected payoff function and not on tags (as in this paper ’ s model).

8. Arifovic and Eaton (1995) They focuses on the use of tags as type- signals.

9. Riolo (1997) added a simple tagging mechanism were able to reach mutual cooperation He studied a multi-agent system, modeled by a simple evolutionary algorithm (EA) in which agents play a short iterated prisoner ’ s dilemma (IPD) game (of only four rounds) against each other. Under these conditions, it is very difficult for cooperation to emerge. When he added a simple tagging mechanism, however, population dynamics changed dramatically and the agents were able to reach mutual cooperation earlier and over extended periods of time. However results indicate that the evolving tag-using populations are still relatively unstable.

10. Our Work Riolo ’ s work will be extended. Investigating the stability of the evolving population. In Riolo, he used a very simple EA in his experiment. In partidular, the reproduction of the agents was modeled as an asexual process (mutation only). Earlier experiments by Axelrod(1984,1987) have demonstrated, for instance, that cooperative societies form more frequently if recombination of the agents ’ strategies occurs.

11. Sexual Reproduction Recombination of strategies is also called sexual reproduction.

12. Problem Description Iterated Prisoner ’ s Dilemma Game (IPD) TaggingRecombination The two player and the N-player game The emergence of cooperation

13. IPD Game (2-person) Player i Cooperate Defect CooperateDefect Cooperate Defect Player 2 Player 1 R = 3 T = 5 S = 0 T = 5 P = 1 R: Reward P: Punishment T: Temptation S: Suckers ?

14. The Agents ’ Tag Tag = “011” Tag = “011” When two agents meet and their tags are Sufficiently similar they will play an IPD game We subsequently distinguish between naive and Discriminating cooperative agents.

15. Experimental Setup A Multi-Agent Software Platform Three Different Recombination Operators Fitness Evaluation The Representation of the Agents ’ Strategies An Overview of the Model Settings

16. A Multi-Agent Software Platform: The Swarm Model main Model Swarm popList Observer Swarm IPD ControlPanel newList winner EZGraph Start Stop Next

17. Three Different Recombination Operators Single-Point Crossover Two-Point Crossover Uniform Crossover

18. Fitness Evaluation Relative performance of the agents is normalized by taking where is the mean population fitness (with standard deviation ). This implies that a player performing one standard deviation above the mean will (on average) get two offspring. Negative and very low fitness values ( <0.1) were reset to 0.1.

19. The Representation of the Agents ’ Strategies Pure (deterministic) Strategies: Cooperate is coded by 1 and Defect by 0. Each Agent Has Three Bits Memory Capacity (only four rounds)Memory Capacity Three Tag Bits Mate Selection Tag 1 1 1 1 0 1 0 1 0 0 0 0 0 1 1 1 1 Initial Memory Block Tag Mate Selection Tag Strategy

20. Memory Capacity Memory Capacity of three previous moves (one move of his own, and two of the opponent). There are 2 3 = 8 possible histories and each history uniquely points to an action on the strategy part of the chromosome [s 0, …,s 7 ].

21. Tag-Based Opponent Selection This algorithm will be repeated until ten opponents have been selected from the population of 400 individuals. An opponent search cost (of 0.02) is associated with each tag trial. When the allowed number of tag search trials is exceeded, the player is matched with a random opponent. After an opponent has been selected, the IPD game is played and the opponent search costs are subtracted from the player ’ s payoff. From Population (400) Compare Tags? No match Search Cost Select Random opponent # of Opponent <max? No Calculate Payoff Play IPD

22. Hamming Distance (tag bias) Example: 101 and 010: Hamming Distance=3 111 and 101: Hamming Distance=1 110 and 110: Hamming Distance=0

23. Mate Selection Tag After Calculate Payoff Compare Mate Tags? No match Generates Parents (Proportional Selection) # of Co-parent <max? No OffspringCrossover Yes

24. An Overview of the Model Settings (Table 1)

25. Results and Discussion AsexualAsexual vs. Sexual Reproduction Sexual Tag-Directed Mate Selection

26. Asexual Reproduction A typical run A typical run of a population of asexual agents playing the IPD using the settings described in Table 1. The oscillatory pattern of the mean population fitness indicates that the population alternates between a state of mutual cooperation and a society of defectors. Societies of cooperators are frequently undermined by “ mimics ” : defecting agents with a tag associated with a group of cooperators. These mimics are not recognized as being defectors and can therefore successfully exploit the naive cooperating agents.

27. A Typical Run (Figure 4) Mean fitness of a population of asexual agents playing the IPD.

28. Sexual Reproduction significant change We obtain a significant change in population dynamics. runs in which a high mean fitness level is achieved and sustainedruns in which a society of defectors forms The oscillatory behavior visible in Fig. 4 disappears, and the individual runs can now be classified as (1) runs in which a high mean fitness level is achieved and sustained, and (2) runs in which a society of defectors forms. Once cooperation emerges it persists over long periods of time.

29. Two Typical Runs (Figure 5) Mean fitness of a population of sexual agents playing the IPD: two typical runs. G to 2.3 G over 2.3 G under 1.7

30. Average Result (over 30 runs) Table 2: Average fitness performance for asexual experiments and sexual experiments. (Numbers are calculated for 30 runs of 1,000 generations; standard deviations in brackets.

31. Explanation: Sexual Reproduction In case of sexual reproduction, the mean fitness over the entire evolution history (MHF) remains rather low. Why? This is due to the fact that it takes longer for cooperative societies to emerge when recombination is used.

32. Explanation: Two-Point Crossover Notice for instance in Table 2 that, when two- point crossover is used, the average number of generations it takes before the fitness first exceeds a value of 2.3, G to 2.3, increases to 7.7*10 2 in the experiments with sexual reproduction. But once a population has exceeded this fitness level, the population fitness never drops below 1.7 (G under 1.7=0) and in fact always remains above 2.3 (G over 2.3=1).

33. Explanation: One-Point Crossover When one-point crossover or uniform crossover was used instead of two point crossover, the same stabilizing effect was observed. However, cooperation was achieved more often using the less disruptive single-point crossover operator. Populations using single-point crossover evolved to cooperative societies in 24 out of 30 runs compared with only 8 out of 30 runs for two-point crossover.

34. G over 2.3, G to 2.3, G under 1.7 G over 2.3 is defined as the fraction of generations in which the mean fitness is above 2.3, counting only generations after G to 2.3. G to 2.3 is the number of generations it takes before the fitness first exceeds a value of 2.3. Analogously, G under 1.7 counts the fraction of generations in which the mean fitness has dropped below 1.7.

35. Tag-Directed Mate Selection 1.68 (0.5) 7.2·10 2 (N/A) Tag-Directed Note: Mate SelectionMate Selection

36. Experiments with an Evolving Tag Bias for Mate Selection We also performed experiments with an evolving tag bias For mate selection. In this setup, a mate was only accepted If the Hamming distance between the two mating tags was Equal to the tag bias. We found, in general, that agents have a strong preference for partners with a similar mating tag, While the average fitness remained the same as in the Experiments with a fixed bias for mate selection.

37. Tagging in the N-person IPD Game (Social Dilemmas) Fundamental Problem Tagging Mechanism Payoff Matrix for an Agent Payoff Matrix for an Agent (4-personIPD) The Strategy of an Agent Mutation Probability is Reduced Computational Results for N=4 Results of More Players The role of Tag

38. Fundamental Problem The two-person IPD game can be used to model many social processes where cooperation is desirable but not easily obtained or sustained. Previous computer simulations of the NIPD with evolutionary algorithms have shown that it becomes substantially more difficult to evolve cooperative societies if the number of players increases.

39. Tagging Mechanism We extend our research on tagging to the NIPD game. To investigate whether the tagging mechanism also fosters stable cooperation in the NIPD, we performed a series of experiments for N>2.

40. Payoff Matrix for an Agent If an agent cooperates, his payoff is equal to 2n c -2 If he defects, he earns a payoff of 2n c +1 Where N c is the total number of cooperating agents.

41. The Strategy of an Agent The agent ’ s previous 3 moves. (3 bits) The number of cooperating agents in previous 3 rounds. (3×2 = 6 bits) As in the 2-person IPD, the length of the tag is equal to the size of the initial memory block. Without a mating tag, the total chromosome length for agents in the 4-person IPD is there equal to 530 bits. Initial Memory Block Tag Strategy

42. Mutation Probability is Reduced In the evolutionary algorithm, the mutation probability (per bit) is reduced to 0.002 (from 0.025, see Table 1) to avoid an excessive increase of the number of mutations due to the much longer chromosome length for N=4. Remaining parameters were kept the same as in the 2-person IPD.

43. Computational Results for N=4 Table 4: Influence of tagging and sexual reproduction in the 4-person IPD. (Statistics are calculated for 30 runs of 10,000 generations; standard deviations in brackets.)

44. Influence of tagging and asexual reproduction Figure 6: In the 4-person IPD, one typical run with cooperation. The horizontal lines indicate the fitness of populations with on (average) 1,2,3 or 4 cooperators.

45. Figure 6: Difficult to Achieve Cooperative Notice that it is very difficult to achieve cooperative societies (the MHF remains low). Although the tagging mechanism increases average fitness levels, population-wide cooperation does not emerge. Remember that the average population fitness would still be equal to 2.25 if there is only one cooperator in each round. Note from Table 4, the number of RCP and RSSC significantly increase in case of tag- using agents.

46. Figure 6: Tag Groups Exhibit Defective Behavior To gain more insight in the nature of the cooperation that occurs, we examined the number of cooperators per tag group and in each round of the game. As in the 2-person IPD, distinct tag groups emerge after approximately 100 generations. Most of these tag groups exhibit defective behavior. Sometimes a tag group discovers cooperative strategies.

47. Figure 6: The Maximum Fitness In most runs, this “ cooperating ” group was of a substantial size, periodically increasing average fitness levels to 3 or even higher. The maximum fitness measured during the experiments was approximately equal to 5, which indicates the emergence of a large group of cooperators (also given the fact given that the agents have to pay tag search costs).

48. Influence of tagging and sexual reproduction Figure 7: In the 4-person IPD. Two typical runs with cooperation. Notice that, once some cooperation is achieved, the population stays out of the defective zone throughout the entire run.

49. Figure 7: Careful Analysis The MHF data presented in Table 4 suggests that sexual cooperation does not help the emergence of cooperation. A more careful analysis however shows that the higher mean fitness in the asexual experiments is due to the fact that in these experiments incidentally very high fitness is achieved. This cooperation level cannot be sustained however.

50. Figure 7: Slow Convergence The experiments further show that (due to slow convergence), fitness levels in the runs without sexual reproduction decrease very slowly, but once all cooperation is completely lost, it is very difficult for the asexual agents to reestablish it (at least not within the 10,000 generations we have examined). Figure 7 show that this is not the case for experiments with sexual agents: after an initial period of low fitness levels a transient towards an increased level of cooperation occurs (after approximately 600 generations).

51. Figure 7: Sustained in the Remainder of the Experiment This increased level of cooperation is then sustained in the remainder of the experiment.

52. 8 or 16 players Only defective societies were observed, with fitness levels always lower than 1.5. small number of rounds We found in additional experiments that this difficulty in achieving cooperation was was caused mainly by the small number of rounds in the game. When the game length increases, average fitness levels rise, and cooperation is achieved more often. As an example, Table 5 shows the results for the 8-person IPD when the number of interactions is increase to 10.Table 5

53. Experimental results for the 8- person IPD Table 5: When the number of Interactions per game is increased to 10. (Statistics are calculated for 10 runs of 10,000 generations; standard deviations in brackets.)

54. The Role of Tag We see that tags help to establish cooperation in societies of agents playing the NIPD. In the 16-person IPD, increasing the number of iteration to 10 cases a small increase of the level of cooperation. In 10 runs of 10,000 generations each, cooperation (i.e., fitness > 1) emerged only once with tag-using agents. These results are roughly compatible with experiments from Yao and Darwen(1994). The failure to reach cooperation can be caused by the large search space (the chromosome length without tags is 32,783 in the 16-person IPD without tags).

55. Conclusions We have investigated the “ evolution of cooperation ” in a population of agents playing the tag-mediated IPD. We have shown that the tagging mechanism and the reproduction process of the agents play a major role in the formation of stable cooperative societies.

56. Conclusions state of mutual cooperation society of defectors In the 2-person IPD, the population alternates between a state of mutual cooperation and a society of defectors in a model with asexual reproduction. A distinct behavior emerges if reproduction of the agents is sexual. We observed, for instance, the formation of very stable societies of cooperative agents, a phenomenon not observed in the experiments with asexual reproduction. not too disruptive Furthermore, we found that cooperative societies emerge more frequently when the recombination operator is not too disruptive (e.g., a single-point crossover scheme).

57. Conclusions Finally, we proposed a tagging mechanism to enable biased partner selection. Results for the N-person IPD showed It becomes more difficult to evolve cooperative societies if the number of player is increased. Tagging does help to achieve cooperation in the N-person IPD game.

58. Conclusions Furthermore, stable long-term cooperation emerges more frequently when sexual recombination of the agents ’ strategies occurs. This is due to the fact that recombination prohibits the emergence of large numbers of naive cooperative strategies and thus reduces the impact and success of mimics. robust strategies Strategic information exchange thus helps to build robust strategies, while tagging creates favorable circumstances for cooperation.

59. Discussion How many dimensions have discussed in this paper?dimensions Reconsider the meaning of tagging mechanism in Economics and our life.the meaning What does recombination mean?recombination Does the recombination scale relate to the number of player per game?recombination scale

60. Dimension Asexual(-) vs. Sexual(+) Tagging Mechanism(+) Mate Tagging Mechanism(?) Game Length(+)

61. The meaning of tagging mechanism Self Identification. Grouping? This is the difference between animal and human. About two thousands years ago, Jesus began to preach, “ Repent, for the kingdom of heaven is near. ” (Matthew 4: 17)

62. Recombination Blessed are those who are persecuted because of righteousness, for theirs is the kingdom of heaven. (Matthew 5:10) You are the light of the world. A city on a hill cannot be hidden. (Matthew 5:14)