1. Evolution: Games, dynamics and algorithms
Karen Page
Bioinformatics Unit
Dept. of Computer Science, UCL

2. Evolution
Darwinian evolution is based on three fundamental principles: reproduction, mutation and selection
Concepts like fitness and natural selection are best defined in terms of mathematical equations
We show how many of the existing frameworks for the mathematical description of evolution may be derived from a single unifying framework

3. Summary of what will be discussed
Games, evolutionary game theory
Key frameworks of evolutionary dynamics
Deriving a unifying framework
An application to Fisher’s Fundamental Theorem
Relationship with Genetic Algorithms

4. What is game theory?
Formal way to analyse interactions between agents who behave strategically
Mathematics of decision making in conflict situations
Usual to assume players are “rational”
Widely applied to the study of economics, warfare, politics, animal behaviour, sociology, business, ecology and evolutionary biology
The objective of each player is to maximise his/her score in the game. Maximising one players score may well not be consitent with maximsing anothers.
Classical game theory looks at how a perfectly “rational” players will play the game. Rational means that the players try to maximize their scores in the game.

5. Assumptions of game theory
The game consists of an interaction between two or more players
Each player can decide between two or more well-defined strategies
For each set of specified choices, each player gets a given score (payoff)
Expand on specified choices: player 1 plays strategy a, player 2 plays strategy b etc.

6. The Prisoners’ Dilemma
Probably most studied of all games
Not enough evidence to convict two suspects of armed robbery, enough for theft of getaway car
Both confess (4 years each), both stay quiet (2 years each), one tells (0 years) the other doesn’t (5 years)
Stay quiet= cooperate (C) ; confess = defect (D)
Payoff to player 1:
Police have arrested two people whom they know have a committed an armed robbery, but they don’t have enough evidence for a jury to convict them. They do however, have evidence that the people stole a getaway car. They offer the suspects a deal. If neither of them confesses to the armed robbery they both get convicted of the theft of the car and do 2 years. If both of them confesses then they both share the blame do 4 years for armed robbery. If one confesses and the other doesn’t then the tell-tale gets set free and his mate does 5 years for armed robbery. So what should a prisoner do?
R is REWARD for mutual cooperation =3
S SUCKER’s payoff =0
T TEMPTATION to defect =5
P PUNISHMENT for mutual defection=1
with T>R>P>S

7. The problem of cooperation
What ever player 2 does, player 1
does better by defecting:
Classical game theory both players D
Shame because they’d do better by both cooperating
Cooperation is a very general problem in biology
Everyone benefits from being in cooperative group, but each can do better by exploiting cooperative efforts of others

8. Trade wars and cartels Import tariffs - Should countries remove them?
Price fixing- why not cheat?
As mentionned at the beginning, game theory is applied to many disciplines. The prisoners dilemma occurs in many walks of life. Here I mention just a couple of examples.

9. Repeated games
In many situations, typically players interact repeatedly- repeated Prisoners Dilemma
Strategies can involve memory, use reciprocity
Tit-for-tat
Pavlov
What one player does in round 4 of the game can depend on what has happened in the previous three rounds.

10. Game theory and a computer tournament
Game theory says it is rational to defect in single game or fixed number of rounds
Axelrod’s tournament- double victory for Tit-for-Tat
First point is a repeat of what was said in “The problem of cooperation”
Backwards induction

11. Evolutionary Game Theory
So how can cooperation be explained?

12. Evolutionary games John Maynard Smith- evolution of animal behaviour
Behaviour shaped by trial and error- adaptation through natural selection or individual learning
Players no longer have to be ‘rational’: follow instincts, procedures, habits rather than computing best strategy.
Games played in a population. Scores are summed. Strategies which do well against the population on average propagate.
Phenotypic approach to evolution
Frequency-dependent selection
At one time it was thought that rational behaviour would prove optimal against “irrational” behaviour. This turned out not to be the case.
Ecology: A scarcity of prey will cause predators to starve and their numbers to decline. Having fewer predators around will favour the growth of the population of prey which in turn will allow more predators to survive. But now higher predator numbers will cause the number of prey to decrease, bringing us full circle. Hence in ecological systems, numbers of predators and prey can oscillate in time, because of the effects that the levels of each have on the other.

13. Simple evolutionary game simulations
Everyone starts with a random strategy
Everyone population plays game against everyone else
The payoffs are added up
The total payoff determines the number of offspring (Selection)
Offspring inherit approximately the strategy of their parents (Mutation)
[Note similarity to genetic algorithms.]
[Nash equilibrium in a population setting- no other strategy can invade]
Talk about darwin’s theory of evolution- the frequencies of genes and increase over time if they are associated with features which lead to the production of more offspring. So the proportion of the given feature within the population will increase over time, eg. finches with a gene for sharp beaks in an environment where such a peak is very important for accessing food.
If the feature is behavioural, such as for instance the propensity to back down in a conflict (cf. hawk-dove game) then whether the gene is selected for depends on the make up of the population. Evolutionary game theory models this kind off evolutionary process. It can be used to show that the proportion of hawks in a population of hawks and doves will tend to fitness gain for winning territory/ fitness loss for getting injured (in our example 1/5). Here it implies that heavily in armed species, such as stags, which can potentially inflict mortal wounds on one another, very few individuals will escalate a conflict. Paradoxically in species of doves who under normal circumstances can’t do each other much damage, escalation is much more likely. Indeed when confined to small cages doves will often peck each other to death.

14. Evolution in the Prisoners’ Dilemma
Standard evolutionary game (random interactions)  all Defect
Modifications- spatial games: Interactions no longer random, but with spatial neighbours:
Sum scores. Player with highest score of 9 shaded takes square (territory, food, mates) in next generation
Some degree of cooperation evolves!
Mention also proportional selection

15. Simulations of the spatial Prisoners Dilemma
75 generations
Winner-takes-all selection
No mutation
Red=d(d last) Blue=c(c last) Yellow=d(c last) Green=c(d last)

16. Conclusions on Evolutionary Games
Game theory can be applied to studying animal and human behaviour (economics - evolutionary biology).
Often traditional game theory’s assumption of ‘rationality’ fails to describe human/ animal behaviour
Instead of working out the optimal strategy, assume that strategies are shaped by trial and error by a process of natural selection or learning. This can be modelled by evolutionary game theory.
Space can matter

17. Evolutionary dynamics

18. General framework Quasispecies equation Replicator-mutator equation
Price equation
Price
equation
Lotka-Volterra
equation
Game dynamical
equation
replicator
Price equation
Adaptive dynamics

19. The replicator equation
Replicator equation describes evolution of frequencies of phenotypes within a population with fitness-proportionate selection
Eg. game theory, replicators like “Game of Life”
Frequency of type i is and fitness of type i is then

20. The equivalence with Lotka Volterra equations
Lotka Volterra systems of ecology describe the numbers of animals (eg. fish) of different species and are of the form:
where is the abundance of species i, its fitness and there are n species in total.
Often these interacting species oscillate in abundance.
There is a precise equivalence with the replicator system for (n+1) types given by the substitution

21. Replicator equation with mutation and quasispecies
Suppose there are errors in replicating. The probability of type j mutating to type i is
We obtain a replicator equation with mutation:
The equivalent with numbers rather than frequencies of types is
When the fitnesses do not depend on frequencies, this is the quasispecies eqn. (Probably the case in most GAs?)

22. Quasispecies equation
Describes molecular evolution (Eigen)
N biochemical sequences
Biochemical species i has frequency yi
Replication at rate fi is error-prone - mutation to type j at rate qij

23. Adaptive dynamics framework
Game consists of a continuous space of strategies (eg.)
Population is assumed to be homogeneous- all players adopt same strategy
Mutation generates variant strategies very close to the resident strategy
If a mutant beats the resident players it takes over otherwise it is rejected
Adaptive dynamics illustrates the nature of evolutionary stable strategies

24. Adaptive dynamics equations
Strategies are described by continuous parameters :
Expected score of mutant against S is given by E(S’,S)
The adaptive dynamics flow in the direction which maximises the score:

25. We can derive Price’s equation from replicator-mutator equation
Price’s equation from population genetics describes any type of selection.
Suppose an individual of type i, frequency , has some trait p of value
, so using the replicator equation with mutation we obtain
This applies when the values of are const.
[p is the expected mutational change in p.]

26. Price’s equation
selection
mutation

27. Price’s equation gives rise to adaptive dynamics
If we assume that the mutation is localised and symmetrical then we can neglect the second term in Price’s eqn.
Assume population is almost homogeneous and fitness is differentiable then we can Taylor expand the fitness, obtaining
cf. adaptive dynamics:

28. General framework Quasispecies equation Replicator-mutator equation
Price equation
Price
equation
Lotka-Volterra
equation
Game dynamical
equation
replicator
Price equation
Adaptive dynamics

29. Fisher’s fundamental theorem
Suppose fitnesses of genotypes constant. Can consider f as the trait p and obtain (for symmetric mutation):
Fisher’s fundamental theorem of NS
In general, fitnesses of genotypes depend on environment. In game theory context, depend on the frequencies of other genotypes. Fisher’s theorem doesn’t apply- eg. PD

30. Generalized version where
We can use Price’s equation to obtain a generalized version of Fisher’s fundamental theorem:
where
This applies when the s depend linearly on the frequencies of genotypes- normally the case in evolutionary game theory.

31. Fisher’s theorem and GAs
In most GAs, fitnesses of particular solutions (chromosomes) probably fixed and so (except for the complication of recombination) Fisher’s theorem should hold:
So for a GA with fitness-proportionate selection, no recombination and fixed fitness for a given solution, the average fitness of the population of solutions increases until there is no diversity left in the fitnesses.

32. Conclusions on unifying evolutionary dynamics
Unifying framework
Different frameworks for different problems.
We derive from Price’s equation a generalized version of Fisher’s Fundamental Theorem of Natural Selection.
The Price – replicator framework can also be applied to discrete time formulations and to formulations with sexual reproduction.

33. Relationship to GAs

34. Evolutionary games and genetic algorithms
Two-way interaction:
1) So far discussed computer simulations of evolutionary processes, eg. evolution of animal behaviour
2) Evolutionary computation, eg. genetic algorithms = computer science based on theory of biological evolution
Evolutionary games very like genetic algorithms- but
1) Population size is usually quite large and may be few phenotypes: space well searched but not v. efficient.
2) Usually no recombination
3) Fitnesses depend on interactions
Refer to Mark Herbster’s course. Ask students about course.

35. Genetic Algorithms
Evolutionary models are computer algorithms which use evolutionary methods of optimisation to solve practical problems (cf. finding stable strategies in games rather than working out ‘rational’ solution)- eg. Evolutionary programming, genetic algorithms
Evolutionary operations involved in genetic algorithms: selection, mutation, recombination:
Explain selection and recombination

36. How evolutionary dynamics relates to GAs
GAs evolve by selection and mutation  their dynamics can be (to some extent) described by the replicator equation with mutation (cf. unifying framework).
The replicator equation describes fitness-proportionate selection.
Ficici, Melnik and Pollack (2000) - effects of different types of selection (eg. truncation) on the dynamics of the Hawk-Dove game + relevance for evolutionary algorithms. Can lead to different dynamics.
Must also consider the effects of recombination.
Ficici, S.G., Melnik, O., and Pollack, J.B. (2000) "A Game-Theoretic Investigation of Selection Methods Used in Evolutionary Algorithms." In Proceedings of the 2000 Congress on Evolutionary Computation. Zalzala, A., et al (eds.). IEEE Press.

37. Incorporating recombination into the replicator framework
Do this by assuming that rjk;i = probability that when parent chromosome of type j combines with parent chromosome of type k, an offspring of type i is formed.
No mutation, recombination after replication:
[NB discrete-time version]

38. Adding in mutation
Add in mutation. Assume, as before, is probability type i mutates to form type j ( large). Assume this happens after recombination.
What we had before was
What we have now is

39. The diversity of the population and adaptive dynamics
From Fisher’s theorem, see that no diversity of fitness in population  no further increase in average fitness.
However, because the variation in the parameters of the your system has become very small (population convergence), does not mean no further evolution.
In the case of small variation, we can apply the adaptive dynamics framework which shows how the average values of traits (parameters) will change in time

40. Relationship: evolutionary games & GAs - Conclusions
Often evolution leads in the long run to ‘optimal’ solutions, like Nash equilibria.
Ability of evolutionary processes to seek out optimal strategies has been exploited in computer science by the development of genetic algorithms and evolutionary computation for problem solving.
Comparing with the use of computer simulations to study biological evolution, we see that there is a two-way interaction between biological evolutionary theory and computer science.

41. Relationship to GAs- Conclusions
Frameworks of evolutionary dynamics can be applied to GAs by modifying them to include recombination.
Which framework is most informative depends on the individual problem, but we have shown they are equivalent.
Eg. can look at detailed dynamics using the replicator-mutator framework
Or we can look at a “converged” population using the adaptive dynamics framework.
Looking further at the relationship between GAs and evolutionary dynamics could yield new solutions/ techniques for both.

42. Acknowledgements Martin Nowak (IAS, Princeton)
Terry Leaves (BNP Paribas, London)
Karl Sigmund (Univ. Vienna)
Steven Frank (Univ. California, Irvine)
Peter Bentley (UCL)
Christoph Hauert (Univ. British Columbia)
Anargyros Sarafopoulos (Univ. Bournemouth)
Bernard Buxton (UCL)
To do: Look at Lande - quantative genetics stuff - G covariance matrix
Stochastic dynamics
Coevolutionary dynamics