E. Altman INRIA, France Advances in Evolutionary Games Bionetics Dcember 2010

Overview of the talk 1. Background on Evolutionary Games and Population Games and Examples 2. Adding time varying states 3. Adding controlled state transitions 4. Examples 5. Mathematical model 6. Computing Equilibria

EGs and PGs in Biology and Engineering BIOLOGY CONTEXT: Central tool defined by Meynard Smith (1972) for explaining and predicting dynamics of large competing populations with many limited local interactions. EG. TELECOM CONTEXT: Competition between protocols, technologies. Can be used to design and regulate evolution ROAD TRAFFIC CONTEXT: Competition between cars over routes. Introduced by Wardrop (1952). PG.

Framework Large population Several strategies (behavior of individuals). Call all those who use a strategy a subpopulation EG: Competition between the strategies through a very large number of interactions each involving a small number of individuals typically pairwize interactions PG: interactions with an infinite number of players

Evolutionary Games: Definitions Evolutionary Stable Strategy (ESS): At ESS, the populations are immuned from being invaded by other small populations (mutations). ESS more robust than standard Nash equilibrium. “State” vector X: fractions of users that belong to different populations. Or fractions of strategies in a population Fitness: J(p,q):= utility when playing pure strategy p and all others play q. J(x,y) fitness when using mixed strategies x and y, resp.

Evolutionary Stable Strategy Evolutionary Stable Strategy (ESS): p is ESS if for all q there is d(q) s.t. for all 0<e<d(q) J(p,p) >= (1-e) J(q,p) + e J(q,q) Equivalent condition: J(p,p) > J(q,p) or J(p,p) = J(q,p) and J(q,q) < J(p,q)

Ex 1: Hawk and Dove Game Large population of animals. Occasionally two animal find themselves in competition on the same piece of food. An animal can adopt an aggressive behavior (Hawk) or a peaceful one (Dove). D-D: peaceful, equal-sharing of the food. fitness of 0.5 to each player. H-D or D-H: 0 fitness to D and 1 for H that gets all the food no fight

HD Game H-H: fight in which with equal chances to obtain the food but also to be wounded. Then the fitness of each player is 0.5-d, -d is the expected loss of fitness due to being injured.

Modeling competition: Generalized HD Game Generalized game: A11<A22<A12 and A21<A22. Simple conditions for H to be unique ESS and for mixed ESS

Ex 2: Competition between protocols There are various flow control protocols to regulate traffic in the Internet. Huge number of file transfers every second Interactions occur between limited number of connections that use the same bottleneck link The average speed of transfer, the delay etc depend on the versions of the protocol involved in the interaction

Competition between protocols

Ex 3: Population Games (PG) in Wireless communications Cellular network contains many mobiles. One base station (BS) per cell CDMA: At each time an individual sends a packet it interacts with all mobiles in the same cell A mobile can transmit with different power levels q 1 < … < q K. Higher power is more costly Objective of : max _k J (k,w) := where w k is the fraction of mobiles that use q k

Replicator Dynamics Delayed case: present growth rate depends on past fitness K and tau : design parameters. Determine speed of convergence and stability

Architecting evolution: impact of K stability iff K tau< θ. Oscillations mean no convergence to ESS.

Individual States in EG and PG Different behaviors may be a result of different inherent characteristics – individual states Example: weather conditions, age, The individual state can be random Description through a Markov chain EG: Local interactions with players chosen at random; their state is unknown PG: Global interactions, the state can be known

Indiv. states in HD Game The decisions H or D determine whether a fight will occur There is also a true identity -- Strong or Weak We call this the individual STATE If there is a fight then the states determine the outcome. Note: the decision H/D are taken without knowing the state of the other.

Indiv. States in Networks Flow control protocol: large end to end delay slows the protocol and decreases its throughput Wireless: - the power received may depend on the radio channel conditions - the transmitted power may depend on the energy level of the battery

MDEG: Markov Decision EG ASG: Anonymous Sequential G Each player has a controlled Markov chain (MDP) A player has finite or infinite life time. It has several interactions each time with another randomly selected player (MDEG) with a large population (ASG) Each interaction results in an immediate fitness that depends on the actions and states of the players involved The states and actions of a player determine also the probability distribution of the next state

Assumptions, References A player maximizes the total expected or average fitness EG average fitness: EA & YH IEEE trans Autom Contr, June 2010 (theory) EG total expected fitness: Infocom 2008 (power control) Evolutionary Ecology Research, 2009 (the theory) (EA, YH, R El-Azouzi, H. Tembine) SAG: Jovanovic & Rosenthal, J Math. Econ, 1988 (disc cost) Assume: The transition probabilities of the MDP of a player depend only on its own actions and states

Ex 1 (MDEG): Hawk and Dove game A bird that looses becomes weaker (less energy) A very weak bird dies State: Energy level Would a weaker bird be more or less aggressive? If the result of the fight are determined by the energy level then the transitions are determined by states and actions of both birds.

Ex 2: (MDEG or ASG): Battery dependent power control Transmitting at higher power empties faster the battery A battery with little energy left is not able to support transmissions at high power The state: remaining energy in the battery The transitions do not depend on other mobiles

Ex 3: channel dependent power control The decision to transmit at power q k may depend on the channel state Seems “degenerate”: the mobile does not control the transitions Restriction: discrete power set; if a power level is chosen then the next power cannot differ by more than one unit. This creates non-trivial transitions. The state = (Channel state, current power level)

MDEG: Local interactions Each local interaction is described by a stochashtic game with partial monitoring The stochastic game has an equilibrium. The game is equivalent to a matrix game where the pure actions of a player are its pure stationary policies Allows us to transform the problem into a standard EG with a huge action space (action=pure policy) We show: equivalence to a polytope game in the space of marginal stationary occupation measures Cardinality: no. of states times number of actions

Model of Individual player Each player is associated with a MDP with POWER CONTROL ASG

The model for an individual State of an individual corresponds to the battery level. Set of actions available at state s: Qs decreases with the energy: smaller powers are available when the battery has less energy Transitions: the probability to stay at a state s if q is used is Recharging: P 0N is the probability to move from 0 to N

The model: Interactions Global state: fraction of mobile in each individual state: Proportion of mobiles using q k at time t is The Reward: Stationary policy the ptob to choose q k in s

Interactions and System model In stationary regime:

Interactions and System model The expected reward: A stationary policy u is an equilibrium if

Results (1) Define the interference of u: Denote the probability distribution with mass 1 at q

Results (2) Threorem. An equilibrium exists within

Results (3)

Future work Branching MDPs: a state-action pair of an individual determines the immediate fitness, the transition probabilities and the number of off- springs