Applications of Game Theory Part II(b) John C.S. Lui Computer Science & Eng. Dept The Chinese University of Hong Kong

On the interaction between Overlay Routing and Underlay Routing Y. Liu, H. Zhang, W. Gong, D. Towsley INFOCOM 2005 First course

Motivation : Interactions Between Application Level Network and Physical Network  physical network control –routing, congestion control,… Control Result? –interactions? –controllers mismatch?  add an overlay  and another……

Outline  Problem Formulation  Simulation Study  Game-theoretic Study  Conclusions

 Routing on physical network level  Inter-domain: BGP, etc.  Intra-domain: OSPF, MPLS, etc. –determine routes for all source-destination traffic demand pairs –minimize network-wide delay, cost, etc. Routing in Underlay Network traffic demand pair: A->B traffic demand pair: A->C traffic demand pair: C->B A C E B D

 An overlay network choose routes at application level to minimize its own delay or cost Routing in Overlay Network A C E B D A C B Overlay demand: A->B logical routes: A->C->B and A->B demand pair: A->C  Overlay  gains advantage better path: delay, loss, throughput, etc  is selfish potential performance degradation to other non-overlay traffic demand pair: C->B demand pair: A->B

Considering overlay and underlay together ?  How do they interact with each other?  How does selfish behavior of overlay routing –affect overall network performance? –affect non-overlay traffic performance? –affect its own performance?

Interactions Between Overlay Routing and Underlay Routing Overlay Routing Optimizer To minimize overlay cost Underlay Routing Optimizer To minimize overall network cost flow allocation on logical links: “X” traffic demand for underlay flow allocation on physical routes: “Y” non-overlay traffic demand overlay traffic demand Iterative Dynamic Process  equilibrium: existence? uniqueness?  dynamic process: convergence? oscillations?  performance of overlay and underlay traffic?

Approach by authors  Focusing interaction in a single AS  Considering two routing models for overlay and one routing model for underlay  Simulating the interaction dynamic process  Studying this process in a Game-theoretic framework

Routing Models  Overlay routing model –Selfish source routing  Individual user controls infinitesimal amount of traffic, to minimize its own delay –Optimal overlay routing  A central entity minimizes the total delay of all overlay traffic demands  Underlay routing model  Optimal underlay routing  A central entity minimizes the total delay of all network traffic, e.g. Traffic Engineering MPLS

Node without overlay Link Node with overlay Simulation Study: Optimal Overlay and Optimal Underlay 14 node tier-1 POP network (Medina et.al. 2002) bimodal normal model of traffic demand 3 overlay nodes

Simulation Study ( case 1: 8% overlay traffic) Optimal Overlay and Optimal Underlay after underlay takes turn after overlay takes turn Iterative process  Underlay takes turn at step 1, 3, 5, …  Overlay takes turn at step 2, 4, 6, … iteration average delay of overlay traffic percentage % overlay performance improvement iteration average delay of all traffic percentage % underlay performance degradation

iteration percentage % overlay performance degradation average delay of overlay traffic after underlay takes turn after overlay takes turn Iterative process  Underlay takes turn at step 1, 3, 5, …  Overlay takes turn at step 2, 4, 6, … iteration percentage % underlay performance degradation average delay of all traffic Simulation Study (case 2: 10% overlay traffic) Optimal Overlay and Optimal Underlay

Game-theoretic Study  Two-player non-zero sum game overlay Underlay X: strategy of “overlay” traffic allocation on logical links Y: strategy of “underlay” traffic allocation on physical links : Cost of “overlay” : Cost of “underlay” : Constraints of “overlay” : Constraints of “underlay”

Game-theoretic Study Best-reply dynamics Nash Equilibrium

Optimal Underlay Routing v.s. Optimal Overlay Routing  Overlay –One central entity calculates routes for all overlay demands, given current underlay routing –Assumption: it knows underlay topology and background traffic AB C X(k) Denote overlay’s routing decision with a single variable X(k): overlay’s flow on path ACB after round k 1-X(k)

 There exists unique Nash equilibrium x*,  x* globally stable: x(k)  x*, from any initial x(1) Best-reply Dynamics iteration k Overlay Routing EvolutionOverlay Delay Evolution x(k) x* x(k)<x(k+1)<x* delay Underlay’s turn Overlay’s turn When x(1)=0, overlay performance improves

Round k x(k) x* Underlay’s turn Overlay’s turn BAD INTERACTION! x(k)>x(k+1)>x* x(k)<x(k+1)<x* Overlay Delay Evolution delay Overlay Routing Evolution Best-reply Dynamics  There exists unique Nash equilibrium x*,  x* globally stable: x(k)  x*, from any initial x(1) When x(1)=0.5, overlay performance degrades

Conclusions & Open Issues  Selfish overlay routing can degrade performance of network as a whole  Interactions between blind optimizations at two levels may lead to lose-lose situation  Future work: –larger topology: analysis/experimentation –overlay routing and inter-domain routing –interactions between multiple overlays (****) –implications on design overlay routing –regulation between overlay and underlay (****)

On the Interaction of Multiple Overlay Routings Performance 2005 Joe W.J. Jiang, D.M. Chiu, John C.S. Lui Second course

Questions These overlays tend to fully utilize available resource. So, is there any anarchy? How do overlay networks co-exist with each other? What is the implication of interactions? How to regulate selfish overlay networks via mechanism design? Can ISPs take advantage of this?

Outline Motivation Mathematical Modeling Overlay Routing Game Implications of Interaction Pricing Conclusion

Motivation Overlays provide a feasibility for users to control their own routing. Routing, possible multi-path, becomes an optimization problem. Interaction occurs (due to same underlay) Interaction between one overlay and underlay traffic engineering, Zhang et al, Infocom ’ 05. Interaction between co-existing overlays ? Adaptive routing controls on multiple layers (overlays, underlay TE -- traffic engineering) over one common physical network Simultaneous feedback controls over one system Stability ? Performance ?

Performance Characteristics Objective: minimize end-to-end delay (e.g., RON) Delay of a physical link e : Performance Characteristics (Underlay) d e (l e ) l e – aggregate traffic traversing link e Average delay ( f : flow)

Performance Characteristics Objective: minimize end-to-end delay Delay of a physical link e : Performance Characteristics (Underlay) d e (l e ) l e – aggregate traffic traversing link e Average delay (multipath routing)

Performance Characteristics Objective: minimize end-to-end delay Delay of a physical link e : Performance Characteristics (Underlay) d e (l e ) l e – aggregate traffic traversing link e Average delay (multipath routing)

System Objectives Network Operators –Min average delay in the whole underlay network Overlay Users –Min average delay experienced by the overlay

How do Overlays Interact? Overlapping physical links. Performance dependent on each other. Selfish routing optimization. Overlays are transparent to each other. Lack of information exchange between overlays.

Contribution What is the form of interaction? Is there routing instability (oscillation), or there is an equilibrium ? Is the routing equilibrium efficient? What is the price of anarchy? Fairness issues Mechanism design: can we lead the selfish behaviors to an efficient equilibrium?

Mathematical Modeling Overlay routing: An optimization problem Decision variable: routing policy s : overlay f : flow r : path

Mathematical Modeling Overlay routing: An optimization problem Objective: average weighted delay (matrix form) Routing Matrix Delay Function (vector form)

Overlay Routing Optimization Convex programming Demand constraint (fixed transmission demand) Capacity Constraint Non-negative Flow Constraint

Algorithmic Solution Unique optimizer –Convex programming –feasible region: convex –delay function: continuous, non-decreasing, strictly convex Solution –Apply any convex programming techniques. –Marginal cost network flow (probabilistic routing ICNP ’ 04). –This is solved in an independent, and distributed fashion by each overlay. But will independent optimization leads to system instability (route flop)?

Overlay Routing Game Nash Routing Game –Player -- N all overlays –Strategy --  s feasible routing policy: feasible region of OVERLAY (s) –Preference relation -- ≥ s low delay: player ’ s utility function is -delay (s) Strategic Game: G overlay

Illustration of Interaction Routing Underlay Overlay 1 Overlay 2 Overlay n … Routing decision on logical paths in overlay 1 Routing decision on logical paths in overlay 2 Routing decision on logical paths in overlay n Aggregate overlay traffic … Underlay (non-overlay) traffic Aggregate traffic on physical links Overlay probing Delay of logical paths in overlay 1 Delay of logical paths in overlay 2 Delay of logical paths in overlay n ∑

Existence of Nash Equilibrium Definition – Nash equilibrium point (NE) A feasible strategy profile y=(y (1),…, y (s),…, y (n) ) T is a Nash equilibrium in the overlay routing game if for every overlay s ∈ N, delay (s) (y (1),…y (s),…y (n) ) ≤delay (s) (y (1),…y’ (s),…y (n) ) for any other feasible strategy profile y’ (s).

Existence of Nash Equilibrium Theorem In the overlay routing game, there exists a Nash equilibrium if the delay function delay (s) (y (s) ; y (-s) ) is continuous, non-decreasing and convex. Good News: NO ROUTE FLOP !!!

Fluid Simulation Six overlays One flow per overlay Congested network Asynchronous routing update

Overlay performance Transient period Quick convergence

Overlay routing decisions

The Price of Anarchy Global Performance (average delay for all flows) GOR: Global Optimal Routing NOR: Nash equilibrium for Overlay Routing Game NSR: Nash equilibrium for Selfish Routing GORNOR NSR Efficiency Loss ?

Selfish Routing (User) selfish routing: a single packet ’ s selfishness Every single packet chooses to route via a shortest (delay) path. A flow is at Nash equilibrium if no packet can improve its delay by changing its route.

Selfish Routing Also a Nash equilibrium of a mixed strategic game –Player: flow { f } –Strategy: p  P f –Preference: low delay System Optimization Problem

Performance Comparison Overlay One Overlay Two Average Delay Centralized Global Optimal Routing NE of Overlay Optimal Routing NE of Selfish Routing

Inspiration Is the equilibrium point efficient (at least Pareto optimal) ? Fairness issues of resource competition between overlays.

Example Network 1 unit y1y1 1-y 1 y2y2 1-y 2

Sub-Optimality physical linkdelay function d e (l e ) 1-51+l 3-4l l Routing (y 1, y 2 ) Average Delay (overlay1, overlay2 ) NE (0.5, 1.0)(1.5, 1.5) Pareto Curve (0.4, 0.9)(1.4, 1.4) y1y1 y2y2 Non Pareto-optimal !

Fairness Paradox physical linkdelay function d e (l e ) 1-5a+l 3-4bl  2-6c+lc+l y1y1 y2y2  a, b, c,  are non-negative parameters  Everything is symmetric except two private links – a & c

Fairness Paradox physical linkdelay function d e (l e ) 1-5a+l 3-4bl  2-6c+lc+l y1y1 y2y2 a < c Overlay 1 has a better “private” link !

Fairness Paradox y1y1 y2y2 a < c  delay 1 < delay 2 Unbounded Unfairness

War of Resource Competition 1 unit y1y1 1-y 1 y2y2 1-y 2 p oil (y 1 +y 2 ) p usa (1-y 1 ) p chn (1-y 2 ) p usa < p chn USAChina Min Cost usa (y 1 ; y 2 ) = y 1 p oil (y 1 +y 2 )+(1-y 1 )p usa (1-y 1 )

War of Resource Competition 1 unit y1y1 1-y 1 y2y2 1-y 2 p oil (y 1 +y 2 ) p usa (1-y 1 ) p chn (1-y 2 ) p usa < p chn USAChina Min Cost chn (y 2 ; y 1 ) = y 2 p oil (y 1 +y 2 )+(1-y 2 )p chn (1-y 2 )

War of Resource Competition 1 unit p oil (y 1 +y 2 ) p usa (1-y 1 ) p chn (1-y 2 ) p usa < p chn  Cost usa > Cost chn USA China

Pricing (opportunity for ISP) Inefficient Nash equilibrium Desired equilibrium Mechanism Design  Performance degradation (sub-optimal)  Fairness paradox  Global optimality  Improve fairness payment new Nash equilibrium

Pricing I – Improve Delay Objective: to achieve global optimality NE of overlay routing game Global optimal l e (s) : traffic of overlay s l e (-s) : traffic other than overlay s

Pricing I – Improve Delay Objective: to achieve global optimality New NE of overlay routing game Global optimal Heterogeneous pricing

Pricing I – Improve Delay New NE of overlay routing game Global optimal KKT condition: p e (s) =l e (-s) d e ’ (l e )

Pricing II – improve fairness Cause of unfairness: –Over-utilize good common resources –Unfair resource (bandwidth) allocation Pricing Scheme ISP maximize profit Improve performance & Reduce cost Overlay price p routing decision

Incentive Resource Allocation For overlays: : sensitivity factor new Nash equilibrium  {l e }

Revenue Distribution For ISPs (links): : profit of link e : revenue : operating cost --

Effectiveness of Pricing

Conclusion Study the interaction between multiple co- existing overlays. Non-cooperative Nash routing game. Prove the existence of NEP. Show the anomalies and implications of the NEP. Present two distributed pricing schemes to address the anomalies.

Third Course Interaction of ISPs: Distributed Resource Allocation and Revenue Maximization Sam C.M. Lee, Joe W.J. Jiang, D.M Chiu, John C.S. Lui

View of ISPs Tier-1 ISP Tier-2 ISP Local ISP Peering link

Tier-2 ISP Local ISP Peering link ISP Peer ISP link ISP Peer

Peer kPeer j Tier-2 ISP (ISP) Peer i 1. performance of the link 2. charge of the link Issues to consider: Optimization problem of peers

Happiness obtained from sending traffic to peers Delay cost in ISP link Payment to ISP Delay costs in peering links Payments to peers

Constraints of peers

Solution to the peers Objective function is strictly concave in every transmission rate The optimal transmission rates and maximum utility are unique and can be found by the Lagrangian method.

Problems for an ISP Maximization of revenue –How to determine the optimal value of unit price Resource distribution –How to determine the capacity for the peers

Information exchange framework ISP peer Bandwidth allocation Bid Compute resource distribution Compute optimal rates Next period

ISP 1: Resource distribution peer1 Bid = 50MBps ??? ISP peer2 peer3 Bid = 100MBps Bid = 150MBps Bandwidth = 600MBps

Proportional share algorithm peer1peer2 peer3 Bid = 50MBps Bid = 100MBps Bid = 150MBps ISP Bandwidth = 600MBps 100MBps200MBps300MBps

Equal share algorithm peer1peer2 peer3 Bid = 50MBps Bid = 100MBps Bid = 150MBps ISP Bandwidth = 600MBps 150MBps200MBps250MBps

Simulations When the happiness coefficients of peers are low PSA ESA

Simulation When the happiness coefficients of peers are high PSA ESA

ISP 2: Maximization of Revenue Unit price Demand by peer i Determine the optimal price Total revenue from the peers

Solution: Maximization of revenue Estimate the aggregate traffic ( ) from all peers in term of the price (P)

Conclusions Utility maximization of a peer Resource distribution of ISP Revenue maximization of ISP

Fourth Course On the Access Pricing Issues of Wireless Mesh Networks ICDCS 2006 Ray K. Lam Dah-Ming Chiu John C.S. Lui

WMN Paints a Bright Future Wireless mesh network (WMN) –Wireless nodes –Multi-hop routing –Form a wireless “mesh” More access to the Internet –More people, rich or poor –More ubiquitous, anywhere, anytime –More opportunities to everyone Internet

The Critical Thing—Cooperation Multi-hop routing Relay packets for each other My concerns: bandwidth, CPU time, security… Community network with symmetric traffic Help each other => mutual benefit Access network with asymmetric traffic Geographically good VS poor Why help the poor? Incentive system needed—pricing

When AP Meets a Client Simple analysis by Musacchio and Walrand [1] A game with 2 players –Access point (AP) provides Internet access –Client buys the service –One deal per time slot APClient p1p1 accept p2p2 p3p3 reject slot 1 slot 2 slot 3 service duration APClient p

A Beautiful Equilibrium AP and client each maximizes her gain –AP: guess the “right” price –Client: compare the price p with service utility U Web browsing utility function A beautiful equilibrium –AP has the same optimal price in every time slot –Client connects if her per-slot service utility is greater than slot price (U > p) Encourages flat-rate pricing

To a Multi-hop Scenario Adding a relaying node, or “reseller” (RS) RS tries to mark up AP’s price to the “right” level AP takes note of RS’s action when setting her price Equilibrium is still flat- rate pricing Multi-hop => multiple RSs APClient c1c1 accept slot 1 slot 2 slot 3 service duration p1p1 accept c2c2 p2p2 c3c3 reject p3p3 RS APClientRS c p

Drawbacks of the Simple Model Assuming unlimited network capacity –2-player game represents whole system –Treat every incoming client the same –Unlimited admission => unlimited capacity Assuming a tree-like network –2-hop / multi-hop linear network extension –Does not consider multiple paths –Pricing competition may occur A tree-like network A graph-like network APClientRS

What If Capacity Limited? Cannot admit unlimited clients –Client demands bandwidth guarantee –AP admission control –AP’s system capacity: m 2-player game not enough –AP deals with each client differently –Client arrival model: Poisson process Like an M/M/m/m/M queuing system

Flat-rate Pricing Fails… Failure scenario –AP is full; m clients admitted –An admitted client a is paying $5/slot –A new client b arrives –AP asks b for $6/slot –If b accepts AP raises price for a to $6/slot, OR Simply kicks a out Flat-rate pricing is not optimal!

Everybody Loves Flat Rate Unrealistic for variable rate More practical—fixed-rate, non- interrupted service –AP charges a client a fixed rate p over time –AP cannot disconnect a client unilaterally AP can still charge different clients at different “fixed” rates –How to set the optimal rate on different occasions?

Best Strategy in New Service Model AP sets price based on remaining capacity –Raises price when becoming full –State price: at state k, AP charges next “to-be- admitted” client at fixed rate p k –Policy of AP characterized by price vector Client’s best strategy –Connect AP if service utility per unit time > price per unit time (U > p)

System Dynamics State transition –Adding a factor P(U > p k ) to regular arrival rate in M/M/m/m/M model Reward structure –Simplification: immediate expected profit when a client connects 012mm-1 …  M P(U > p 0 )  M-1) P(U > p 1 )  M-m+1) P(U > p m-1 )  mm State transition diagram

System Dynamics State transition –Adding a factor P(U > p k ) to regular arrival rate in M/M/m/m/M model Reward structure –Simplification: immediate expected profit when a client connects Reward Structure 012mm-1 … p0/p0/ 000 p1/p1/ p m-1 / 

Finding Optimal Price Vector Classical optimization –Solution for queuing system gives limiting state probability for each state k,  k –Gain of AP is a function of price vector –Complicated to optimize with classical techniques Policy-iteration method in Markovian decision theory –Reduces computational complexity by iterative algorithm –Guarantees convergence to the best policy

Numerical Results Capacity m=5, population M=10, departure rate  =1 Vary arrival rate from 0.2 to 10 Utility U uniformly distributed on [0,10] U normally distributed with mean 5, s.d Price increases number of clients in AP and with

Limited Capacity in Multi-hop Case Simplification –Traffic merges at AP –AP is the bottleneck –Only AP controls admission AP’s policy specified by a price matrix –At each state, different prices for requests from different distances –p ki : price at state k for a client i-hop away APClientRS Internet bandwidth bottleneck

System Dynamics Removing finite population –Complicates state information –Different arrival rates for clients at different distances 01 … 1  P(U > m 1 (p 0,1 ))  mm 2  P(U > m 2 (p 0,2 )) n  P(U > m n (p 0,n )) … m-1m 1  P(U > m 1 (p m-1,1 )) 2  P(U > m 2 (p m-1,2 )) n  P(U > m n (p m-1,n )) … Client 1-hop away arrives Client 2-hop away arrives Client n-hop away arrives State transition diagram

System Dynamics 01 … 0 0 p 0,n /  … m-1m … Client 1-hop away arrives Client 2-hop away arrives Client n-hop away arrives p 0,2 /  p 0,1 /  p m-1,n /  p m-1,2 /  p m-1,1 /  Reward Structure Removing finite population –Complicates state information –Different arrival rates for clients at different distances

Conclusion Contributions –Show that fixed-rate pricing fails with limited capacity –Generalize unlimited capacity model into limited capacity model –Devise optimal pricing for fixed-rate, non-interrupted service with Markovian decision theory References [1]J. Musacchio and J. Walrand. WiFi access point pricing as a dynamic game. IEEE/ACM Trans. Networking. to appear in.