1. Stochastic Optimization for Markov Modulated Networks with Application to Delay Constrained Wireless Scheduling Michael J. Neely University of Southern California http://www-rcf.usc.edu/~mjneely Proc. 48 th IEEE Conf. on Decision and Control (CDC), Dec. 2009 *Sponsored in part by the DARPA IT-MANET Program, NSF OCE-0520324, NSF Career CCF-0747525 A 1 (t) A 2 (t) A L (t) State 1 State 2 State 3 Control-Dependent Transition Probabilities

2. Motivating Problem: Delay Constrained Opportunistic Scheduling A 2 (t) A K (t) A 1 (t) S 1 (t) S 2 (t) S K (t) Status Quo: Lyapunov Based Max-Weight: [Georgiadis, Neely, Tassiulas F&T 2006] Treats stability/energy/thruput-utility with low complexity Cannot treat average delay constraints Dynamic Programming / Markov Decision (MDP) Theory: Curse of Dimensionality Need to know Traffic/Channel Probabilities

3. Insights for Our New Approach: Combine Lyapunov/Max-Weight Theory with Renewals/MDP A 2 (t) A K (t) A M (t) A 1 (t) S 1 (t) S 2 (t) S K (t) A K+1 (t) S K+1 (t) S M (t) Consider “Small” number of Control-Driven Markov States K Queues with Avg. Delay Constraints (K “small”) N Queues with Stability Constraints (N arbitrarily large)  Lyapunov Functions  Max-Weight Theory  Virtual Queues  Lyapunov Functions  Max-Weight Theory  Virtual Queues  Renewal Theory  Stochastic Shortest Paths  MDP Theory  Renewal Theory  Stochastic Shortest Paths  MDP Theory Example : Delay Constrained Not Delay Constrained

4. Key Results: Unify Lyapunov/Max-Weight Theory with Renewals/MDP “Weighted Stochastic Shortest Path (WSSP)” “Max Weight (MW)” Treat General Markov Decision Networks Use Lyapunov Analysis and Virtual Queues to Optimize and Compute Performance Bounds Use Existing SSP Approx Algs (Robbins-Monro) to Implement For Example Delay Problem: Meet all K Average Delay Constraints, Stabilize all N other queues Utility close to optimal, with tradeoff in delay of N other queues All Delays and Convergence Times are polynomial in (N+K) Per-Slot Complexity geometric in K

5. General Problem Formulation: (slotted time t = {0,1,2,…}) Q n (t) = Collection of N queues to be stabilized S(t) = Random Event (e.g. random traffic, channels) Z(t) = Markov State Variable (|Z| states) I(t) = Control Action (e.g. service, resource alloc.) x m (t) = Additional Penalties Incurred by action on slot t Q n (t) R n (t)  n (t)  n (t) =  n (I(t), S(t), Z(t)) R n (t) = R n (I(t), S(t), Z(t)) x m (t) = x m (I(t), S(t), Z(t)) Z(t) Z(t+1) I(t), S(t) State 1 State 2 State 3 Control-Dependent Transition Probs: General functions for  (t), R(t), x(t):

6. General Problem Formulation: (slotted time t = {0,1,2,…}) Q n (t) = Collection of N queues to be stabilized S(t) = Random Event (e.g. random traffic, channels) Z(t) = Markov State Variable (|Z| states) I(t) = Control Action (e.g. service, resource alloc.) x m (t) = Additional Penalties Incurred by action on slot t Q n (t) R n (t)  n (t)  n (t) =  n (I(t), S(t), Z(t)) R n (t) = R n (I(t), S(t), Z(t)) x m (t) = x m (I(t), S(t), Z(t)) General functions for  (t), R(t), x(t): Goal: Minimize: x 0 Subject to: x m < x m av, all m Q m stable, all m

7. Applications of this Formulation: For K of the queues, let: Z(t) = (Q 1 (t), …, Q K (t)) These K have Finite Buffer: Q k (t) in {0, 1, …, B max } Cardinality of states: |Z| = (B max +1) K Recall: Penalties have the form: x m (t) = x m (I(t), S(t), Z(t)) 1)Penalty for Congestion: Define Penalty: x k (t) = Z k (t) Can then do one of the following (for example): Minimize: x k Minimize: x 1 + … + x K Constraints: x k < x k av

8. Applications of this Formulation: For K of the queues, let: Z(t) = (Q 1 (t), …, Q K (t)) These K have Finite Buffer: Q k (t) in {0, 1, …, B max } Cardinality of states: |Z| = (B max +1) K Recall: Penalties have the form: x m (t) = x m (I(t), S(t), Z(t)) 2) Penalty for Packet Drops: Define Penalty: x k (t) = Drops k (t) Can then do one of the following (for example): Minimize: x k Minimize: x 1 + … + x K Constraints: x k < x k av

9. Applications of this Formulation: For K of the queues, let: Z(t) = (Q 1 (t), …, Q K (t)) These K have Finite Buffer: Q k (t) in {0, 1, …, B max } Cardinality of states: |Z| = (B max +1) K Recall: Penalties have the form: x m (t) = x m (I(t), S(t), Z(t)) 3)A Nice Trick for Average Delay Constraints: Suppose we want: W < 5 slots : Define Penalty: x k (t) = Q k (t) – 5 x Arrivals k (t) Then by Little’s Theorem… x k < 0 equivalent to: Q k – 5 x k < 0 equivalent to: W k x k – 5 x k < 0 equivalent to: W k < 5

10. Solution to the General Problem: Minimize: x 0 Subject to: x m < x m av, all m Q k stable, all k Define Virtual Queues for Each Penalty Constraint: Define Lyapunov Function: L(t) = Q k (t) 2 + Y m (t) 2 Y m (t) x m (t) x m av

11. Solution to the General Problem: Define Forced Renewals every slot i.i.d. probability  State 1 State 2 State 3 Renewal State 0 Example for K Delay-Constrained Queue Problem: Every slot, with probability , drop all packets in all K Delay-Constrained Queues (loss rate < B max  ) Renewals “Reset” the system

12. Solution to the General Problem: Define Variable Slot Lyapunov Drift over Renewal Period  T (Q(t), Y(t)) = E{L(t+T) – L(t)| Q(t), Y(t)} where T = Random Renewal Period Duration tt+T Control Rule: Every Renewal time t, observe queues, Take action to Min the following over 1 Renewal Period: Minimize:  T (Q(t), Y(t)) + VE{ x 0 (  ) | Q(t), Y(t)}  =t t+T-1 *Generalizes our previous max-weight rule from [F&T 2006] !

13.  =t t+T-1 Minimize:  T (Q(t), Y(t)) + VE{ x 0 (  ) | Q(t), Y(t)} Max-Weight (MW) Weighted Stochastic Shortest Path (WSSP) Suppose we implement a (C,  )-approximate SSP, so that every renewal period we have… Achieved Cost < Optimal SSP + C +  [ Q k + Y m + V] Can achieve this using approximate DP Theory, Neurodynamic Programming, etc., (see [Bertsekas, Tsitsiklis Neurodynamic Programming]) together with a Delayed-Queue-Analysis.

14. Theorem: If there exists a policy that meets all Constraints with “  max slackness,” then any (C,  ) approximate SSP implementation yields: 1)All (virtual and actual) Queues Stable, and: E{Q sum } < (B/  + C  ) + V(  + x max )  max -  2)All Time Average Constraints are satisfied ( x m < x m av ) 3)Time Average Cost satisfies: x 0 < x 0 (optimal) + (B/  + C  ) +  x max /  max ) V (recall that  = forced renewal probability)

15. Proof Sketch: (Consider exact SSP for simplicity)  =t t+T-1  T (Q(t), Y(t)) + VE{ x 0 (  ) | Q(t), Y(t)} < B + VE{ x 0 (  ) | Q(t), Y(t)}  =t t+T-1 - Q k (t)E{ [  k (  ) – R k (  )] | Q(t), Y(t)}  =t t+T-1 - Y m (t)E{ [x m av – x m (  )] | Q(t), Y(t)}  =t t+T-1 [We take control action to minimize the Right Hand Side above over the Renewal Period. This is the Weighted SSP problem of interest]

16. Proof Sketch: (Consider exact SSP for simplicity)  =t t+T-1  T (Q(t), Y(t)) + VE{ x 0 (  ) | Q(t), Y(t)} < B + VE{ x 0 *(  ) | Q(t), Y(t)}  =t t+T-1 - Q k (t)E{ [  k *(  ) – R k *(  )] | Q(t), Y(t)}  =t t+T-1 - Y m (t)E{ [x m av – x m *(  )] | Q(t), Y(t)}  =t t+T-1 [We can thus plug in any alternative control policy in the Right Hand Side, including the one that yields the optimum time average subject to all time average constraints]

17. Proof Sketch: (Consider exact SSP for simplicity)  =t t+T-1  T (Q(t), Y(t)) + VE{ x 0 (  ) | Q(t), Y(t)} < B + VE{ x 0 *(  ) | Q(t), Y(t)}  =t t+T-1 - Q k (t)E{ [  k *(  ) – R k *(  )] | Q(t), Y(t)}  =t t+T-1 - Y m (t)E{ [x m av – x m *(  )] | Q(t), Y(t)}  =t t+T-1 [Note by RENEWAL THEORY, the infinite horizon time average is exactly achieved over any renewal period ]

18. Proof Sketch: (Consider exact SSP for simplicity)  =t t+T-1  T (Q(t), Y(t)) + VE{ x 0 (  ) | Q(t), Y(t)} < B + VE{ x 0 *(  ) | Q(t), Y(t)}  =t t+T-1 - Q k (t)E{ [  k *(  ) – R k *(  )] | Q(t), Y(t)}  =t t+T-1 - Y m (t)E{ [x m av – x m *(  )] | Q(t), Y(t)}  =t t+T-1 0 0 X 0 (optimum) E{T} [Note by RENEWAL THEORY, the infinite horizon time average is exactly achieved over any renewal period ]

19. Proof Sketch: (Consider exact SSP for simplicity)  =t t+T-1  T (Q(t), Y(t)) + VE{ x 0 (  ) | Q(t), Y(t)} < B + VX 0 (optimum) E{T} [Sum the resulting telescoping series to get the utility performance bound! ]

20. Implementation of Approximate Weighted SSP: Use a simple 1-step Robbins-Monro Iteration with past history Of W samples {S(t 1 ), S(t 2 ), …, S(t W )}. To avoid subtle correlations between samples and queue weights, use a Delayed Queue Analysis. Algorithm requires no a-priori knowledge of statistics, and takes roughly |Z| operations per slot to perform Robbins-Monro. Convergence and Delay are log(|Z|). For K Delay constrained queues, |Z| = B max K (geometric in K). Can modify implementation for constant per-slot complexity, but then convergence time is geometric in K. (Either way, we want K small).

21. Conclusions: Treat general Markov Decision Networks Generalize Max-Weight/Lyapunov Optimization to Min Weighted Stochastic Shortest Path (W-SSP) Can solve delay constrained network problems: Convergence Times, Delays Polynomial in (N+K) Per-Slot Computation Complexity of Solving Robbins-Monro is geometric in K. (want K small) A 1 (t) A 2 (t) A L (t) State 1 State 2 State 3 Control-Dependent Transition Probabilities