1. Delay Reduction via Lagrange Multipliers in Stochastic Network Optimization Longbo Huang Michael J. Neely EE@USC WiOpt 2009 1 *Sponsored in part by NSF Career CCF and DARPA IT-MANET Program

2. Problem formulation Backlog behavior under Quadratic Lyapunov function based Algorithm (QLA): an example General backlog behavior result of QLA for general SNO problems The Fast-QLA algorithm (FQLA) Simulation results Summary Outline 2

3. Problem Description: A Network of r Queues Slotted Time, t=0,1,2,… S(t) = Network State, Time-Varying, IID over slots (e.g. channel conditions, random arrivals, etc.) x(t) = Control Action, chosen in some abstract set X (S(t)) (e.g. power/bandwidth allocation, routing) (S(t), x(t)) costs: f(t)=f(S(t), x(t)) generates: A j (t)=g j (S(t), x(t)) packets to queue j serves: μ j (t)=b j (S(t), x(t)) packets in queue j 3 The stochastic problem: minimize: time average cost subject to: queue stability. [f(), g(), b() are only assumed to be non-negative, continuous, bounded]

4. Slotted Time, t=0,1,2,… S(t) = Network State, Time-Varying, IID over slots (e.g. channel conditions, random arrivals, etc.) x(t) = Control Action, chosen in some abstract set X (S(t)) (e.g. power/bandwidth allocation, routing) (S(t), x(t)) costs: f(t)=f(S(t), x(t)) generates: A j (t)=g j (S(t), x(t)) packets to queue j serves: μ j (t)=b j (S(t), x(t)) packets in queue j 4 The stochastic problem: minimize: time average cost subject to: queue stability. QLA achieves: [G-N-T FnT 06] Avg. cost: f av <= f* av + O(1/V) Avg. Backlog: U av <= O(V) [f(), g(), b() are only assumed to be non-negative, continuous, bounded] Problem Description: A Network of r Queues

5. 5 An Energy Minimization Example: The QLA algorithm U1 U2 S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 R(t) μ 1 (t) μ 2 (t) μ 3 (t) μ 4 (t) μ 5 (t) The QLA algorithm (built on Backpressure): 1. Compute the differentiable backlog W ii+1 (t)=max[U i (t)-U i+1 (t), 0], 2. Choose (P 1 (t), …P 5 (t) that maximizes: Σ i [W ii+1 (t)μ i (P i (t)) -VP i (t)] =Σ i [W ii+1 (t) S i (t) –V]P i (t) e.g., if S 2 (t)=2, then if W 23 (t)*2>V, we set P 2 (t)=1. Link 2->3 Goal: allocate power to support the flow with min avg. energy expenditure, i.e.: Min: avg. Σ i P i s.t. Queue stability W 23 (t) U2 U3

6. 6 An Energy Minimization Example: Backlog under QLA U1 U2 S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 R(t) μ 1 (t) μ 2 (t) μ 3 (t) μ 4 (t) μ 5 (t) Queue snapshot under QLA with V=100, first 100 slots: U1U1 U2U2 U3U3 U4U4 U5U5 Goal: Min: avg. Σ i P i s.t. Queue stability size time

7. 7 An Energy Minimization Example: Backlog under QLA U1 U2 S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 R(t) μ 1 (t) μ 2 (t) μ 3 (t) μ 4 (t) μ 5 (t) Queue snapshot under QLA with V=100, first 500 slots: U1U1 U2U2 U3U3 U4U4 U5U5 size time Goal: Min: avg. Σ i P i s.t. Queue stability

8. 8 An Energy Minimization Example: Backlog under QLA U1 U2 S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 R(t) μ 1 (t) μ 2 (t) μ 3 (t) μ 4 (t) μ 5 (t) Queue snapshot under QLA with V=100, first 1000 slots: U1U1 U2U2 U3U3 U4U4 U5U5 size time Goal: Min: avg. Σ i P i s.t. Queue stability

9. 9 An Energy Minimization Example: Backlog under QLA U1 U2 S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 R(t) μ 1 (t) μ 2 (t) μ 3 (t) μ 4 (t) μ 5 (t) Queue snapshot under QLA with V=100, first 5000 slots: U1U1 U2U2 U3U3 U4U4 U5U5 size time Goal: Min: avg. Σ i P i s.t. Queue stability

10. 10 An Energy Minimization Example: Backlog under QLA U1 U2 S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 R(t) μ 1 (t) μ 2 (t) μ 3 (t) μ 4 (t) μ 5 (t) Queue snapshot under QLA with V=100, (U1(t), U2(t)): (500,400) t=1:500k Goal: Min: avg. Σ i P i s.t. Queue stability

11. 11 An Energy Minimization Example: Backlog under QLA U1 U2 S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 R(t) μ 1 (t) μ 2 (t) μ 3 (t) μ 4 (t) μ 5 (t) t=1:500kt=5k:500k Queue snapshot under QLA with V=100, (U1(t), U2(t)): (500,400) Goal: Min: avg. Σ i P i s.t. Queue stability

12. General result: Backlog under QLA 12 Implications: (1) Delay under QLA is Θ(V), not just O(V); (2) The network stores a backlog vector ≈U V *. Theorem 1: If q(U) satisfies C1: for some L>0 independent of V, then under QLA, in steady state, U(t) is mostly within O(log(V)) distance from U V * = Θ(V).

13. General result: Backlog under QLA 13 Implications: (1) Delay under QLA is Θ(V), not just O(V); (2) The network stores a backlog vector ≈U V *. Let’s “subtract out” U V * from the network! Replace most of the U V * data with Place-Holder bits Theorem 1: If q(U) satisfies C1: for some L>0 independent of V, then under QLA, in steady state, U(t) is mostly within O(log(V)) distance from U V * = Θ(V).

14. Fast-QLA (FQLA): Using place-holder bits A single queue example: 14 Start here First idea: (1) choose number of place- holder bits Q, s.t., if U(t 0 )>=Q, then U(t)>=Q for all t>=t 0. (2) Let U(0)=Q, run QLA.

15. Fast-QLA (FQLA): Using place-holder bits A single queue example: 15 Start here reduced Advantage: delay reduced by Q, same utility performance. First idea: (1) choose number of place- holder bits Q, s.t., if U(t 0 )>=Q, then U(t)>=Q for all t>=t 0. (2) Let U(0)=Q, run QLA. actual backlog

16. Fast-QLA (FQLA): Using place-holder bits A single queue example: 16 Start here actual backlog ≈Θ(V) Advantage: delay reduced by Q, same utility performance. Problem: Q ≈ U V *-Θ(V), delay Θ(V). reduced First idea: (1) choose number of place- holder bits Q, s.t., if U(t 0 )>=Q, then U(t)>=Q for all t>=t 0. (2) Let U(0)=Q, run QLA.

17. Fast-QLA (FQLA): Using place-holder bits A single queue example: 17 FQLA idea: Choose # of place-holder bits Q such that backlog under QLA rarely goes below Q. Problem: (1) U(t) will eventually get below Q, what to do? (2) How to ensure utility performance?

18. Fast-QLA (FQLA): Using place-holder bits A single queue example: 18 FQLA idea: Choose # of place-holder bits Q such that backlog under QLA rarely goes below Q. Problem: (1) U(t) will eventually get below Q, what to do? (2) How to ensure utility performance? Answer: use virtual backlog process W(t) + careful pkt dropping

19. Fast-QLA (FQLA): Using place-holder bits A single queue example: 19 FQLA: (1) Choose # of place-holder bits Q such that backlog under QLA rarely goes below Q. (2) Use a virtual backlog process W(t) with W(0)=Q to track the backlog that should have been generated by QLA. (3) Obtain action by running QLA based on W(t), modify the action carefully.

20. Fast-QLA (FQLA): Using place-holder bits A single queue example: 20 Modifying the action If W(t)>=Q, same as QLA, admit A(t), serve μ(t), i.e., FQLA=QLA. If W(t)<Q, serve μ(t), only admit: A’(t)=max[A(t)-(Q-W(t)), 0]. This modification ensures: U(t) ≈ max[W(t)-Q, 0].

21. Fast-QLA (FQLA): Using place-holder bits A single queue example: 21 Modifying the action If W(t)>=Q, same as QLA, admit A(t), serve μ(t), i.e., FQLA=QLA. If W(t)<Q, serve μ(t), only admit: A’(t)=max[A(t)-(Q-W(t)), 0]. This modification ensures: U(t) ≈ max[W(t)-Q, 0].

22. Fast-QLA (FQLA): Using place-holder bits A single queue example: 22 Modifying the action If W(t)>=Q, same as QLA, admit A(t), serve μ(t), i.e., FQLA=QLA. If W(t)<Q, serve μ(t), only admit: A’(t)=max[A(t)-(Q-W(t)), 0]. This modification ensures: U(t) ≈ max[W(t)-Q, 0].

23. Fast-QLA (FQLA): Using place-holder bits A single queue example: 23 Modifying the action If W(t)>=Q, same as QLA, admit A(t), serve μ(t), i.e., FQLA=QLA. If W(t)<Q, serve μ(t), only admit: A’(t)=max[A(t)-(Q-W(t)), 0]. This modification ensures: U(t) ≈ max[W(t)-Q, 0].

24. Fast-QLA (FQLA): Using place-holder bits A single queue example: 24 Modifying the action If W(t)>=Q, same as QLA, admit A(t), serve μ(t), i.e., FQLA=QLA. If W(t)<Q, serve μ(t), only admit: A’(t)=max[A(t)-(Q-W(t)), 0]. This modification ensures: U(t) ≈ max[W(t)-Q, 0].

25. Fast-QLA (FQLA): Using place-holder bits A single queue example: 25 Modifying the action If W(t)>=Q, same as QLA, admit A(t), serve μ(t), i.e., FQLA=QLA. If W(t)<Q, serve μ(t), only admit: A’(t)=max[A(t)-(Q-W(t)), 0]. This modification ensures: U(t) ≈ max[W(t)-Q, 0].

26. Fast-QLA (FQLA): Using place-holder bits A single queue example: 26 Now choose: Q=max[U V *-log 2 (V), 0] (1) ensures: Low delay: average U ≈ log 2 (V), (2) ensures W(t) rarely below Q, implying: Good utility & few pkt dropped: very few action modifications. Modifying the action If W(t)>=Q, same as QLA, admit A(t), serve μ(t), i.e., FQLA=QLA. If W(t)<Q, serve μ(t), only admit: A’(t)=max[A(t)-(Q-W(t)), 0]. This modification ensures: U(t) ≈ max[W(t)-Q, 0].

27. Fast-QLA (FQLA): Performance Theorem 2: If condition C1 in Theorem 1 holds, then we have under FQLA-Ideal: 27 Recall: under QLA:

28. 28 Simulation U1 U2 R(t) S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 Simulation parameters: - V= 50, 100, 200, 500, 1000, 2000, - Each with 5x10 6 slots, - U V *=(5V, 4V, 3V, 2V, V) T. Backlog % of pkt dropped

29. 29 Simulation U1 U2 R(t) S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 Simulation parameters: - V= 50, 100, 200, 500, 1000, 2000, - Each with 5x10 6 slots, - U V *=(5V, 4V, 3V, 2V, V) T. Sample (W 1 (t),W 2 (t)) process: V=1000, t=10000:110000 Note: W 1 (t)>Q 1 =4952 & W 2 (t)>Q 2 =3952

30. 30 Simulation U1 U2 R(t) S 1 (t) S 2 (t) S 3 (t) S 4 (t) S 5 (t) U3U4 U5 Backlog% of pkt dropped Quick comparison V=1000, U QLA≈15V=15000 U FQLA≈5log 2 (V)=250 60 times better! Simulation parameters: - V= 50, 100, 200, 500, 1000, 2000, - Each with 5x10 6 slots, - U V *=(5V, 4V, 3V, 2V, V) T.

31. Summary 31 1.Under QLA, the backlog vector usually stays close to an “attractor” – the optimal Lagrange multiplier U V *. 2.FQLA subtracts out the Lagrange multiplier from the system induced by QLA by using place-holder bits to reduce delay.

32. Summary Note: (1) Theorem 1 also holds when S(t) is Markovian, (2) FQLA-General for the case where U V * is not known, performance similar to FQLA-Ideal, (3) when q 0 (U) is “smooth”, we prove O(sqrt{V}) deviation bound, (4) The “Network Gravity” role of Lagrange multiplier. Details see ArXiv report 0904.3795 32 1.Under QLA, the backlog vector usually stays close to an “attractor” – the optimal Lagrange multiplier U V *. 2.FQLA subtracts out the Lagrange multiplier from the system induced by QLA by using place-holder bits to reduce delay.

33. Thank you ! Questions or Comments? 33