Fundamental Characteristics of Queues with Fluctuating Load (appeared in SIGMETRICS 2006) VARUN GUPTA Joint with: Mor Harchol-Balter Carnegie Mellon Univ. Alan Scheller-Wolf Carnegie Mellon Univ. Uri Yechiali Tel Aviv Univ.

2 Motivation Clients Server Farm Requests

3 Motivation Clients Server Farm Requests

4 Motivation Clients Server Farm Requests

5 Motivation Clients Server Farm Requests

6 Motivation Clients Server Farm Requests

7 Motivation Clients Server Farm Requests

8 Motivation Clients Server Farm Requests

9 Motivation Clients Server Farm Requests Real World  Fluctuating arrival and service intensities

10 A Simple Model HH LL exp(  H ) exp(  L ) High Load Low Load

11 Poisson Arrivals Exponential Job Size Distribution H /  H > L /  L H >  H possible, only need stability A Simple Model High Load Low Load  H,  H L,  L exp(  ) H HH L LL

12 The Markov Chain Phase Number of jobs L H H HH  LL L  H HH LL L...  Solving the Markov chain provides no behavioral insight

13  H HH L LL N = Number of jobs in the fluctuating load system Lets try approximating N using (simpler) non- fluctuating systems

14  H HH L LL Method 1 N mix

15 H HH L LL Q: Is N mix ≈ N? A: Only when   0 Method 1 N mix ½½ ½½ +   ,

16  H HH L LL Method 2

17 avg( H, L ) avg(  H,  L ) Method 2 ≡ N avg Q: Is N avg ≈ N? A: When      ,

18 Example    H =1, H =0.99  L =1, L =0.01 E[N mix ] ≈ 49.5 E[N avg ] = 1 00 

19 Observations Fluctuating system can be worse than non- fluctuating   0 and    asymptotes can be very far apart E[N mix ] > E[N avg ] E[N mix ]  E[N avg ]

20 Questions Is fluctuation always bad? Is E[N] monotonic in  ? Is there a simple closed form approximation for E[N] for intermediate  ’s? How do queue lengths during High Load and Low Load phase compare? How do they compare with N avg ? More than 40 years of research has not addressed such fundamental questions!

21 Outline  Is E[N mix ] ≥ E[N avg ], always?  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

22 Prior Work Fluid/Diffusion Approximations Transforms Matrix Analytic & Spectral Analysis - P. Harrison - Adan and Kulkarni Numerical Approaches Involves solution of cubic - Clarke - Neuts - Yechiali and Naor Involves solution of cubic - Massey - Newell - Abate, Choudhary, Whitt Limiting Behavior But cubic equations have a close form solution… ?

23 Good luck understanding this!

24 Asymptotics for E[N] ( H <  H ) E[N avg ] E[N mix ] E[N]  (switching rate) High fluctuation    H =1, H =0.99  L =1, L =0.01 E[N mix ] > E[N avg ]  Low fluctuation

25 Asymptotics for E[N] ( H <  H )  E[N] E[N mix ] E[N avg ] Agrees with our example (  H =  L ) Ross’s conjecture for systems with constant service rate: “Fluctuation increases mean delay” Q: Is this behavior possible? A: Yes  E[N] E[N avg ] E[N mix ]

26 Our Results  E[N] (  H - H ) > (  L - L ) (  H - H ) = (  L - L ) (  H - H ) < (  L - L ) Define the slacks during L and H as s L =  L - L s H =  H - H  E[N] 

27 Our Results Define the slacks during L and H as s L =  L - L s H =  H - H Not load but slacks determine the response times! s H > s L s H = s L s H < s L KEY IDEA  E[N]  

28 Outline  Is E[N mix ] ≥ E[N avg ], always?  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

29 Outline Is E[N mix ] ≥ E[N avg ], always? No  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

30 Monotonicity of E[N]  : Mean Queue Length H L  ’ : Mean Queue Length H L

31 Monotonicity of E[N] We show : E[N] is monotonic in   : Mean Queue Length H L  ’ : Mean Queue Length H L Not obvious that true for all ,  ’ with  ’<  !

32 Outline Is E[N mix ] ≥ E[N avg ], always? No  Is E[N] monotonic in  ?  Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

33 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes  Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

34 Approximating E[N] Express the first moment as * E[N] = E[N mix ]r+E[N avg ](1-r) Approximate r by the root of a quadratic KEY IDEA * True for H <  H ; a similar expression exists for case of transient overload

35 Approximating E[N]  E[N] Exact Approx.  H =  L =1, H =0.95, L =0.2

36 Approximating E[N]  E[N] Exact Approx.  H =  L =1, H =0.95, L =0.2

37 Approximating E[N]  Exact Approx.  H =  L =1, H =1.2, L = E[N]

38 Approximating E[N]  Exact Approx.  H =  L =1, H =1.2, L = E[N]

39 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes  Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

40 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

41 Stochastic Ordering refresher For random variables X and Y X  st Y  Pr{X  i}  Pr{Y  i} for all i. X  st Y  E[f(X)]  E[f(Y)] for all increasing f –E[X k ]  E[Y k ] for all k  0.

42 Notation N H : Number of jobs in system during H phase N L : Number of jobs in system during L phase N = (N H +N L )/2 H,  H L,  L exp(  ) NHNH NLNL

43 Stochastic Orderings for N L, N H N L ≥ st N M/M/1/L N H ≤ st N M/M/1/H N H ≥ st N L N H ≥ st N avg N L  st N avg ? ? ? ? ? H,  H L,  L exp(  ) NHNH NLNL N H increases stochastically as  ↓ Conjecture:

44 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes Simple closed form approximation for E[N]  Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

45 Outline Is E[N mix ] ≥ E[N avg ], always? No Is E[N] monotonic in  ? Yes Simple closed form approximation for E[N] Stochastic orderings for the number of jobs seen by an arrival into an H phase, L phase  Application: Capacity Planning

46 Scenario Application: Capacity Provisioning  H HH L LL  2 H HH 2 L LL Aim: To keep the mean response times same

47 Scenario Application: Capacity Provisioning  H HH L LL  2 H 2H2H 2 L 2L2L Question: What is the effect of doubling the arrival and service rates on the mean response time?

48 What happens to the mean response time when,  are doubled in the fluctuating load queue? Halves Remains almost the sameReduces by less than half Reduces by more than half  A:  D:  C:  B:

49 What happens to the mean response time when,  are doubled in the fluctuating load queue? Halves Remains almost the sameReduces by less than half Reduces by more than half  A:  D:  C:  B:

50 What happens to the mean response time when,  are doubled in the fluctuating load queue? Halves Remains almost the sameReduces by less than half Reduces by more than half  A:  D:  C:  B: Look at slacks! A: s H = s L B: s H > s L C: s H < s L D: s H < 0,   0  reduces by half  more than half  less than half  remains same

51 Our Contributions Give a simple characterization of the behavior of E[N] vs.  Provide simple (and tight) quadratic approximations for E[N] Prove the first stochastic ordering results for the fluctuating load model

52 Thank you

53 Analysis of E[N] First steps: –Note that it suffices to look at switching points –Express N L = f(N H ) N H = g(N L ) –The problem reduces to finding  Pr{N H =0} and Pr{N L =0} H,  H L,  L NHNH NLNL N L =f(g(N L )) f g

54 –Find the root  of a cubic (the characteristic matrix polynomial in the Spectral Expansion method) –Express E[N] in terms of  E[N] = The simple way forward… H,  H L,  L f g A  A - A  H (  L - L )  0 H +  L (  H - H )  0 L - (  L - L )(  H - H ) 2  (  A - A ) + Where  0 L =  0 H =  (  A - A )   L (  -1)(  H  - H )  (  A - A )   H (  -1)(  L  - L ) NHNH NLNL Difficult to even prove the monotonicity of E[N] wrt  using this!

55 Our approach (contd.) Express the first moment as E[N] = f 1 (  )r+f 0 (  )(1-r) –r is the root of a (different) cubic –r  1 as  0 and r  0 as  KEY IDEA

56 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r 

57 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r  Need at least 3 roots for  when r=c 1 but  has at most 2 roots c1c1

58 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r  Need at least 2 positive roots for  when r=c 2 but for r>1 product of roots is negative c2c2

59 Monotonicity of E[N] E[N] = f 1 (  )r+f 0 (  )(1-r) r is monotonic in   E[N] is monotonic in  The cubic for r has maximum power of  as r  E[N] is monotonic in  !

60 Why do slacks matter? Fact: The mean response time in an M/M/1 queue is (  - ) -1 –Higher slacks  Lower mean response times What is the fraction of customers departing during H H,  H L,  L exp(  ) when  ? H,  H L,  L exp(  ) when  0? HH  H +  L H H + L ?

61 Why do slacks matter? when  ? H,  H L,  L exp(  ) when  0? HH  H +  L H H + L ? Fact: The mean response time in an M/M/1 queue is (  - ) -1 –Higher slacks  Lower mean response times What is the fraction of customers departing during H H,  H L,  L exp(  )

62 Why do slacks matter? when  ? H,  H L,  L exp(  ) when  0? AA HH ?  Fact: The mean response time in an M/M/1 queue is (  - ) -1 –Higher slacks  Lower mean response times What is the fraction of customers departing during H As switching rates decrease, larger fraction of customers experience lower mean response times when s H >s L H,  H L,  L exp(  )

63 Q: What happens to E[N] when we double ’s and  ’s? A: System A:, ,  System B: 2, 2 ,  ?

64 Q: What happens to E[N] when we double ’s and  ’s? A: System A:, ,  System B: 2, 2 ,  System C: 2, 2 , 2  E[N] remains same in going from A to C A) s L = s H : remains same B) s L > s H : increases, but by less than twice C) s L < s H : decreases D)  0,  H >1 : queue lengths become twice as switching rates halve, E[N] doubles