1. I think your suggestion is, Can we do two things at once? Well, we’re of the view that we can walk and chew gum at the same time. —Richard Armitage, deputy secretary of state, June 2, 2004 To do two things at once is to do neither. —Publilius Syrus, Roman slave, first century B.C. Therefore… Multitask a few…..let the rest queue.

2. Computer systems are not different… Multi-Programming Level (# concurrent tasks/threads) Efficiency (useful work per second) thrashing Therefore… Server Active tasks Tasks not-yet-started Q: Max number of tasks allowed to share server? Common Soln: K* Admission Control K*

3. Presenting How to Multitask Smartly

4. 4 A Queueing-theoretic abstraction server # jobs at server (n) Total speed  (n) K* (known)

5. 5 A Queueing-theoretic abstraction PS server 1 2 K K = MPL # jobs at server (n) Total speed  (n) K* FIFO buffer The G/G/PS-MPL model General arrival process for jobs Job sizes i.i.d. ~ X Sizes unknown, distribution of X known GOAL: Find MPL (i.e. K) to minimize mean response time response time n jobs each job gets rate  (n) n  :: (known)

6. Optimal MPL= K* ? # jobs at server (n) Total speed (jobs/sec) K*=5 Exponential job sizes (C 2 = 1) Weibull job sizes (C 2 = 19) Mean response time MPL Mean response time K* OPT MPL Example Poisson(0.8) arrival process 45% improvement

7. Optimal MPL= K* ? # jobs at server (n) Total speed (jobs/sec) K*=5 Exponential job sizes (C 2 = 1) Weibull job sizes (C 2 = 19) Mean response time MPL Mean response time K* OPT MPL Example Poisson(0.8) arrival process DEPENDS! 45% improvement

8. 8 Intuition for the effect of MPL FIFO server Processor Sharing (PS)

9. 9 Intuition for the effect of MPL FIFO server Processor Sharing (PS) High job-size variability (C 2 ) High arrival rate   # jobs at server (n) Total speed  (n) K* MPL

10. 10 Intuition for the effect of MPL server FIFO Processor Sharing (PS) High job-size variability (C 2 ) High arrival rate Optimal MPL ↑ C 2 ↑    Arrival rate ↑  Optimal MPL → K* MPL

11. 11 Why is finding the optimal MPL hard? - No analysis even for the case of Poisson arrivals (M/G/PS-MPL) Hurdle 1: State dependent service rates Hurdle 2: Finite Multi- Programming-Limit Hurdle 3: General (high C 2 ) job size distribution Mean response time only depends on E[X] [Kelly 79] Transform of response time known for Exponential job size distribution [Rege Sengupta 85] Heavy traffic diffusion approximation [Zhang Zwart 09] Our work

12. 12 Where we are… GOAL: Optimal MPL for M/G/PS-MPL SUBGOAL: Approximation for M/G/PS-MPL What if the traffic rate is unknown? What if the traffic is not Poisson? GOAL #2: Traffic-oblivious dynamic MPL control policies ?

13. 13 Where we are… GOAL: Optimal MPL for M/G/PS-MPL SUBGOAL: Approximation for M/G/PS-MPL What if the traffic rate is unknown? What if the traffic is not Poisson? GOAL #2: Traffic-oblivious dynamic MPL control policies ?

14. 14 RESULT: M/G/PS-MPL approximation PS Server  (n) 1 2 K FIFO buffer Job sizes ~ X Poisson( ) Easy for X  Exp (C 2 =1)

15. 15 RESULT: M/G/PS-MPL approximation Involves only the first two moments of job size distribution Generalizes -Lee and Longton approx. for M/G/K multi-server system:  (n)= n·  -Zhang and Zwart approx. for the case  (n)=  THEOREM: The above approx. is exact for the following degenerate hyper-exponential job size distribution: - 2 degrees of freedom - Can match any E[X] and C 2 ≥ 1

16. 16 Where we are… GOAL: Optimal MPL for M/G/PS-MPL SUBGOAL: Approximation for M/G/PS-MPL What if the traffic rate is unknown? What if the traffic is not Poisson? GOAL #2: Traffic-oblivious dynamic MPL control policies ?

17. 17 Where we are… GOAL: Optimal MPL for M/G/PS-MPL SUBGOAL: Approximation for M/G/PS-MPL What if the traffic rate is unknown? What if the traffic is not Poisson? GOAL #2: Traffic-oblivious dynamic MPL control policies ?

18. 18 Static MPL policy for M/G/PS-MPL Set MPL = MPL*, where:

19. 19 Approx accurately predicts the behavior of the curve as a function of MPL MPL* decreases as arrival rate increases (Poisson(0.8) arrivals) (Poisson(0.9) arrivals) # jobs at server (n) Total speed  (n) MPL Mean Resp. Time Our approx. Weibull (C 2 =19) K* K*K* MPL* K* gives 25% worse performance than MPL* K* gives 45% worse performance than MPL* Our approx.

20. 20 Where we are… GOAL: Optimal MPL for M/G/PS-MPL SUBGOAL: Approximation for M/G/PS-MPL What if the traffic rate is unknown? What if the traffic is not Poisson? GOAL #2: Traffic-oblivious dynamic MPL control policies ?

21. 21 Where we are… GOAL: Optimal MPL for M/G/PS-MPL SUBGOAL: Approximation for M/G/PS-MPL What if the traffic rate is unknown? What if the traffic is not Poisson? GOAL #2: Traffic-oblivious dynamic MPL control policies ?

22. 22 Traffic-oblivious policies Straw man proposal 1: Choose a “robust” static MPL –Must choose MPL=K* : but suboptimal in light/moderate traffic Straw man proposal 2: Learn the arrival rate –Can’t adapt to changes on small time scale/correlations WE DEMONSTRATE: Dynamic MPL control policies which are 1. Traffic-oblivious: self-adapt to variations in the arrival process 2. Light-weight: make decisions based only on current queue length, Q(t), and current MPL, K(t) I don’t know the arrival rate!! My arrivals are not Poisson!!

23. 23 Traffic-oblivious policies (KEY STEPS) GOAL: A robust MPL-control policy π OPT STEP 1:Approximate the orig. job size distribution X ≈ _____ STEP 2:Construct candidate policies π 1 π 2 π 3 … π N STEP 3:Pick the “best” candidate policy

24. 24 Traffic-oblivious policies (KEY STEPS) GOAL: A robust MPL-control policy π OPT STEP 1:Approximate the orig. job size distribution X ≈ _____ STEP 2:Construct candidate policies π 1 π 2 π 3 … π N STEP 3:Pick the “best” candidate policy

25. 25 GOAL: Approximate the job size distribution by a tractable distribution IDEA: The H* distribution: Models many small vs. few large jobs Tractable due to Exponential branch Traffic-oblivious policies (KEY STEPS) For age/size independent policies, H* provides a good approximation for mean response time p and γ chosen to match E[X] and C 2

26. 26 Traffic-oblivious policies (KEY STEPS) GOAL: A robust MPL-control policy π OPT STEP 1:Approximate the orig. job size distribution X ≈ H* STEP 2:Construct candidate policies π 1 π 2 π 3 … π N STEP 3:Pick the “best” candidate policy

27. 27 Traffic-oblivious policies (KEY STEPS) GOAL: A robust MPL-control policy π OPT STEP 1:Approximate the orig. job size distribution X ≈ H* STEP 2:Construct candidate policies π 1 π 2 π 3 … π N STEP 3:Pick the “best” candidate policy

28. 28 GOAL: A family of candidate dynamic policies IDEA: 1. Assume arrival ~ Poisson( ) 2. Solve for 3. Obtain the family by varying Traffic-oblivious policies (KEY STEPS) Optimal Dynamic MPL policy to minimize mean response time under H* job sizes  Intuitively : What should this look like?

29. 29 Structure of Current Queue Length Current MPL MPL ↓ MPL ↑ K*= obtained by combining policy iteration with some new tricks to handle infinite state space “Stochastic” component“Fluid” = Target MPL

30. 30 Traffic-oblivious policies (KEY STEPS) GOAL: A robust MPL-control policy π OPT STEP 1:Approximate the orig. job size distribution X ≈ H* STEP 2:Construct candidate policies π(λ 1 ) π(λ 2 ) … π(λ N ) STEP 3:Pick the “best” candidate policy

31. 31 Traffic-oblivious policies (KEY STEPS) GOAL: A robust MPL-control policy π OPT STEP 1:Approximate the orig. job size distribution X ≈ H* STEP 2:Construct candidate policies π(λ 1 ) π(λ 2 ) … π(λ N ) STEP 3:Pick the “best” candidate policy

32. 32 GOAL: Robustly choose a member from family IDEA: Minimize the worst-case relative deviation in an adversarial setting Traffic-oblivious policies (KEY STEPS) a perf. ratio( , )  Mean resp. time of policy  under Poisson( ) Mean resp. time of OPT static MPL under Poisson( ) a* = argmin max perf. ratio(  (a), ) - You pick  - Adversary sees , picks - You lose perf. ratio( , ) - Optimize against worst adversary

33. 33 Traffic-oblivious policies (KEY STEPS) GOAL: A robust MPL-control policy π OPT STEP 1:Approximate the orig. job size distribution X ≈ H* STEP 2:Construct candidate policies π(λ 1 ) π(λ 2 ) … π(λ N ) STEP 3:Pick the “best” candidate policy min max OPT

34. 34 Back to picking optimal MPL… Scenario #1 Arrival process ~ Poisson(?) Scenario #2 Arrival process ~ ?(?) (results for Batch Poisson arrivals) Worst Perf. Loss (vs. OPT Static MPL) K* Our Dynamic Policy K* Our Dynamic Policy - Dynamic policies remain “robust” to non-Poisson arrival process - Static policies become worse 45%20% 14%50%

35. 35 SUMMARY Efficiency ≠ Min. response time Static MPL policies via the first approximation for M/G/PS-MPL with state dependent service rates First traffic-oblivious MPL control policies for resource sharing systems See paper for more results MPL* MPL ↓ MPL ↑

36. 36 True arrival rate ( )

37. 37 True arrival rate ( )

38. 38 Max. dev. < 14% True arrival rate ( )

39. 39 vs. (Simulation results for Weibull job size dist.) rel. dev. Mean arrival rate Poisson arrivalsBatch Poisson (BPP) arrivals Dynamic policies remain robust to non-Poisson arrival processes Traffic-oblivious dynamic policies Traffic-aware static policies