1. 1 Mor Harchol-Balter Carnegie Mellon University School of Computer Science

2. 2 “size” = service requirement load  < 1 Q: Which minimizes mean response time?

3. 3 “size” = service requirement jobs SRPT jobs load  < 1 jobs PS FCFS Q: Which best represents scheduling in web servers ?

4. 4 IDEA How about using SRPT instead of PS in web servers? Linux 0.S. WEB SERVER (Apache) client 1 client 2 client 3 “Get File 1” “Get File 2” “Get File 3” Internet

5. 5 Many servers receive mostly static web requests. “GET FILE” For static web requests, know file size Approx. know service requirement of request. Immediate Objections 1) Can’t assume known job size 2) But the big jobs will starve...

6. 6 Outline of Talk [BH – Sigmetrics 01] “Analysis of SRPT: Investigating Unfairness” [HSW-Performance 02] “Asymptotic Convergence of Scheduling Policies…” [WH – Sigmetrics 03*] “Classifying Scheduling Policies wrt Unfairness …” THEORY IMPLEMENT www.cs.cmu.edu/~harchol/ [HSBA – TOCS 03] “Size-based Scheduling to Improve Web Performance” [SH – ITC 03*] “Web servers under overload: How scheduling can help” [MSAH – ICDE03] “Priority Mechanisms for OLTP and Web Applications” (M/G/1) Schroeder Wierman

7. 7 THEORY SRPT has a long history... 1966 Schrage & Miller derive M/G/1/SRPT response time: 1968 Schrage proves optimality 1979 Pechinkin & Solovyev & Yashkov generalize 1990 Schassberger derives distribution on queue length BUT WHAT DOES IT ALL MEAN?

8. 8 THEORY SRPT has a long history (cont.) 1990 - 97 7-year long study at Univ. of Aachen under Schreiber SRPT WINS BIG ON MEAN! 1998, 1999 Slowdown for SRPT under adversary: Rajmohan, Gehrke, Muthukrishnan, Rajaraman, Shaheen, Bender, Chakrabarti, etc. SRPT STARVES BIG JOBS! Various o.s. books: Silberschatz, Stallings, Tannenbaum: Warn about starvation of big jobs... Kleinrock’s Conservation Law: “Preferential treatment given to one class of customers is afforded at the expense of other customers.”

9. 9 Unfairness Question SRPT PS ? ? Let  =0.9. Let G: Bounded Pareto(  = 1.1, max=10 10 ) Question: Which queue does biggest job prefer? M/G/1

10. 10 Results on Unfairness Let  =0.9. Let G: Bounded Pareto(  = 1.1, max=10 10 ) SRPT PS I SRPT

11. 11 Unfairness – General Distribution All-can-win-theorem: For all distributions, if   ½, E[T(x)] SRPT  E[T(x)] PS for all x.

12. 12 All-can-win-theorem: For all distributions, if   ½, E[T(x)] SRPT  E[T(x)] PS for all x. Proof idea:    x t dt 0 )1   )   x xFx 2 2 ))1(2 (  0  x dttft 2 )( Waiting time (SRPT)Residence (SRPT)Total (PS)

13. 13 Classification of Scheduling Policies ALWAYS FAIR For all loads, for all service distributions, ALWAYS UNFAIR For all loads, for all service distributions, SOMETIMES UNFAIR For some loads: For other loads : PSP xTExTEx)]([ ([,   PSP xxTExTE,)]([ ([   PSP xTExTEx)]([ ([,   PSP xxTExTE,)]([ ([  

14. 14 Classification of Scheduling Policies Always FAIR Always Unfair Sometimes Unfair Age- Based Policies Preemptive Size-based Policies Remaining Size-based Policies Non- preemptive PS PLCFS FB PSJF LRPT FCFS LJF SJF SRPT Lots of open problems… FSP

15. 15 What does SRPT mean within a Web server? Many devices: Where to do the scheduling? No longer one job at a time. IMPLEMENT From theory to practice:

16. 16 Server’s Performance Bottleneck IMPLEMENT 5 Linux 0.S. WEB SERVER (Apache) client 1 client 2 client 3 “Get File 1” “Get File 2” “Get File 3” Rest of Internet ISP Site buys limited fraction of ISP’s bandwidth We model bottleneck by limiting bandwidth on server’s uplink.

17. 17 Network/O.S. insides of traditional Web server Sockets take turns draining --- FAIR = PS. Web Server Socket 1 Socket 3 Socket 2 Network Card Client1 Client3 Client2 BOTTLENECK IMPLEMENT

18. 18 Network/O.S. insides of our improved Web server Socket corresponding to file with smallest remaining data gets to feed first. Web Server Socket 1 Socket 3 Socket 2 Network Card Client1 Client3 Client2 priority queues. 1 st 2 nd 3 rd S M L BOTTLENECK IMPLEMENT

19. 19 Experimental Setup Implementation SRPT-based scheduling: 1) Modifications to Linux O.S.: 6 priority Levels 2) Modifications to Apache Web server 3) Priority algorithm design. Linux 0.S. 1 2 3 APACHE WEB SERVER Linux 1 2 3 200 Linux 1 2 3 200 Linux 1 2 3 200 switch WAN EMU

20. 20 Experimental Setup APACHE WEB SERVER Linux 0.S. 1 2 3 Linux 1 2 3 200 Linux 1 2 3 200 Linux 1 2 3 200 switch WAN EMU Trace-based workload: Number requests made: 1,000,000 Size of file requested: 41B -- 2 MB Distribution of file sizes requested has HT property. Flash Apache WAN EMU Geographically- dispersed clients 10Mbps uplink 100Mbps uplink Surge Trace-based Open system Partly-open Load < 1 Transient overload + Other effects: initial RTO; user abort/reload; persistent connections, etc.

21. 21 Preliminary Comments Job throughput, byte throughput, and bandwidth utilization were same under SRPT and FAIR scheduling. Same set of requests complete. No additional CPU overhead under SRPT scheduling. Network was bottleneck in all experiments. APACHE WEB SERVER Linux 0.S. 1 2 3 Linux 1 2 3 200 Linux 1 2 3 200 Linux 1 2 3 200 switch WAN EMU

22. 22 Load FAIR SRPT Mean Response Time (sec) Results: Mean Response Time (LAN)......

23. 23 Percentile of Request Size Mean Response time (  s) FAIR SRPT Load =0.8 Mean Response Time vs. Size Percentile (LAN)

24. 24 Transient Overload      

25. 25 Transient Overload - Baseline Mean response time SRPT FAIR

26. 26 Transient overload Response time as function of job size small jobs win big! big jobs aren’t hurt! FAIR SRPT WHY?

27. 27 Baseline Case WAN propagation delays WAN loss Persistent Connections Initial RTO value SYN Cookies User Abort/Reload Packet Length Realistic Scenario WAN loss + delay RTT: 0 – 150 ms Loss: 0 – 15% RTT: 0 – 150 ms, 0 – 10 requests/conn. RTO = 0.5 sec – 3 sec ON/OFF Abort after 3 – 15 sec, with 2,4,6,8 retries. Packet length = 536 – 1500 Bytes RTT = 100 ms; Loss = 5%; 5 requests/conn., RTO = 3 sec; pkt len = 1500B; User aborts After 7 sec and retries up to 3 times. FACTORS

28. 28 Transient Overload - Realistic Mean response time FAIR SRPT

29. 29  SRPT scheduling is a promising solution for reducing mean response time seen by clients, particularly when the load at server bottleneck is high, or under transient overload conditions.  SRPT results in negligible or zero unfairness to large requests.  SRPT is easy to implement and efficient. No CPU overhead. No drop in throughput.  Results corroborated via implementation and analysis. Conclusion so far …

30. 30 More questions … STATIC web requests Everything so far in talk … DYNAMIC web requests Current work… Schroeder McWherter Schroeder Wierman

31. 31 Online Shopping Internet client 1 client 2 client 3 “buy” Web Server (eg: Apache/Linux) Database (eg: DB2, Oracle, PostgreSQL) Dynamic responses take much longer – 10sec Database is bottleneck.

32. 32 Online Shopping Internet client 1 client 2 client 3 “$$$buy$$$” “buy” Web Server (eg: Apache/Linux) Database (eg: DB2, Oracle, PostgreSQL) Goal: Prioritize requests

33. 33 Isn’t “prioritizing requests” problem already solved? Internet “$$$buy$$$” “buy” Web Server (eg: Apache/Linux) Database (eg: DB2, Oracle, PostgreSQL) No.Prior work mostly simulation or RTDBMS.

34. 34 Which resource to prioritize? “$$$buy$$$” “buy” Web Server (eg: Apache/Linux) Internet Database Disks Locks CPU(s) High-Priority clientLow-Priority client

35. 35 Q: Which resource to prioritize? “$$$buy$$$” “buy” Web Server (eg: Apache/Linux) Internet Database Disks Locks CPU(s) High-Priority clientLow-Priority client A: 2PL  Lock Queues

36. 36 What is bottleneck resource? IBM DB2 -- Lock waiting time (yellow) is bottleneck. Therefore, need to schedule lock queues to have impact. Fix at 10 warehouses #clients = 10 x #warehouses

37. 37 Why lock scheduling is hard Lock resource 1 HHLLLHHLLL Lock resource 2 NP  H may wait long time NPinherit  Speeding up L may hurt H in long run Pabort  Rollback cost + wasted work + really hurt L’s.

38. 38 Results: Implementation study of NP, NPinherit, Pabort under TPC-C workload, Shore DBMS Develop new policy POW (Preempt on Wait)

39. 39 Results: Think time Response time Pabort NPinherit Pabort NPinherit

40. 40 Results: Think time Response time Pabort NPinherit Pabort NPinherit POW: Best of both

41. 41 More work in SYNC project… DBMS (eg: DB2, Oracle) QoS Internet “$$$buy$$$” “buy” Web Server  QoS from outside the box  Scheduling the TeraGrid  Time-varying load in systems  Impact of closed versus open system models PSC SDSC NCSA

42. 42 Scheduling is a very cheap solution… No need to buy new hardware No need to buy more memory Small software modifications …with a potentially very big win in some situations. Conclusion Thank you!