1. Carnegie Mellon University Computer Science Department 1 OPEN VERSUS CLOSED: A CAUTIONARY TALE Bianca Schroeder Adam Wierman Mor Harchol-Balter Computer Science Department Carnegie Mellon University OPEN VERSUS CLOSED: A CAUTIONARY TALE Bianca Schroeder Adam Wierman Mor Harchol-Balter Computer Science Department Carnegie Mellon University To appear at NSDI 2006 presenter: 吳泰廷

2. Carnegie Mellon University Computer Science Department 2 standard system new system old new new system has smaller response time! This comparison requires testing the two systems on realistic workloads THE RESEARCH PROCESS

3. Carnegie Mellon University Computer Science Department 3 INTRODUTION Need system models that “accurately represent" the real system. Representing a system accurately involves many things: bottleneck resource behavior, the scheduling of requests at that bottleneck, workload parameters such as the distribution of service request demands……. One factor that researchers typically pay little attention to is whether the job arrivals obey a closed or an open system model.

4. Carnegie Mellon University Computer Science Department 4 We show that closed and open system models yield significantly different results, even when both models are run with the same load and service demands. Conclude with guidelines for choosing a system model.

5. Carnegie Mellon University Computer Science Department 5 think receive send server CLOSED SYSTEM MODEL User requests web page, receives page, reads page, clicks on new link Closed System N=MPL (multiprogramming level)

6. Carnegie Mellon University Computer Science Department 6 1:01.12 ip1 GET a.gif HTTP/1.0 1:01.20 ip2 GET b.htm HTTP/1.0 1:01.25 ip1 GET c.jpg HTTP/1.0 1:01.27 ip1 GET d.txt HTTP/1.0 1:01.28 ip3 GET a.htm HTTP/1.0 1:01.35 ip4 GET d.gif HTTP/1.0 1:01.45 ip2 GET e.htm HTTP/1.0 : Trace driven OPEN SYSTEM MODEL service demands x x x server new arrivals arrival times file sizes from trace next arrival time from trace Open System

7. Carnegie Mellon University Computer Science Department 7 Distribution driven Use distributions of interarrival times and service demands (typically using trace info) x x x server new arrivals OPEN SYSTEM MODEL interarrival time dist. service demand dist. sample dist. sample dist. Open System

8. Carnegie Mellon University Computer Science Department 8 OPEN MODEL CLOSED MODEL Arrivals are independent of completions Arrivals are completely dependent on completions There is no max number of simultaneous users There is a fixed population of users, called the Multi-Programming-Level (MPL)

9. Carnegie Mellon University Computer Science Department 9

10. Carnegie Mellon University Computer Science Department 10 WEB WORKLOAD GENERATORS CLOSED MODEL CLOSED MODEL OPEN MODEL Surge SPECWeb TPC-W Sclient RUBiS WebBench Webjamma 1.Workload generators for the same purpose use different system models! 2.It’s often not clear which model workload generators use! Do you use an open or closed model?

11. Carnegie Mellon University Computer Science Department 11 NEITHER THE OPEN OR CLOSED MODEL IS COMPLETELY REALISTIC

12. Carnegie Mellon University Computer Science Department 12 x x x new arrivals server think send receive leave system with probability q return to the system PARTLY-OPEN MODEL PARTLY-OPEN SYSTEM

13. Carnegie Mellon University Computer Science Department 13 What is the impact of the choice of an open or closed model? OUR GOAL

14. Carnegie Mellon University Computer Science Department 14 HOW DO WE COMPARE OPEN AND CLOSED SYSTEMS? CLOSED CLOSED OPEN 1.Fix the service distribution across the systems 2.Fix the load across the systems load depends only on mean arrival rate and mean service demands load depends on MPL, think times, mean of service demands, variability of service demands … adjust load using the think time adjust load using the arrival rate

15. Carnegie Mellon University Computer Science Department 15 How do open and closed response times compare? FCFS scheduling open  Poisson arrival process closed  Exponential think times

16. Carnegie Mellon University Computer Science Department 16 load 00.250.50.75 1 mean response time 1000 100 10 FCFS scheduling open  Poisson arrival process closed  Exponential think times Open Closed (MPL=10) CLOSED << OPEN

17. Carnegie Mellon University Computer Science Department 17 load 00.250.50.75 1 mean response time 1000 100 10 Open Closed (MPL=10) Closed (MPL=100) Closed (MPL=1000) CLOSED  OPEN FCFS scheduling open  Poisson arrival process closed  Exponential think times

18. Carnegie Mellon University Computer Science Department 18 OPEN MODEL CLOSED MODEL VS CLOSED  OPEN AS MPL GROWS Schatte [36, 37] proves formally that as N grows to infinity, a closed FCFS queue converges to an open queue.

19. Carnegie Mellon University Computer Science Department 19 low variabilityhigh variability mean response time 1500 1000 500 Open Closed (MPL=10) Closed (MPL=100) Closed (MPL=1000) Web Workloads How quickly does Closed  Open?

20. Carnegie Mellon University Computer Science Department 20 There principles 1.For a given load, mean response times are significantly lower in closed systems than in open systems. 2. As the MPL grows, closed systems become open, but convergence is slow for practical purposes. 3.While variability has a large effect in open systems, the effect is much smaller in closed systems.

21. Carnegie Mellon University Computer Science Department 21 What is the impact of the choice of an open or closed model? OUR GOAL It matters a lot! 1. What is the impact on the effectiveness of scheduling? 2.What is the impact in practice?

22. Carnegie Mellon University Computer Science Department 22 FCFS (First-Come-First-Served) PS (Processor-Sharing) PESJF (Preemptive-Expected-Shortest-Job-First) SRPT (Shortest-Remaining-Processing-Time-First) PELJF (Preemptive-Expected-Longest-Job-First)

23. Carnegie Mellon University Computer Science Department 23 SCHEDULING IS A KEY COMPONENT OF SYSTEM DESIGN Improved design Shortest Remaining Processing Time (SRPT) Standard design Processor Sharing (PS) WEB SERVERS Does the effectiveness of scheduling depend on the system model (open vs. closed)? Compare using a workload generator

24. Carnegie Mellon University Computer Science Department 24 SCHEDULING IN OPEN SYSTEMS OPEN mean response time 1000 600 300 0 load 0.25.5.75 1 PLJF FCFS PS SRPT How do the closed results compare?

25. Carnegie Mellon University Computer Science Department 25 CONTRASTING THE IMPACT OF SCHEDULING OPEN CLOSED mean response time 1000 600 300 0 load 0.25.5.75 1 load PLJF FCFS PS SRPT PLJF FCFS PS SRPT

26. Carnegie Mellon University Computer Science Department 26

27. Carnegie Mellon University Computer Science Department 27 Three priciples 1.While open systems benefit significantly from scheduling with respect to response time, closed systems improve much less. 2. Scheduling only significantly improves response time in closed systems under very specific parameter settings: moderate load (think times). 3. Scheduling can limit the effect of variability in both open and closed systems.

28. Carnegie Mellon University Computer Science Department 28 What is the impact of the choice of an open or closed model? OUR GOAL It matters a lot! Especially when evaluating scheduling policies What is the impact in practice?

29. Carnegie Mellon University Computer Science Department 29 OPEN VS CLOSED IN PRACTICE 3 CASE STUDIES 1.Serving static web content 2.Database backend of an e-commerce site 3. Auctioning web site testbed implementation trace-based simulation

30. Carnegie Mellon University Computer Science Department 30 Case study Open generator Closed generator Scheduling policies Static web (LAN) Sclient on World Cup trace Modified Sclient on World Cup trace PS, SRPT E-commerceModified TPC-WTPC-WPS, PESJF Auctioning Trace-based simulation (top 10 auction site trace) PS, SRPT

31. Carnegie Mellon University Computer Science Department 31 OPEN VS CLOSED IN PRACTICE OPENCLOSED mean response time PS SRPT PS SRPT 300 200 100 load 0.25.5.75 1 load STATIC WEB SERVER Different models give different conclusion about benefits of SRPT MPL=50

32. Carnegie Mellon University Computer Science Department 32 OPENCLOSED PS mean response time 10 8 4 0 load PS SRPT PS SRPT 20 14 7 0 load 0.25.5.75 1 load PESJF E-COMMERCE SITE AUCTION SITE MPL=50

33. Carnegie Mellon University Computer Science Department 33 What is the impact of the choice of an open or closed model? OUR GOAL TODAY It matters a lot in practice! Especially when evaluating scheduling policies How can we identify whether to use an open or closed model?

34. Carnegie Mellon University Computer Science Department 34 A MORE REALISTIC ALTERNATIVE x x x new arrivals server think send receive leave system with probability q return to the system PARTLY-OPEN MODEL What parameters affect the load? Does think time affect the load? How do think times affect response times?

35. Carnegie Mellon University Computer Science Department 35 FITTING A PARTLY-OPEN MODEL 12 ip1 GET a.gif HTTP/1.0 20 ip2 GET b.htm HTTP/1.0 25 ip1 GET c.jpg HTTP/1.0 27 ip1 GET d.txt HTTP/1.0 28 ip3 GET a.htm HTTP/1.0 35 ip4 GET d.gif HTTP/1.0 45 ip2 GET e.htm HTTP/1.0 : Trace service demands file sizes from trace PARTLY-OPEN PARTLY-OPEN

36. Carnegie Mellon University Computer Science Department 36 FITTING A PARTLY-OPEN MODEL 12 ip1 GET a.gif HTTP/1.0 20 ip2 GET b.htm HTTP/1.0 25 ip1 GET c.jpg HTTP/1.0 27 ip1 GET d.txt HTTP/1.0 28 ip3 GET a.htm HTTP/1.0 35 ip4 GET d.gif HTTP/1.0 45 ip2 GET e.htm HTTP/1.0 : Trace PARTLY-OPEN PARTLY-OPEN Fitting the interarrival times Distinguish users e.g. use ip address in a web trace Identify user session boundaries  Use periods of inactivity of length > timeout

37. Carnegie Mellon University Computer Science Department 37 CHOOSING A TIMEOUT VALUE Number of sessions 2e5 1e5 0 0 30min Timeout length financial world cup dept store

38. Carnegie Mellon University Computer Science Department 38 THE EFFECT OF THINK TIME STATIC WEB SERVER 1 10 100 1000 mean think time 300 200 100 0 mean response time SRPT PS

39. Carnegie Mellon University Computer Science Department 39 CLOSED OPEN q1q1 q0q0 x x x new arrivals server think send receive leave system with probability q return to the system PARTLY-OPEN MODEL number of requests per visit ↑ number of requests per visit ↓ ?? A MORE REALISTIC ALTERNATIVE Workload generators are only Open/Closed!

40. Carnegie Mellon University Computer Science Department 40 THE TRANSITION FROM OPEN  CLOSED STATIC WEB SERVER 300 200 100 0 0 5 10 15 20 PS open PS closed PS SRPT mean response time mean number of requests per visit OPEN CLOSED

41. Carnegie Mellon University Computer Science Department 41 THE PARTLY-OPEN SYSTEM IN PRACTICE mean number of requests per visit mean response time STATIC WEB PS SRPT 0 5 10 15 20 200 100 0 E-COMMERCE SITE 0 5 10 15 20 96309630 PS PESJF AUCTIONING 0 5 10 15 20 15 10 5 0 PS SRPT

42. Carnegie Mellon University Computer Science Department 42 PS SRPT PS SRPT OPEN CLOSED PS SRPTPARTLY-OPENVS THESE DIFFERENCES ARE IMPORTANT IN PRACTICE

43. Carnegie Mellon University Computer Science Department 43 Two Principles 1.A partly-open system behaves similarly to an open system when the expected number of requests per session is small (≤ 5) and similarly to a closed system when the expected number of requests per session is large (≥ 10 as a rule-of-thumb). 2.In a partly-open system, think time has little effect on mean response time.

44. Carnegie Mellon University Computer Science Department 44 CHOOSING A SYSTEM MODEL Web workloads Open or closed? Use a partly-open model... 1. Large corporate web 2. CMU web server 3. Online department store 4. Science institute (USGS) 5. Online gaming site 6. Financial service provider 7. Supercomputing web site 8. Kasparov-DeepBlue match 9. Site seeing “slashdot effect” 10. Soccer world cup

45. Carnegie Mellon University Computer Science Department 45 CHOOSING A SYSTEM MODEL Web workloads Open or closed? Use a partly-open model......to decide which is more accurate 1. Large corporate web 2. CMU web server 3. Online department store 4. Science institute (USGS) 5. Online gaming site 6. Financial service provider 7. Supercomputing web site 8. Kasparov-DeepBlue match 9. Site seeing “slashdot effect” 10. Soccer world cup

46. Carnegie Mellon University Computer Science Department 46 HOW TO CHOOSE A SYSTEM MODEL Gather a trace How many simult. users are there? Fit a partly open model to the trace OPEN ≈ CLOSED >>1000 else What is the expected num. of visits? OPENCLOSED??? <55-10 >10 Mean num. of visits 15 10 5 0 0 30min Timeout length world cup dept store financial

47. Carnegie Mellon University Computer Science Department 47

48. Carnegie Mellon University Computer Science Department 48

49. Carnegie Mellon University Computer Science Department 49 CHOOSING A SYSTEM MODEL <5 expected visits >10 expected visits CLOSED 5-10 expected visits Web Workloads OPEN PARTLY OPEN 1. Large corporate web 2. CMU web server 3. Online department store 4. Science institute (USGS) 5. Online gaming site 6. Financial service provider 7. Supercomputing web site 8. Kasparov-DeepBlue match 9. Site seeing “slashdot effect” 10. Soccer world cup

50. Carnegie Mellon University Computer Science Department 50 CHOOSING A SYSTEM MODEL <5 expected visits 1. Large corporate web 2. CMU web server 4. Science institute (USGS) 6. Financial service provider 8. Kasparov-DeepBlue match 9. Site seeing “slashdot effect” >10 expected visits 5. Online gaming site 10. Soccer world cup CLOSED 5-10 expected visits 3. Online department store 7. Supercomputing web site Web Workloads OPEN PARTLY OPEN

51. Carnegie Mellon University Computer Science Department 51 CONCLUSION The differences in behavior of closed, open,and partly-open systems. These principles underscore the importance of choosing the appropriate system model. Our findings provide guidelines for choosing whether an open or closed model is the better approximation based on characteristics of the workload. Understanding the appropriate system model is essential to understanding the impact of scheduling.