1. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science Dynamic Provisioning for Multi-tier Internet Applications Bhuvan Urgaonkar, Prashant Shenoy, Abhishek Chandra, Pawan Goyal University of Massachusetts University of Minnesota Veritas Software India Pvt. Ltd.

2. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 2 Internet Applications  Proliferation of Internet applications auction siteonline gameonline store  Growing significance in personal, business affairs  Focus: Internet server applications

3. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 3 Multi-tiered Internet Applications  Internet applications: multiple tiers  Example: 3 tiers: HTTP, J2EE app server, database  Replicable components  Individual tiers: partially or fully replicable  Example: clustered HTTP, J2EE server, shared-nothing db  Each tier uses a dispatcher: load balancing requests http J2EE database Load balancer

4. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 4 Internet Workloads Are Dynamic  Multi-time-scale variations  Time-of-day, hour-of-day  Flash crowds Key issue: How to provide desired response time under varying workloads? 0 20000 40000 60000 80000 100000 120000 140000 05101520 Time (hrs) Request Rate (req/min) 0 12 24 Time (hours) Time (days) 0 12345 Arrivals per min 0 0 140K 1200

5. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 5 Internet Data Center  Internet applications run on data centers  Server farms o Provide computational and storage resources  Applications share data center resources Problem: How should the platform allocate resources to absorb workload variations?

6. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 6 Our Provisioning Approach  Flexible queuing theoretic model  Captures all tiers in the application  Predictive provisioning  Long-term workload variations  Reactive provisioning  Short-term variations, flash crowds

7. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 7 Talk Outline Introduction Internet data center model  Existing provisioning approaches  Dynamic capacity provisioning  Implementation and evaluation  Summary

8. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 8 Data Center Model  Dedicated hosting: each application runs on a subset of servers in the data center  Subsets are mutually exclusive: no server sharing  Data center hosts multiple applications  Free server pool: unused servers Retail Web site streaming

9. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 9 Single-tier Provisioning  Single tier provisioning well studied [Muse]  Non-trivial to extend to multiple-tiers  Strawman #1: use single-tier provisioning independently at each tier  Problem: independent tier provisioning may not increase goodput C=15 C=10 C=10.1 14 req/s 14 10 dropped 4 req/s

10. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 10 Single-tier Provisioning  Single tier provisioning well studied [Muse]  Non-trivial to extend to multiple-tiers  Strawman #1: use single-tier provisioning independently at each tier  Problem: independent tier provisioning may not increase goodput C=15 C=10.1 14 req/s 14 C=20 14 dropped 3.9 req/s 10.1

11. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 11 Model-based Provisioning  Black box approach  Treat application as a black box  Measure response time from outside  Increase allocation if response time > SLA o Use a model to determine how much to allocate  Strawman #2: use black box for multi-tier apps  Problems:  Unclear which tier needs more capacity  May not increase goodput if bottleneck tier is not replicable 14 req/s C=15 C=10.1 14 C=20 14 10.1

12. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 12 Provisioning Multi-tier Apps  Approach: holistic view of multi-tier application  Determine tier-specific capacity independently  Allocate capacity by looking at all tiers (and other apps)  Predictive provisioning  Long-term provisioning: time scale of hours  Maintain long-term workload statistics  Predict and provision for the next few hours  Reactive provisioning  Short term provisioning: time scale of several minutes  React to “current” workload trends  Correct errors of long-term provisioning  Handle flash crowds (inherently unpredictable)

13. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 13 Predictive Provisioning  Workload predictor  Predicts workload based on past observations  Application model  Infers capacity needed to handle given workload PredictorModel past workload predicted workload required capacity response time target

14. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 14 Workload Prediction  Long term workload monitoring and prediction  Monitor workload for multiple days  Maintain a histogram for each hour of the day o Capture time of day effects  Forecast based on o Observed workload for that hour in the past o Observed workload for the past few hours of the current day  Predict a high percentile of expected workload Mon Tue Wed Today

15. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 15 Model-based Capacity Inference  Queuing theoretic application model  Each individual server is a G/G/1 queue  Derive per-tier E(r) from end-to-end SLA  Monitor other parameters and determine  per-server capacity)  Use predicted workload pred to determine # servers per tier o Assumes perfect load balancing in each tier G/G/1 pred

16. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 16 Reactive Provisioning  Idea: react to current conditions  Useful for capturing significant short-term fluctuations  Can correct errors in predictions  Track error between long-term predictions and actual  Allocate additional servers if error exceeds a threshold  Account for prediction errors  Can be invoked if request drop rate exceeds a threshold  Handles sudden flash crowds  Operates over time scale of a few minutes  Pure reactive provisioning: lags workload  Reactive + predictive more effective! Prediction error pred actual error >  Invoke reactor time series allocate servers

17. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 17 Talk Outline Introduction Internet data center model Existing provisioning approaches Dynamic capacity provisioning Implementation and evaluation  Summary

18. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 18 Prototype Data Center  40+ Linux servers  Gigabit switches  Multi-tier applications  Auction (RUBiS)  Bulletin-board (RUBBoS)  Apache, Tomcat (replicable)  Mysql database Control Plane Dynamic provisioning Nucleus Apps OS Server Node Applications Resource monitoring Parameter estimation Nucleus Apps OS Nucleus Apps OS

19. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 19 Only Predictive Provisioning WorkloadResponse time  Auction application RUBiS  Factor of 4 increase in 30 min  Predictor fails during [15, 30] resulting in under-provisioning  Response time violations occur

20. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 20 Only Reactive Provisioning WorkloadResponse time  Auction application RUBiS  Factor of 4 increase in 30 min  Response time shows oscillatory behavior  Several response time violations occur Time (min) Resp time (msec)

21. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 21 Predictive + Reactive Provisioning WorkloadResponse time Server allocations  Auction application RUBiS  Factor of 4 increase in 30 min  Server allocations increased to match increased workload  Response time kept below 2 seconds

22. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 22 Summary  Dynamic provisioning for multi-tier applications  Flexible queuing theoretic model o Captures all tiers in the application  Predictive provisioning  Reactive provisioning  Implementation and evaluation on a Linux cluster

23. U NIVERSITY OF M ASSACHUSETTS, A MHERST – Department of Computer Science 23 Thank you! More information at: http://www.cs.umass.edu/~bhuvan