1. Computer Science Cataclysm: Policing Extreme Overloads in Internet Applications Bhuvan Urgaonkar and Prashant Shenoy University of Massachusetts

2. Computer Science Motivation  Internet applications used in a variety of domains  Online banking, online brokerage, online music store, e-commerce  Internet usage continues to grow rapidly  Broadband deployment is accelerating  Outages of Internet applications more common “Site not responding” “connection timed out”

3. Computer Science Internet Application Outages Down for 30 minutes Average download time ~ 260 sec Periodic outages over 4 days Cause: Too many users leading to overload Holiday Shopping Season 2000: 9/11: site inaccessible for brief periods

4. Computer Science Internet Data Centers Internet applications run on data centers Server farms Provide computational and storage resources Applications share data center resources Problem: How can the platform handle extreme overloads seen by applications?

5. Computer Science Handling Extreme Overloads  Existing work is based on three approaches  Request policing [Kanodia00, Li00, Verma03, Welsh03, …]  Dynamic capacity provisioning [Chase01, Ranjan04]  Degrade performance of admitted requests [Abdelzaher99]  Shortcomings of existing work:  Does not attempt to integrate these three approaches  Does not address scalability of the policer! The policer itself may become the bottleneck during overloads

6. Computer Science Our Contribution: Cataclysm  Comprehensive approach  Novel policer that can scale during overloads  Dynamic provisioning for both application and policer  SLA-based performance adaptation  Implementation and evaluation on a Linux cluster  Focus of this talk: design of the policer

7. Computer Science Talk Outline Motivation  Internet data center model  Request policing  Cataclysm Server Platform  Experimental results  Summary

8. Computer Science Data Center Model  Dedicated hosting: each application runs on a subset of servers in the data center  Subsets are mutually exclusive: no server sharing  Data center hosts multiple applications  Free server pool: unused servers Retail Web site streaming

9. Computer Science Internet Application Model  Internet applications replicated on multiple servers  E.g., clustered HTTP  Each application employs a sentry  Load balancing and request policing  One or more request classes  Service-level agreement  Specifies certain guaranteed request admission rate per class  Specifies allowed degradation in response time with arrival rate requests http load balancing sentry dropped requests

10. Computer Science Talk Outline Motivation Internet data center model  Request policing  Cataclysm Server Platform  Experimental results  Summary

11. Computer Science Policer: Design Goals  Class-based differentiation  Each class should sustain its guaranteed admission rate  Revenue maximization  Challenging due to online nature of the problem An admitted request may cause a more important request arriving later to be dropped  Approach: Preferential admission to higher class requests  Scalability  The policer should remain operational even under extremely high arrival rates

12. Computer Science Overview of Policer Design  Cataclysm policer has three components  Request classifier and per-class leaky buckets  Class-specific queues  Admission control Classifier Leaky buckets Class gold Class silver Class bronze Class-specific queues Admission control d gold d silver d bronze dropped admitted

13. Computer Science Class-based Differentiation Classifier Leaky buckets Class gold Class silver Class bronze Class-specific queues d gold d silver d bronze  Each incoming request undergoes classification  Per-class leaky buckets used to ensure that rates guaranteed in SLA are admitted Admission control dropped admitted

14. Computer Science Revenue Maximization Classifier Leaky buckets Class gold Class silver Class bronze Class-specific queues d gold d silver d bronze  Idea: Add different delays in processing of requests of different classes  More important requests processed more frequently  Methodology to compute delay values in online manner  Bounds probability of a request denying admission to a more important request Admission control dropped admitted

15. Computer Science Admission Control Classifier Leaky buckets Class gold Class silver Class bronze Class-specific queues d gold d silver d bronze Admission control  Goal: Ensure that an admitted request meets its response time target  Measurement-based admission control algorithm  Use information about current load on servers and estimated size of new request to make decision dropped admitted

16. Computer Science Scalability of Admission Control  Idea #1: Reduce the per-request admission control cost  Admission control on every request may be expensive  Bursty arrivals during overloads => batches get formed  Delays for class-based differentiation => batches get formed Admission control test that operates on batches instead of requests  Idea #2: Sacrifice accuracy for computational overhead  When batch-based processing becomes prohibitive  Threshold-based scheme E.g., Admit all Gold requests, drop all Silver and Bronze requests Thresholds chosen based on observed arrival rates and service times  Extremely efficient  Wrong threshold => bad response times or fewer requests admitted

17. Computer Science Scaling Even Further …  Protocol processing overheads will saturate sentry resources at extremely high arrival rates  Indiscriminate dropping of requests will occur Important requests may be turned away without even undergoing the admission control test Loss in revenue!  Sentry should still be able to process each arriving request!  Idea: Dynamic capacity provisioning for sentry  Pull in an additional sentry if CPU utilization of existing sentries exceeds a threshold (e.g., 90%)  Round-robin DNS to load balance among sentries

18. Computer Science Talk Outline Motivation Internet data center model Request policing  Cataclysm Server Platform  Experimental results  Summary

19. Computer Science Cataclysm Server Platform  Prototype data center  20 Pentium servers  Gigabit switches  Linux-based platform  Sentry implemented in Layer-7 switch  Linux module ktcpvs  Replicated Web server applications using Apache  Dynamic content using PHP

20. Computer Science Class-based Differentiation  Three classes of requests: Gold, Silver, Bronze  Policer successful in providing preferential admission to important requests

21. Computer Science Threshold-based: Higher Scalability  Threshold-based processing allows the policer to handle upto 4 times higher arrival rate  Single sentry can handle about 19000 req/s

22. Computer Science Threshold-based: Loss of Accuracy  Higher scalability comes at a loss in accuracy of admission control  Occasional violations of response time targets

23. Computer Science Sentry Provisioning

24. Computer Science Summary  Cataclysm: a comprehensive overload management technique consisting of  Request policing  Dynamic capacity provisioning  SLA-based performance adaptation  Cataclysm achieves the following  Class-based differentiation  Revenue maximization  Ability to scale to extreme overloads  More information: http://lass.cs.umass.edu

25. Computer Science Policing and Provisioning

26. Computer Science Policing and Provisioning