Presentation on theme: "Capacity Management for Web Operations John Allspaw Operations Engineering."— Presentation transcript:
Capacity Management for Web Operations John Allspaw Operations Engineering
the book Im writing
Rules of Thumb Planning/Forecasting Stupid Capacity Tricks (with some Flickr statistics sprinkled in)
bugs (disguised as capacity problems) edge cases (disguised as capacity problems) security incidents real capacity problems* * (should be the last thing you need to worry about) Things that can cause downtime
Capacity != Performance Forget about performance for right now Measure what you have right NOW Dont count on it getting any better
Thank You HPC Industry! Automated Stuff Scalable Metric Collection/Display a lot of great deployment and management tricks come from them, adopted by web ops
Good Measuremen t Tools record and store metrics in/out custom metrics easily compare lightweight-ish I
Clouds need planning too Makes deployment and procurement easy and quick But clouds are still resources with costs and limits, just like your own stuff Black-boxes: you may need to pay even more attention than before
Metrics System Statistics
Metrics Application Level (photos processed per minute) (average processing time per photo) (apache requests) (concurrent busy apache procs)
Metrics App-level meets system-level here, total CPU = ~1.12 * # busy apache procs (ymmv)
2400 photos per minute being uploaded right NOW (Tuesday afternoon)
Ceiling s the most amount of work your resources will allow before degradation or failure
Find your ceilings The End what you have left
Use real live production data to find ceilings Production: its like a lab, but bigger!
Like: database ceilings replication lag: bad!
Ceilings waiting on disk too much sustained disk I/O wait for >40% creates slave lag* *for us, YMMV
35,000 photo requests per second on a Tuesday peak
Ceiling * Factor of Safety = UR LIMITZ
Safety Factors webserver!
what you have left safe CPU Safety Factors 85% total CPU = ~76 busy apache procs
Safety Factors Yahoo Front Page link to Chinese NewYear Photos (photo requests/second) (8% spike)
Fictional Example: webservers
Forecasting Fictional example: 15 webservers. 1 week. peak of the week
...bigger sample, 6 weeks....isolate the peaks... Forecasting
...Add a Trendline with some decent correlation... Forecasting not too shabby now
Forecasting 15 busy apache proc limit = 1140 total procs when is this? this will tell you when it is ceiling what you have left
Forecasting (week #10, duh) ( ) / = 9.68
Writing excel macros is boring All we want is days remaining, so all we need is the curve-fit Forecasting Automation Use tohttp://fityk.sf.net automate the curve-fit
Stupid Capacity Tricks quick and dirty management ~]# dsh -N group.of.servers dsh> date executing 'date' www100: Mon Jun 23 14:14:53 UTC 2008 www118: Mon Jun 23 14:14:53 UTC 2008 dbcontacts3: Mon Jun 23 07:14:53 PDT 2008 admin1: Mon Jun 23 14:14:53 UTC 2008 admin2: Mon Jun 23 14:14:53 UTC 2008 dsh>
Stupid Capacity Tricks Turn Stuff OFF Disable heavy-ish features of the site(on/off switches) We have 195 different things to disable in case of emergency.
Stupid Capacity Tricks Turn Stuff OFF uploads (photo) uploads (video) uploads by various API things various mobile things various search things etc., etc.
Host your outage/status/blog page in more than one datacenter. Tell your users WTF is going on, theyll appreciate it. Stupid Capacity Tricks Outages Happen
Stupid Capacity Tricks Hit the Pause Button Bake the dynamic into static Some Y! properties have a big red button to instantly bake (and un- bake) at will