1. Cutting the Electric Bill for Internet-Scale Systems Andreas Andreou Cambridge University, R02 aa773@cam.ac.uk

2. What’s this all about? Energy expenses are an increasingly important fraction of data center operating costs Electricity prices show both temporal and geographical variation Exploit variations in electricity prices for economic gain

3. Key observations Electricity prices vary Prices vary on an hourly basis Often not well correlated at different locations Substantial variations Large distributed systems already incorporate request routing and replication Dynamic request routing to map clients to servers Mechanisms to replicate data necessary to process requests at multiples sites

4. Problem Specification Large system composed of server clusters spread out geographically Map client requests to clusters such that the total electricity cost is minimized Assumptions System fully replicated Optimize for cost every hour No knowledge of the future Rate of change slow enough to be compatible with existing routing mechanisms Fast enough to respond to electricity market fluctuations Incorporate bandwidth and performance goals as constraints

5. Terminology Energy Elasticity Degree to which energy consumed by a cluster depends on the load placed on it Ideally: no load, no power Worst case: no difference between peak and idle power State-of-the-art: idle power around 60% of peak Differential Duration Number of hours one location is favored over another by more than $5/MWh PUE Power usage effectiveness (measure of data center energy efficiency)

6. Background

7. Wholesale Electricity Markets (1) Generation Government and independent power producers Coal (~50%), natural gas (~20%), nuclear power (~20%), hydroelectric generation (~6%) Different regions, different power generation profiles Transmission Producers and consumers are connected to an electric grid 8 reliability regions

8. Wholesale Electricity Markets (2) Market Structure Each region managed by Regional Transmission Organization (RTO) RTO administer wholesale electricity markets Auctioning mechanism: Producers present supply offers Consumers present demand bids Coordinating body determines flow and sets prices Market Types Day-ahead markets Real-time markets

9. Wholesale Electricity Markets (3) Market Structure Assumptions Real-time prices are known and vary hourly Electric bill is proportional to consumption and indexed to wholesale prices Request routing behavior induced by our method doesn’t significantly alter prices and market behavior

10. Daily Variation

11. Different Market Types Hourly real-time (RT) market is more volatile than day-ahead market

12. Hour-to-Hour Volatility

13. Geographic Correlation

14. Price Differentials

15. Differential Distributions

16. Time-of-Day

17. Differential Duration

18. Akamai: Traffic and Bandwidth Over 2000 content provider customers in the US 9-region traffic with electricity price data Data covering 24 days worth of traffic Traffic data of 5-minute intervals from public clusters Bandwidth costs are significant Aggressively optimized to reduce bandwidth costs 95/5 billing model Client-Server Distances Use geographic distance as a coarse proxy for network performance

19. Cluster Energy Consumption (1) Roughly linear to its utilization P idle : average idle power draw of single server P peak : average peak power draw of single server r: empirical derived constant u t : average CPU utilization at time t what is important in determining savings

20. Routing Energy Increased path lengths will not alter energy consumption significantly Average energy for a packet to pas through is on the order of 2mJ Incremental energy dissipated by each packet passing through a core router would be as low as 50μJ per medium size packet New routes may overload existing routers Additional bandwidth could lead to upgrade Can ignore by incorporating 95/5 bandwidth constraints

21. Simulation Strategy Real-time market prices for 29 different locations Traffic data for Akamai public clusters in 9 of those Data set spanning Jan 2006 through Mar 2009 Workload data set contains 5-minute samples in 25 cities Period of 24 days and some hours Discarded 7 and grouped remaining 18 cities to 9 clusters Akamai’s geographic server distribution Two routing schemes Akamai’s original allocation Distance constrained electricity price optimizer Energy model as shown before

22. 24 Days of Traffic (1) Energy Elasticity Bandwidth Costs

23. 24 Days of Traffic (2) Distance and savings

24. 39 Months of Prices Derived from 24-day Akamai workload (US traffic only) Dynamic beats static

25. Results Existing systems can reduce energy costs be at least 2% without any increase in bandwidth costs or significant reduction in client performance Google-like energy elasticity Akamai-like server distribution 95/5 bandwidth constraints Savings increase with energy elasticity Fully elastic system with relaxed bandwidth constraints can reduce energy cost be 30% (13% with bandwidth constraints) Allowing increase of client-server distances leads to increased savings

26. Considerations (1) Not reacting immediately to price changes noticebly reduces overall savings

27. Considerations (2) Server operators should be able to negotiate contractual arrangements Distributed systems with energy elastic clusters can be more flexible than traditional consumers Triggered demand response programs

28. Future Work Implementing Joint Optimization RTO Interaction Weather Differentials Environmental Cost