Insights into Google's PUE Results Spring 2009

Published Google PUE Results

Today, more details on... Innovations and best practices that enabled the results Measurement method & accuracy Benefits of measuring PUE And, a PUE update + a Google server on display Insights into Google's PUE Results

*Reference: Silicon Valley Leadership Group, Data Center Energy Forecast, Final Report July, 2008 Google E Data Center energy-weighted average PUE results from Q2-Q1’09 (to 3/15/09) Google Data Center ‘E’ PUE = 1.16 Typical PUE = 2.0 Impact of power & cooling innovations and best practices PUE Components: Typical vs. Google

*Reference: Silicon Valley Leadership Group, Data Center Energy Forecast, Final Report July, 2008 Google E Data Center energy-weighted average PUE results from Q2-Q1’09 (to 3/15/09) 85% reduction in cooling energy Google Data Center ‘E’ PUE = 1.16 Typical PUE = 2.0 Impact of power & cooling innovations and best practices PUE Components: Typical vs. Google

*Reference: Silicon Valley Leadership Group, Data Center Energy Forecast, Final Report July, 2008 Google E Data Center energy-weighted average PUE results from Q2-Q1’09 (to 3/15/09) 82% reduction in power distribution/backup losses Google Data Center ‘E’ PUE = 1.16 Typical PUE = 2.0 Impact of power & cooling innovations and best practices PUE Components: Typical vs. Google

ResultHow?Details Cooling energy reduced by 85% Close-coupled cooling Raise the temperatures Economizers This afternoon Power distribution & backup losses reduced by 82% 99.9% efficient UPSNext PUE Components: Typical vs. Google

Google's On-Board UPS

Facility UPS Inverter Battery Pack Rectifier Grid ACIT ACUPS

Facility UPS Grid ACIT AC

Facility UPS Grid ACIT AC

Motherboard Disk PSU Typical Server

Motherboard Disk PSU Google Server * DCP++

Motherboard Disk PSU Google Server * Featured in the Climate Savers Computing (initiative)

Motherboard Disk PSU Google Server

Motherboard Disk PSU Battery Google Server

Timeout or low battery Shutdown Float Discharge Recharge AC on AC loss AC back Battery full AC loss Google UPS State Diagram

Timeout or low battery Shutdown Float Discharge Recharge AC on AC loss AC back Battery full AC loss Google UPS State Diagram

Float: V SUPPLY ≈ V LOAD ≈ V BATT > 13V AC live, load running, battery float charging R LOAD ≈ 0.5Ω V SUPPLY ≈ 13V R CHARGER ≈ 20Ω V BATT ≈ 13V R BATT ≈ 100mΩ Discharge Battery leakage (self-discharge) current 99.99%

Design Summary and Achievements Examine the entire cluster for a solution, not just the UPS Eliminate central UPS, distribute to each machine Eliminate AC-DC-AC double conversion Small battery, just enough to bridge until generators start up, or clean machine shutdown Incremental deployment of UPS capacity - no waste! Single-voltage PSU is the enabler, also made servers more power-efficient (not included in PUE computation) Real-world measured efficiency of >99.99%

PUE Measurement Methods

Total Facility Power IT Equipment Power PUE = Technology Investment ScenarioPUE Typical 2006 Data Center2.0 Current Trends1.9 Improved Operations1.7 Best Practices1.3 State-of-the-Art Era Data Center References: The Green Grid, 2007, “The Green Grid Data Center Power Efficiency Metrics: PUE and DCiE,” Technical Committee White Paper, Environmental Protection Agency, ENERGY STAR Program, 2007, “Report to Congress on Server and Data Energy Efficiency”. EPA PUE Projections Green Grid Data Center Efficiency Metric PUE Definition & Trends

Total Facility Power IT Equipment Power PUE = Technology Investment ScenarioPUE Typical 2006 Data Center2.0 Current Trends1.9 Improved Operations1.7 Best Practices1.3 State-of-the-Art Era Data Center EPA PUE Projections Green Grid Data Center Efficiency Metric PUE Definition & Trends Remember this number! References: The Green Grid, 2007, “The Green Grid Data Center Power Efficiency Metrics: PUE and DCiE,” Technical Committee White Paper, Environmental Protection Agency, ENERGY STAR Program, 2007, “Report to Congress on Server and Data Energy Efficiency”.

Level 1 (Basic)Level 2 (Intermediate) Level 3 (Advanced) IT Equipment Power Measurement From... UPSPDUServers Total Facility Power Measurement From... Data Center Input Power Data Center Input Less Shared HVAC Data Center Input Less Shared HVAC plus Building, Lighting, Security Minimal Measurement Interval Monthly/WeeklyDailyContinuous Reference: The Green Grid, 2008, “The Green Grid Metrics: Data Center Infrastructure Efficiency (DCiE) Detailed Analysis,” Technical Committee White Paper Green Grid Measurement Accuracy Classifications

Reference: The Green Grid, 2008, “The Green Grid Metrics: Data Center Infrastructure Efficiency (DCiE) Detailed Analysis,” Technical Committee White Paper Level 1 (Basic)Level 2 (Intermediate) Level 3 (Advanced) IT Equipment Power Measurement From... UPSPDUServers Total Facility Power Measurement From... Data Center Input Power Data Center Input Less Shared HVAC Data Center Input Less Shared HVAC plus Building, Lighting, Security Minimal Measurement Interval Monthly/WeeklyDailyContinuous Enabled on newer servers 2 of 6 reported data centers 4 of 6 reported data centers Green Grid Measurement Accuracy Classifications Google Data Centers: 2+

MV Switchgear Overhead (Cooling plant...) Overhead (CRACs...) ServersUPS NetworkUPS NetworkUPS L3 L2 IT Equipment Power Losses & Overhead Power Substation 99.99% Efficient UPS High Voltage Utility Distribution Medium Voltage Low Voltage Google Data Center Power Distribution Schematic

MV Switchgear Overhead (Cooling plant...) Overhead (CRACs...) ServersUPS NetworkUPS NetworkUPS L3 L2 Substation Total Data Center power extrapolated to here IT power extrapolated to here PUE = Total Facility Power IT Equipment Power ~ Σ Unit Substations + Small Terms Σ IT Unit Substations - Small Terms PUE Uncertainty <2% Achieved with Accurate Unit Substation Meters IT Equipment Power Losses & Overhead Power

MV Switchgear Overhead (Cooling plant...) Overhead (CRACs...) ServersUPS NetworkUPS NetworkUPS L3 L2 Substation IT power extrapolated to here Measurement Uncertainty (CT + PT + Meter) = <±2% Primary Power Measurement Point High-Accuracy Metering at the Unit Substations

MV Switchgear Overhead (Cooling plant...) Overhead (CRACs...) ServersUPS NetworkUPS NetworkUPS L3 L2 Substation Total Data Center power extrapolated to here IT power extrapolated to here Unit Substation Measurement 10% of total error Example: Data Center 'E': PUE = 1.16 ± 1.0% Unit Substation Measurement Uncertainty IT Equipment Power Losses & Overhead Power

MV Switchgear Overhead (Cooling plant...) Overhead (CRACs...) ServersUPS NetworkUPS NetworkUPS L3 L2 Substation Overhead Unit Substation M Total Data Center power extrapolated to here IT power extrapolated to here Substation-to-Meter Losses 2% total power 13% of error M Example: Data Center 'E': PUE = 1.16 ± 1.0% Substation-to-Meter Uncertainty IT Equipment Power Losses & Overhead Power

IT power extrapolated to here Meter-to-IT Losses 1% total power 5% of error MV Switchgear Overhead (Cooling plant...) Overhead (CRACs...) ServersUPS Network UPS L3 L2 Substation Overhead Unit Substation M Total Data Center power extrapolated to here M CRACs & Parasitics 3% total power 73% of error Network UPS M Network 0.2% total power 0.2% of error Example: Data Center 'E': PUE = 1.16 ± 1.0% Low Voltage-Side Uncertainty IT Equipment Power Losses & Overhead Power

Uncertainty Component% Unit Substation Energy% Contribution to Total Uncertainty Unit Substation Measurement 100%10% Substation-to-Meter Estimate 1.9%13% Network Estimate0.2% CRACs & Parasitics Estimate 3.2%73% Meter-to-IT Estimate0.6%5% Very accurate measurement of Unit Substation power dominates uncertainty calculation Estimated components are less accurate, but small in magnitude Example: Data Center 'E': PUE = 1.16 ± 1.0% Summary

PUE Results

Data published quarterly for all Google data centers with 5+ MW IT load for at least 6 months Commissioning & seasonal effects January 2009 PUE Results

Data published quarterly for all Google data centers with 5+ MW IT load for at least 6 months Continued optimization & beneficial seasonal effects Latest PUE Results

ScenarioQ1’09 to 3/15Q4’08Q3’08 Quarterly energy-weighted average PUE * Trailing twelve-month energy- weighted avg. PUE *1.21 Individual facility minimum quarterly PUE *1.17 Individual facility minimum TTM PUE** 1.15 Individual facility maximum quarterly PUE * Individual facility maximum TTM PUE** *Audit revealed a 1-2% error in previously published results. **Only facilities with at least twelve months of operation are eligible for individual facility TTM PUE reporting PUE Results Update: Q1'09 to 3/15

ScenarioQ1’09 to 3/15Q4’08Q3’08 Quarterly energy-weighted average PUE * Trailing twelve-month energy- weighted avg. PUE *1.21 Individual facility minimum quarterly PUE *1.17 Individual facility minimum TTM PUE** 1.15 Individual facility maximum quarterly PUE * Individual facility maximum TTM PUE** *Audit revealed a 1-2% error in previously published results. **Only facilities with at least twelve months of operation are eligible for individual facility TTM PUE reporting PUE Results Update: Q1'09 to 3/15 Better than the 2011 EPA State-of-the-Art data center PUE of 1.20

Deeper insight into commissioning & seasonal effects Q2’08 Q3’08Q4’08 Q1’09 Daily PUE Overhead – Google ‘E’ PUE Components We Measure by the Minute

Unusual weather impacts chiller runtime Google Data Center ‘E’ January 2009 Google Data Center ‘E’ Week 1 January 2009 Hourly Average PUE Reporting Minute-by-Minute Measurements

Identify commissioning & operational issues, design & deployment plan decisions IT Growth Weather, commissioning & plant scaling Insights from Measurements PUE vs. IT Load

80+% reduction in power & cooling overhead vs. typical facility o Google UPS solution delivers >99.99% efficiency  Consider options in the market that approach this efficiency  Line-interactive & multi-mode UPS, flywheels, DC distribution o Cooling best practices  Close-coupled cooling, raise the temperatures, & economizers Continued improvement in latest Google PUE result o TTM Average PUE Trend:  1.21 (Q3'08) to 1.20 (Q4'08) to 1.19 (to 3/15/'09) Know your PUE What's next? Conclusions

Best practice: minimize chiller operating hours Best embodiment: chiller-less data centers o Reduce capital costs o Reduce the PUE spikes o Target TTM average PUE ~ 1.1 Chiller-less Belgium data center operational The Limit: Chiller-Less Data Center

Thank you! Q&A