1. 1 Copyright © 2016, The Green Grid The webcast will begin shortly Today’s live session will be recorded

2. 2 Copyright © 2016, The Green Grid WI#14-011 The Performance Indicator: Assessing & Visualizing Data Center Cooling Performance Mark Seymour & Maira Bana 22 June 2016

3. 3 Copyright © 2016, The Green Grid Agenda 1.Introduction 2.The Metrics 3.Levels of Assessment 4.A Case Study

4. 4 Copyright © 2016, The Green Grid A method for assessing and visualizing data center cooling performance in terms of key metrics – thermal conformance, thermal resilience, and energy efficiency Objective of strong cooling performance: – Maintain safe and suitable thermal operation of IT – Minimize risk of equipment overheat Now and in the future At partial and full loads – Always considering efficiency Introduction Energy Efficiency Thermal Conformance Thermal Resilience

5. 5 Copyright © 2016, The Green Grid Introduction to Performance Indicator The Performance Indicator (PI) may be used to: 1.Visualize the balance between the three metrics 2.Assess a facility’s performance in relation to the company’s target range 3.Track a facility’s progress over time, as and when changes to the facility and IT are implemented 4.Assess performance effects of changes before actual implementations, from IT deployments to the installation of containment 5.Compare alternative configuration options

6. 6 Copyright © 2016, The Green Grid Introduction to Metrics There are three metrics to consider: – PUE ratio (PUEr): How effectively is the facility operating in relation to defined energy efficiency ratings? – IT Thermal Conformance: How much of the IT equipment is operating with suitable equipment inlet air temperatures during normal operation? – IT Thermal Resilience: Is any equipment at risk of overheating in the case of cooling failure or planned maintenance? For a complete and accurate view, future thermal states should be calculated in addition to the current operational state

7. 7 Copyright © 2016, The Green Grid The Metrics – PUEr(X) Based on PUE* and Energy Efficiency Ratings The PUEr Equation – PUEr(X) = PUE ref (X) ÷ PUE actual – where PUE ref (X) is the lowest PUE in Rating X Ratio represents the deviations from the intended operation * PUE was originally defined by The Green Grid and is now documented in ISO 30134 Part 2. PUE ranges for the Energy Efficiency Ratings Measured iPUE Reference or intended PUE

8. 8 Copyright © 2016, The Green Grid Example – PUEr(X) A facility with a target Energy Efficiency Rating C, and a PUE of 1.5  PUEr(C) = 1.35/1.5 = 90% PUEr(A) results in DCiE The target range minimum PUEr(X) is the ratio of Energy Efficiency Rating’s range  PUEr(C) = 1.35/1.63 = 83% PUE ranges for the Energy Efficiency Ratings

9. 9 Copyright © 2016, The Green Grid Example – PUEr(X) Using Energy Efficiency Ratings and business-specified targets allows for appropriate interpretation of performance balance

10. 10 Copyright © 2016, The Green Grid The Metrics – IT Thermal Conformance IT Thermal Conformance is the percentage of IT load operating within the recommended temperature spec at a given IT load and distribution What constitutes the temperature spec may be based on ASHRAE Temperature Compliance Guidelines (i.e., inlet temperatures below 27 °C), manufacturers’ specified thermal limits, RCI, etc. How safe is it to house IT during normal operation – Current load/distribution? – For some future plans? – At full design capacity? <1515-1818-2727-32>32 ASHRAE Temperature Compliance °C °F <5959-64.464.4-80.680.6-89.6>89.6

11. 11 Copyright © 2016, The Green Grid The Metrics – IT Thermal Resilience How safe is it to house IT during worst case failure scenario operation – Now, for some future plan, at full design capacity? When redundant cooling units are off-line, IT Thermal Resilience is the percentage of IT load operating within the allowable temperature spec (e.g., inlet temperatures below 32 °C) 1821.52528.532 °C °F 64.470.77783.389.6

12. 12 Copyright © 2016, The Green Grid The Metrics – IT Thermal Resilience The Green Grid does not recommend turning off cooling units with the sole intent of calculating resilience Measurements can be taken when redundant cooling units are down for maintenance A better way – without risk to the facility – simulation results can be used 1821.52528.532 °C °F 64.470.77783.389.6

13. 13 Copyright © 2016, The Green Grid Planning the Future How safe is my next IT expansion? Consider the impact of today’s changes on tomorrow – Can I still utilize 100% design capacity? – What is the future resilience? How will a change in one metric affect other metrics? What-if scenarios: ‘fill’ the 3D model to desired load/distribution scenarios and run the simulations IT equipment highlighted in red is lost or stranded thermal capacity

14. 14 Copyright © 2016, The Green Grid Levels of Assessment Assess current and future states through measurements and simulation Four options available to suit the owner/operator A basic current assessment is quick and easy Level of AssessmentCurrent StateFuture States Level 1 Basic (Measured) N/A Level 2 Advanced (Measured) N/A Level 3 Basic/Advanced (Simulated or Measured) Basic Level 4 Advanced (Simulated or Measured) Advanced

15. 15 Copyright © 2016, The Green Grid Levels of Assessment Advanced temperature monitoring: Measurements taken at an equipment or chassis level – from sensors, thermal images, DCIM Power monitoring level: Resolution at which heat load is measured for the calculation of the IT power draw CFD: An established scientific approach to thermal modeling Calibrated model: One that is verified to represent the characteristics of the real facility Real-Time MeasurementPredictive Simulation Level of Assessment Advanced Temperature MonitoringPower Monitoring Level Computational Fluid Dynamics (CFD) Model Fully Calibrated CFD Model Level 1 Level 2 (Rack/equipment) Level 3 (Room/rack/equipment) Level 4Optional (Rack/equipment)

16. 16 Copyright © 2016, The Green Grid A Case Study: The Facility Design Load: 490 kW (=107 W/sq. ft.) N+1 Cooling Redundancy Energy Efficiency Rating C Performance Targets: IT Thermal Conformance: 93-100% IT Thermal Resilience: 98-100% PUEr(C): 83-100%

17. 17 Copyright © 2016, The Green Grid A Case Study: Day 0 – Design Phase Partial Cold Aisle Containment: Roof panels avoided due to the implementation of a fire suppression system 8x DX 70 kW Cooling Units: The numbers add up: Load and airflow demand of the IT is matched by the 7 cooling units, +1 for redundancy Temperature Control: Cooling units controlled to average supply temperature of 18 ° C Uniformly Loaded Facility: 136 server racks at 3.5 kW, and 120 cfm/KW each 17 Network Racks

18. 18 Copyright © 2016, The Green Grid A Case Study: Day 1 – 30% Part Load 02.04.0 Rack Power kW Rack-Level Power Monitoring: Non-uniform equipment load in the room No containment on end aisle Deployment Deviation: Installation of vendor- configured racks in end row 18°C 18°C 18°C Temperature Monitoring: Thermal images provided the thermal profile, and temperature sensors (arranged in the ASHRAE-recommended configuration) calibrated the profile All sensors and thermal images showed temperatures within the recommended ASHRAE range (18-27°C), resulting in IT Thermal Conformance = 100%, PUEr(C) = 64% 1821.52528.532 °C°C °F°F 64.4 70.77783.389.6

19. 19 Copyright © 2016, The Green Grid A Case Study: Day 2 – CRAC Unit Maintenance Temperature monitoring undertaken (sensors and thermal imaging) in the maintenance scenario with only 7 (N) cooling units operating All measured temperatures are below the ASHRAE allowable maximum (32°C/89.6°F), resulting in IT Thermal Resilience = 100%

20. 20 Copyright © 2016, The Green Grid A Case Study – The Performance Indicator (Measured) The owner/operator: – Has measured data for all 3 metrics – Can plot the current day performance At 30% part load facility fails to meet target PUEr(C) criteria The degraded range (which considers the effects of part load) supports this expectation

21. 21 Copyright © 2016, The Green Grid A Case Study – Efficiency Improvements The PI could be used to compare facility configurations in the bid to improve efficiency 1.CRAC flow rate control: providing the ability to reduce flow via a control system 2.Supply temperature increase Equipment inlet temperatures are monitored during this change

22. 22 Copyright © 2016, The Green Grid A Case Study: Day 3 – 60% Load Installation of new equipment: networking, servers, and vendor-configured racks Some sensors now showing elevated temperatures Decides to upgrade to Level 4 assessment to diagnose causes and experiment with solutions Day 3Day 1 Day 3 Day 1 New deployment of vendor-configured racks 1821.52528.532 °C°C °F°F 64.4 70.77783.389.6 03.757.5 Rack Power kW

23. 23 Copyright © 2016, The Green Grid A Case Study: Model Calibration A model is built, and simulation results are compared with measured data Strong agreement between the two indicates a calibrated model – a model that is an accurate representation of the facility Measurement tools include: – Flow hood – Flow grid – Anemometer – Temperature monitoring

24. 24 Copyright © 2016, The Green Grid A Case Study: Model Calibration Examples of measured 1, uncalibrated 2, and calibrated 3 results: 1 2 3 123 123

25. 25 Copyright © 2016, The Green Grid A Case Study: 60% Load Results are extracted from the calibrated model, and the PI metrics are calculated: 2.5 kW 0.3 kW 2.3 kW

26. 26 Copyright © 2016, The Green Grid A Case Study: Operational Plans A series of operational plans are undertaken to rectify issues and improve the PI score: 1.Customized blanking 2.Relocation of equipment 3.Cooling of recirculating air

27. 27 Copyright © 2016, The Green Grid Another series of operational plans are simulated (before implementation) to assess their effectiveness: 1.Operation at full design capacity (as shown below) 2.Operation at 80% part load 97% A Case Study: Operational Plans

28. 28 Copyright © 2016, The Green Grid A Case Study: Operational Plans The PI is then used to compare facility configurations in the bid to improve efficiency: 1.CRAC flow rate control: providing the ability to reduce flow via a control system 2.Supply temperature increase: a supply temperature control set-point of 21 °C rather than 18 °C CRAC Flow Reduction Supply Temperature Increase

29. 29 Copyright © 2016, The Green Grid Summary Baseline – Where I am What-if scenarios – Improving PUE by increasing inlet air temperatures – Different IT loads – Different load distributions Knowing before doing – Higher availability – Higher certainty of capacity – More efficiency and more savings

30. 30 Copyright © 2016, The Green Grid Thank you www.thegreengrid.org