Data Center Thermal Management and Efficiency Jay Ries Regional Sales Manager Liebert Thermal Management Emerson Network Power

Agenda Where is energy consumed in the data center? 3/31/2017 Agenda Where is energy consumed in the data center? Energy consumption example Cooling energy consumption breakdown Strategies for saving energy Low cost strategies Medium cost strategies Higher cost strategies Taking it a step further (beyond cooling) Summary

Where is Energy Consumed in the Data Center? 52% is consumed by IT equipment 48% is consumed by power and cooling support

Energy Consumption Example Baseline Building design Existing building Limitation to physical changes that can be made Best suited for modifications to existing equipment Full equipment replacement is a last resort 1MW of facility power usage (all data center) Baseline Cooling design Centrifugal water cooled chiller No economization Standard computer room cooling units No variable speed fans or advanced controls Return air control 45° F chilled water 72° F return air, 50% RH

Energy Consumption Example Power Usage Processors – 150kW Other Services – 150kW Server Power Supply – 140kW Storage – 40kW Communication Equipment – 40kW Cooling – 380kW UPS – 50kW MV Transformer and Switchgear – 30kW Lighting – 10kW PDU – 10kW IT Power Usage = 520kW Support Power Usage = 480kW Total Facility Power Usage = 1000kW Annualized Facility PUE = 1.92  Work our way to 1.35 Cooling is the only area that will be modified. In the real world, each variable will have an impact on the others

Cooling Energy Consumption Breakdown Air Cooled System Water Cooled System Chilled Water System

Low Cost Strategies Implementing best practices Adjust the unit control methods Dew point control Unit operating range Change to supply air control Running at higher chilled water temperatures

Low Cost Strategies 1. Implementing Best Practices If you have a raised floor, use it properly. Underfloor resistance wastes energy. Utilize hot aisle / cold aisle, regardless if you have a raised floor

Low Cost Strategies 1. Implementing Best Practices Get air where it is supposed to go. Blanking panels Fix unplanned outside infiltrations and any unecessary gaps in the raised floor Return plenums to the cooling unit Isolate the room, particularly if you want to control humidity

Low Cost Strategies 2. Adjust Unit Settings Dew Point Standard design points used to be 72° return air temperature and 50% relative humidity (RH) New, more aggressive design points can be 90°+ return air temperature and an unspecified relative humidity Why shouldn’t you fix at 50% relative humidity (RH) Dew point @ 72°, 50% = 52° Dew point @ 95°, 50% = 74° If the return temperature is increased at a fixed RH, the dew point will rise, requiring the equipment to waste energy to remove moisture that didn’t need to be there in the first place

Low Cost Strategies 2. Adjust Unit Settings Unit operation settings Expanding the operating range for the temperature and humidity keeps unit components from cycling too frequently. Higher return air temperatures allow CRAH units to run more efficiently Capacity increase up to 70% for chilled water units Capacity increase up to 50% for compressor based units The more efficiently the units operate, the fewer that are required to control the space, saving energy.

Increased Capacity at Higher Temps 3/31/2017 Low Cost Strategies 2. Adjust Unit Settings Increased Capacity at Higher Temps Operating at a higher return air temperature increases the CRAC unit’s capacity. This graph shows how the sensible cooling capacity typically increases at different return air temperatures for Liebert CW and DX CRAC units. 75 degrees F is used as the base point, so the percentage increases are the sensible capacity increase above the 75 degree capacity. These are general curves and the actual values will vary slightly from model to model but not significantly. We are going to look at the results for our DX units in a second but before we leave this graph I want to look briefly at the CW curve. For a CW unit – shown as red line -- the capacity increase is higher than the DX unit. You must remember however that here are many other components involved in the overall chilled water system that must be sized correctly to produce these results…… This requires a whole separate study. Next slide

Low Cost Strategies 3. Supply Air Control Supplies a consistent temperature to the cold aisle Saves energy because it allows the return air temperature to be increased, allowing the CRAH unit to run more efficiently.

Low Cost Strategies 4. Running At Higher Water Temperatures 45° chilled water temperature has been the standard design point for many years Higher chilled water temperatures are starting to become more prevalent Why? At higher temperatures, there are huge potential savings on the chiller For every 1 degree increase in the chilled water supply temperature, a 2% energy savings can be realized on the chiller plant 45°chilled water = Baseline 55°chilled water = 20% energy savings

Low Cost Strategies The Results of Implementation Applying Low Cost Strategies Changes to cooling system Best practices implemented Supply air control 50° F chilled water 85° F return air with dew point control Support Power Usage = 480kW Total Facility Power Usage = 1000kW Annualized Facility PUE = 1.92 Total cooling power usage drops from 380kW to 314kW. The number of units stay the same, but some units can be turned off. 414kW 934kW 1.79

Medium Cost Strategies Variable speed fan retrofits (EC Fan / VFD) Aisle containment Control retrofits Rack level sensors

Medium Cost Strategies 1. Variable speed fan retrofits (EC Fan / VFD) Floor-mount cooling fans typically run at 100% rated rpm By utilizing variable speed technology, fan speed can be varied based upon room conditions Energy savings based on a single 10HP motor Fan Speed   Energy Consumed Savings 100% 8.1kWH 90% 5.9kWH 27% 80% 4.2kWH 48% 70% 2.8kWH 65% 60% 1.8kWH 78%

Medium Cost Strategies 2. Aisle Containment Allows for proper air separation Able to be done either the hot or cold aisle, though it is easier to retrofit the cold aisle of an existing room Physical containment varies from simple curtains to a pre-fabricated system designed to match the racks.

Medium Cost Strategies 2. Aisle Containment Containment Strategies Contained hot aisle Requires full containment to trap hot air Can be difficult to retrofit in perimeter designs Easier to retrofit in row cooling designs Overhead fire suppression concerns on full containment Contained cold aisle Multiple containment options Doors only Curtains only Full containment Can be easier to retrofit in all cooling designs

Medium Cost Strategies 3. Control Retrofits Allows for upgraded control schemes that save energy New controls allow units to be networked together Give more visibility of full system Eliminate fighting of units, - one cooling while one is heating

Medium Cost Strategies 4. Remote Sensors Usually associated with a control retrofit or a designed scheme through a building management system Increased visibility and quicker reaction to changes at the rack Generally applied with supply air sensors “Bath tub effect”

ROI is generally less than 1 year for these strategies Low + Medium Cost Strategies The Results of Implementation Applying Low + Medium Cost Strategies Changes to cooling system Best practices implemented Supply air control +55° F chilled water +90° F return air with dew point control + Remote sensors + Aisle containment + Variable speed fans + Control retrofits Support Power Usage = 414kW Total Facility Power Usage = 934kW Annualized Facility PUE = 1.79 Total cooling power usage drops from 314kW to 184kW. All units are now on, running at a reduced speed. 284kW 804kW 1.55 ROI is generally less than 1 year for these strategies

Higher Cost Strategies (Major Capital Expenditures) Bringing cooling closer to the source Variable capacity compressors Economization Air economizers Water economizers Refrigerant Economizers

Higher Cost Strategies 1. Bringing Cooling Closer to the Source Rack-based configuration Rear door configuration Row-based configuration Bring the cooling closer minimizes the need for large fans, reducing energy Some rear door designs don’t have fans, instead utilizing the server fans to move the air Generally produce a better sensible cooling to power ratio than a typical system – more cooling for less energy

Higher Cost Strategies 1. Bringing Cooling Closer to the Source 3/31/2017 Higher Cost Strategies 1. Bringing Cooling Closer to the Source Rack Based Solutions Pump Refrigerant Technology Dew Point Controlled Pumped Refrigerant Cooling Base Infrastructure (160 kw) Cooling Modules (mix and match)

Higher Cost Strategies 1. Bringing Cooling Closer to the Source Rear Door Solutions Refrigerant Based Rear Door Refrigerant based, rear door heat exchanger A rear door with 10kW to 40kW of cooling Connect up to 16 doors onto a single pumped refrigerant loop Designed to accommodate various racks Energy story – passive door (no fans) that uses the server fans to transfer air through the coil Performance Provides room neutral high density rack cooling Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration Product Overview: Extension of the Liebert XD high density cooling product line using pumped refrigerant A rear door for a computer rack which supplies up to 20 KW of cooling. Cooling module swings out of the way to allow free access to the back of the server rack Designed to accommodate various racks Server fans transfer air from rack front through coils Open architecture design provides room cooling for back-up protection Piping connections from the cabinet top Status: Limited availability in April 2009 Full availability at end of Q2 2009 Performance Provides room neutral high density rack cooling Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration

Higher Cost Strategies 1. Bringing Cooling Closer to the Source Rear Door Solutions Chilled Water Based Rear Door Chilled water based, rear door heat exchanger A rear door with 16kW to 35kW of cooling Designed to accommodate various racks Energy story – passive door (no fans) that uses the server fans to transfer air through the coil Performance Provides room neutral high density rack cooling Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration Product Overview: Extension of the Liebert XD high density cooling product line using pumped refrigerant A rear door for a computer rack which supplies up to 20 KW of cooling. Cooling module swings out of the way to allow free access to the back of the server rack Designed to accommodate various racks Server fans transfer air from rack front through coils Open architecture design provides room cooling for back-up protection Piping connections from the cabinet top Status: Limited availability in April 2009 Full availability at end of Q2 2009 Performance Provides room neutral high density rack cooling Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration

Higher Cost Strategies 1. Bringing Cooling Closer to the Source 3/31/2017 Higher Cost Strategies 1. Bringing Cooling Closer to the Source Row Based Solutions Row Based Solutions Precise temperature and Humidity control 12” or 24” wide designs Air, Water, Glycol and Chilled Water models Energy efficient, load matching Digital scroll compressor, 20-100% cooling capacity modulation Variable speed EC plug fans Performance Real-time environment control Automatic performance optimization Adaptive component monitoring Adjustable air baffle direction Lots of features, same frame construction as Sys/3, powder coated panels, IR, pump-down cycle….

Higher Cost Strategies 1. Bringing Cooling Closer to the Source Rack-based configuration Rear door configuration Row-based configuration Fan Energy for 30kW of Cooling Perimeter Unit = 4.24 kW Rack Based = 0.54 kW Row-Based Unit = 1.38 kW Rear Door = 0.00 kW (no fans)

Higher Cost Strategies 2. Variable Capacity Compressors Digital Scroll Compressors Matches room load in unlimited step increments Reliable Not field repairable. Must be replaced. 4-step Semi-Hermetic Compressors Matches room load in 4 step increments Field repairable Compressors w/ VFD Control Usually not field repairable. The digital scroll also operates to save energy but does it in a slightly different. The control band is set up exactly like the 4-step. Since the digital is a proportional capacity from 10 to 100% compressor operation begins at 10% (72.2F) and will increase to precisely match the % call for cooling. The compressor is controlled by a solenoid valve, when the valve is open it operates at no load and when closed it operates at full load. The solenoid is controlled over a 15 second duty cycle to vary the capacity. As example, at a 33% (72.66F) call for cooling the solenoid valve would be closed for 5 seconds of the cycle and open for 10 seconds of the cycle. The energy savings of the digital scroll is realized during the no load cycle. During the no load cycle compressor consumes only enough energy to maintain the motor speed while doing no work on the refrigeration cycle. Intended for partially loaded rooms. May be used in conjunction with variable speed fans for even greater energy savings.

Higher Cost Strategies 3. Economization Air side economizers For chilled water or compressorized systems Utilize outside air based on dew point, minimizing compressor and/or chiller usage Water side economizers For chilled water systems Uses water cooled by a cooling tower or a dry cooler (fluid cooler) in low temperature conditions to minimize chiller operation Pumped refrigerant economizers New technology for compressorized systems Uses refrigerant cooled in low temperature conditions to minimize condenser and compressor operation Similar utilization as water side economizers

Annual Utility Cost ($1000’s) Reliable, Low-Maintenance Operation Higher Cost Strategies 3. Economization – Pumped Refrigerant Liebert DSE –The Most Efficient DX Data Center Cooling System Annual Utility Cost ($1000’s) 60% Reliable, Low-Maintenance Operation No water usage No water treatment No outside air contamination No dampers and louvers to maintain Instant, automatic economizer changeover Liebert DSE with EconoPhase Pumped Refrigerant Economizer Cooling PUE 1.3 - 1.05

Thermal System Manager with iCOM Higher Cost Strategies 3. Economization – Pumped Refrigerant Liebert DSE System Overview Thermal System Manager with iCOM Liebert EconoPhase First ever pumped refrigerant economizer Liebert MC Intelligent, high efficiency condensers Liebert Proprietary Data Center Management Intelligence and Optimized Aisle Liebert DSE Indoor Unit Next generation data center cooling system

Liebert DSE System: DX Operation Mode Cooling Mode Outdoor Temp Cooling pPUE SCOP System kW   DX 95º F 1.26 3.8 24.9 3.9 kW Check Valve 4.1 kW MC Condenser Solenoid Valve Refrigerant Pump DSE Check Valve 8.5 kW 8.7 kW 3.2 kW Evaporator 8.5 kW Check Valve 3.4 kW Circuit 2 Electronic expansion valve Circuit 1 8.7 kW Compressor

Liebert DSE System: DX + Pump Operation Mode Cooling Mode Outdoor Temp Cooling pPUE SCOP System kW   DX 95º F 1.26 3.8 24.9 Partial 60º F 1.14 7.0 13.6 EconoPhase 3.9 kW Check Valve 0.1 kW 0.3kW MC Condenser Solenoid Valve Refrigerant Pump DSE Check Valve 9.8 kW Evaporator Check Valve 3.4 kW Circuit 2 Electronic expansion valve Circuit 1 0.0 kW Compressor

Liebert DSE System: Pump Operation Mode Cooling Mode Outdoor Temp Cooling pPUE SCOP System kW   DX 95º F 1.26 3.8 24.9 Partial 60º F 1.14 7.0 13.6 Full 45º F 1.09 10.6 9.0 EconoPhase 3.9 kW Check Valve 4.8 kW 0.4 kW MC Condenser 0.4 kW Solenoid Valve Refrigerant Pump DSE Check Valve 0.0 kW Evaporator Check Valve 3.4 kW Circuit 2 Electronic expansion valve Circuit 1 0.0 kW Compressor

Liebert DSE System: Pump Operation Mode Cooling Mode Outdoor Temp Cooling pPUE SCOP System kW   DX 95º F 1.26 3.8 24.9 Partial 60º F 1.14 7.0 13.6 Full 45º F 1.09 10.6 9.0 30º F 1.05 20.7 4.6 EconoPhase 3.9 kW Check Valve 0.2 kW 0.5 kW MC Condenser 0.5 kW Solenoid Valve Refrigerant Pump DSE Check Valve 0.0 kW Evaporator Check Valve 3.4 kW Circuit 2 Electronic expansion valve Circuit 1 0.0 kW Compressor

Minneapolis, MN Bin Data – EconoPhase, Partial, Compressor

Higher Cost Strategies 3. Economization 1MW of IT load 90°F return air; 20% + redundancy; No humidity control Which is best? It depends on the customer drivers First cost/capital cost Energy savings/PUE Total cost of ownership Redundancy/availability Reliability LIEBERT® DSE

ROI is generally less than 3 years for these strategies Low + Medium + Higher Cost Strategies The Results of Implementation Applying Low + Medium + Higher Cost Strategies Key cooling system features Supply air control 90° F return air with dew point control Rack level sensors Aisle containment Variable Speed Fans Advanced Controls + Pumped Refrigerant Economizers + Variable Capacity Compressors Support Power Usage = 284kW Total Facility Power Usage = 804kW Annualized Facility PUE = 1.55 Total cooling power usage drops from 184kW to 83kW. All CW units have been replaced with new units. 183kW 703kW 1.35 ROI is generally less than 3 years for these strategies

Taking It a Step Further The annualized cooling PUE for cooling only is 1.09 for the last scenario. Why is the overall PUE 1.35? Not implementing virtualization with the servers Inefficiencies in the power distribution: UPS modules PDUs Generators Batteries Switchgear Lighting Lack of monitoring Not having real time data means you cannot react quickly

Taking It a Step Further How can I get an even better cooling PUE? Raise water and air temperatures even higher Implement alternate technologies that remove or greatly reduce cooling Improve server monitoring RISK PUE AVAILABILITY PUE SERVER LOADS

Implementing the Strategies Multiple strategies to consider Low cost Medium cost Higher cost Combination of any or all of the above Implementing any of these strategies can be somewhat difficult Where do I start? What can I implement? Can the current equipment be upgraded? Do I have budget for equipment upgrades? Do I need outside help?

Summary You don’t have to spend a fortune to get energy savings However, to get to a world class level, major changes generally have to be made Total energy consumption needs to be considered along with PUE Focusing only on PUE can increase risk and availability Works with some data center models, but not for all For more information on this topic, please check out the updated vendor neutral Energy Logic 2 white paper, available on the Emerson Network Power website

Or call Uptime Solutions Inc. Thank you! Questions ? jay.ries@emerson.com Or call Uptime Solutions Inc. 937-237-3400