1. Data Center Thermal Management and Efficiency
Jay Ries
Regional Sales Manager
Liebert Thermal Management
Emerson Network Power

2. Agenda Where is energy consumed in the data center?
3/31/2017
Agenda
Where is energy consumed in the data center?
Energy consumption example
Cooling energy consumption breakdown
Strategies for saving energy
Low cost strategies
Medium cost strategies
Higher cost strategies
Taking it a step further (beyond cooling)
Summary

4. Where is Energy Consumed in the Data Center?
52% is consumed by IT equipment
48% is consumed by power and cooling support

5. Energy Consumption Example
Baseline Building design
Existing building
Limitation to physical changes that can be made
Best suited for modifications to existing equipment
Full equipment replacement is a last resort
1MW of facility power usage (all data center)
Baseline Cooling design
Centrifugal water cooled chiller
No economization
Standard computer room cooling units
No variable speed fans or advanced controls
Return air control
45° F chilled water
72° F return air, 50% RH

6. Energy Consumption Example
Power Usage
Processors – 150kW
Other Services – 150kW
Server Power Supply – 140kW
Storage – 40kW
Communication Equipment – 40kW
Cooling – 380kW
UPS – 50kW
MV Transformer and Switchgear – 30kW
Lighting – 10kW
PDU – 10kW
IT Power Usage = 520kW
Support Power Usage = 480kW
Total Facility Power Usage = 1000kW
Annualized Facility PUE = 1.92  Work our way to 1.35
Cooling is the only area that will be modified. In the real world, each variable will have an impact on the others

7. Cooling Energy Consumption Breakdown
Air Cooled System
Water Cooled System
Chilled Water System

8. Low Cost Strategies Implementing best practices
Adjust the unit control methods
Dew point control
Unit operating range
Change to supply air control
Running at higher chilled water temperatures

9. Low Cost Strategies 1. Implementing Best Practices
If you have a raised floor, use it properly. Underfloor resistance wastes energy.
Utilize hot aisle / cold aisle, regardless if you have a raised floor

10. Low Cost Strategies 1. Implementing Best Practices
Get air where it is supposed to go.
Blanking panels
Fix unplanned outside infiltrations and any unecessary gaps in the raised floor
Return plenums to the cooling unit
Isolate the room, particularly if you want to control humidity

11. Low Cost Strategies 2. Adjust Unit Settings
Dew Point
Standard design points used to be 72° return air temperature and 50% relative humidity (RH)
New, more aggressive design points can be 90°+ return air temperature and an unspecified relative humidity
Why shouldn’t you fix at 50% relative humidity (RH)
Dew 72°, 50% = 52°
Dew 95°, 50% = 74°
If the return temperature is increased at a fixed RH, the dew point will rise, requiring the equipment to waste energy to remove moisture that didn’t need to be there in the first place

12. Low Cost Strategies 2. Adjust Unit Settings
Unit operation settings
Expanding the operating range for the temperature and humidity keeps unit components from cycling too frequently.
Higher return air temperatures allow CRAH units to run more efficiently
Capacity increase up to 70% for chilled water units
Capacity increase up to 50% for compressor based units
The more efficiently the units operate, the fewer that are required to control the space, saving energy.

13. Increased Capacity at Higher Temps
3/31/2017
Low Cost Strategies 2. Adjust Unit Settings
Increased Capacity at Higher Temps
Operating at a higher return air temperature increases the CRAC unit’s capacity. This graph shows how the sensible cooling capacity typically increases at different return air temperatures for Liebert CW and DX CRAC units.
75 degrees F is used as the base point, so the percentage increases are the sensible capacity increase above the 75 degree capacity.
These are general curves and the actual values will vary slightly from model to model but not significantly.
We are going to look at the results for our DX units in a second but before we leave this graph I want to look briefly at the CW curve. For a CW unit – shown as red line -- the capacity increase is higher than the DX unit. You must remember however that here are many other components involved in the overall chilled water system that must be sized correctly to produce these results…… This requires a whole separate study.
Next slide

14. Low Cost Strategies 3. Supply Air Control
Supplies a consistent temperature to the cold aisle
Saves energy because it allows the return air temperature to be increased, allowing the CRAH unit to run more efficiently.

15. Low Cost Strategies 4. Running At Higher Water Temperatures
45° chilled water temperature has been the standard design point for many years
Higher chilled water temperatures are starting to become more prevalent
Why? At higher temperatures, there are huge potential savings on the chiller
For every 1 degree increase in the chilled water supply temperature, a 2% energy savings can be realized on the chiller plant
45°chilled water = Baseline
55°chilled water = 20% energy savings

16. Low Cost Strategies The Results of Implementation
Applying Low Cost Strategies
Changes to cooling system
Best practices implemented
Supply air control
50° F chilled water
85° F return air with dew point control
Support Power Usage = 480kW
Total Facility Power Usage = 1000kW
Annualized Facility PUE = 1.92
Total cooling power usage drops from 380kW to 314kW. The number of units stay the same, but some units can be turned off.
414kW
934kW
1.79

17. Medium Cost Strategies
Variable speed fan retrofits (EC Fan / VFD)
Aisle containment
Control retrofits
Rack level sensors

18. Medium Cost Strategies 1. Variable speed fan retrofits (EC Fan / VFD)
Floor-mount cooling fans typically run at 100% rated rpm
By utilizing variable speed technology, fan speed can be varied based upon room conditions
Energy savings based on a single 10HP motor
Fan Speed
Energy Consumed
Savings
100%
8.1kWH
90%
5.9kWH
27%
80%
4.2kWH
48%
70%
2.8kWH
65%
60%
1.8kWH
78%

19. Medium Cost Strategies 2. Aisle Containment
Allows for proper air separation
Able to be done either the hot or cold aisle, though it is easier to retrofit the cold aisle of an existing room
Physical containment varies from simple curtains to a pre-fabricated system designed to match the racks.

20. Medium Cost Strategies 2. Aisle Containment
Containment Strategies
Contained hot aisle
Requires full containment to trap hot air
Can be difficult to retrofit in perimeter designs
Easier to retrofit in row cooling designs
Overhead fire suppression concerns on full containment
Contained cold aisle
Multiple containment options
Doors only
Curtains only
Full containment
Can be easier to retrofit in all cooling designs

21. Medium Cost Strategies 3. Control Retrofits
Allows for upgraded control schemes that save energy
New controls allow units to be networked together
Give more visibility of full system
Eliminate fighting of units, - one cooling while one is heating

22. Medium Cost Strategies 4. Remote Sensors
Usually associated with a control retrofit or a designed scheme through a building management system
Increased visibility and quicker reaction to changes at the rack
Generally applied with supply air sensors
“Bath tub effect”

23. ROI is generally less than 1 year for these strategies
Low + Medium Cost Strategies The Results of Implementation
Applying Low + Medium Cost Strategies
Changes to cooling system
Best practices implemented
Supply air control
+55° F chilled water
+90° F return air with dew point control
+ Remote sensors
+ Aisle containment
+ Variable speed fans
+ Control retrofits
Support Power Usage = 414kW
Total Facility Power Usage = 934kW
Annualized Facility PUE = 1.79
Total cooling power usage drops from 314kW to 184kW. All units are now on, running at a reduced speed.
284kW
804kW
1.55
ROI is generally less than 1 year for these strategies

24. Higher Cost Strategies (Major Capital Expenditures)
Bringing cooling closer to the source
Variable capacity compressors
Economization
Air economizers
Water economizers
Refrigerant Economizers

25. Higher Cost Strategies 1. Bringing Cooling Closer to the Source
Rack-based configuration
Rear door configuration
Row-based configuration
Bring the cooling closer minimizes the need for large fans, reducing energy
Some rear door designs don’t have fans, instead utilizing the server fans to move the air
Generally produce a better sensible cooling to power ratio than a typical system – more cooling for less energy

26. Higher Cost Strategies 1. Bringing Cooling Closer to the Source
3/31/2017
Higher Cost Strategies 1. Bringing Cooling Closer to the Source
Rack Based Solutions
Pump Refrigerant Technology
Dew Point Controlled
Pumped Refrigerant Cooling
Base Infrastructure (160 kw)
Cooling Modules (mix and match)

27. Higher Cost Strategies 1. Bringing Cooling Closer to the Source
Rear Door Solutions
Refrigerant Based Rear Door
Refrigerant based, rear door heat exchanger
A rear door with 10kW to 40kW of cooling
Connect up to 16 doors onto a single pumped refrigerant loop
Designed to accommodate various racks
Energy story – passive door (no fans) that uses the server fans to transfer air through the coil
Performance
Provides room neutral high density rack cooling
Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration
Product Overview:
Extension of the Liebert XD high density cooling product line using pumped refrigerant
A rear door for a computer rack which supplies up to 20 KW of cooling.
Cooling module swings out of the way to allow free access to the back of the server rack
Designed to accommodate various racks
Server fans transfer air from rack front through coils
Open architecture design provides room cooling for back-up protection
Piping connections from the cabinet top
Status:
Limited availability in April 2009
Full availability at end of Q2 2009
Performance
Provides room neutral high density rack cooling
Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration

28. Higher Cost Strategies 1. Bringing Cooling Closer to the Source
Rear Door Solutions
Chilled Water Based Rear Door
Chilled water based, rear door heat exchanger
A rear door with 16kW to 35kW of cooling
Designed to accommodate various racks
Energy story – passive door (no fans) that uses the server fans to transfer air through the coil
Performance
Provides room neutral high density rack cooling
Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration
Product Overview:
Extension of the Liebert XD high density cooling product line using pumped refrigerant
A rear door for a computer rack which supplies up to 20 KW of cooling.
Cooling module swings out of the way to allow free access to the back of the server rack
Designed to accommodate various racks
Server fans transfer air from rack front through coils
Open architecture design provides room cooling for back-up protection
Piping connections from the cabinet top
Status:
Limited availability in April 2009
Full availability at end of Q2 2009
Performance
Provides room neutral high density rack cooling
Applicable for atypical room layouts and rooms without hot aisle / cold aisle configuration

29. Higher Cost Strategies 1. Bringing Cooling Closer to the Source
3/31/2017
Higher Cost Strategies 1. Bringing Cooling Closer to the Source
Row Based Solutions
Row Based Solutions
Precise temperature and Humidity control
12” or 24” wide designs
Air, Water, Glycol and Chilled Water models
Energy efficient, load matching
Digital scroll compressor, % cooling capacity modulation
Variable speed EC plug fans
Performance
Real-time environment control
Automatic performance optimization
Adaptive component monitoring
Adjustable air baffle direction
Lots of features, same frame construction as Sys/3, powder coated panels, IR, pump-down cycle….

30. Higher Cost Strategies 1. Bringing Cooling Closer to the Source
Rack-based configuration
Rear door configuration
Row-based configuration
Fan Energy for 30kW of Cooling
Perimeter Unit = 4.24 kW
Rack Based = 0.54 kW
Row-Based Unit = 1.38 kW
Rear Door = 0.00 kW (no fans)

31. Higher Cost Strategies 2. Variable Capacity Compressors
Digital Scroll Compressors
Matches room load in unlimited step increments
Reliable
Not field repairable. Must be replaced.
4-step Semi-Hermetic Compressors
Matches room load in 4 step increments
Field repairable
Compressors w/ VFD Control
Usually not field repairable.
The digital scroll also operates to save energy but does it in a slightly different. The control band is set up exactly like the 4-step. Since the digital is a proportional capacity from 10 to 100% compressor operation begins at 10% (72.2F) and will increase to precisely match the % call for cooling. The compressor is controlled by a solenoid valve, when the valve is open it operates at no load and when closed it operates at full load. The solenoid is controlled over a 15 second duty cycle to vary the capacity. As example, at a 33% (72.66F) call for cooling the solenoid valve would be closed for 5 seconds of the cycle and open for 10 seconds of the cycle. The energy savings of the digital scroll is realized during the no load cycle. During the no load cycle compressor consumes only enough energy to maintain the motor speed while doing no work on the refrigeration cycle.
Intended for partially loaded rooms. May be used in conjunction with variable speed fans for even greater energy savings.

32. Higher Cost Strategies 3. Economization
Air side economizers
For chilled water or compressorized systems
Utilize outside air based on dew point, minimizing compressor and/or chiller usage
Water side economizers
For chilled water systems
Uses water cooled by a cooling tower or a dry cooler (fluid cooler) in low temperature conditions to minimize chiller operation
Pumped refrigerant economizers
New technology for compressorized systems
Uses refrigerant cooled in low temperature conditions to minimize condenser and compressor operation
Similar utilization as water side economizers

33. Annual Utility Cost ($1000’s) Reliable, Low-Maintenance Operation
Higher Cost Strategies 3. Economization – Pumped Refrigerant
Liebert DSE –The Most Efficient DX Data Center Cooling System
Annual Utility Cost ($1000’s)
60%
Reliable, Low-Maintenance Operation
No water usage
No water treatment
No outside air contamination
No dampers and louvers to maintain
Instant, automatic economizer changeover
Liebert DSE with EconoPhase Pumped Refrigerant Economizer
Cooling PUE

34. Thermal System Manager with iCOM
Higher Cost Strategies 3. Economization – Pumped Refrigerant
Liebert DSE System Overview
Thermal System Manager with iCOM
Liebert EconoPhase
First ever pumped refrigerant economizer
Liebert MC
Intelligent, high efficiency condensers
Liebert Proprietary Data Center Management Intelligence and Optimized Aisle
Liebert DSE Indoor Unit
Next generation data center cooling system

35. Liebert DSE System: DX Operation Mode
Cooling Mode
Outdoor Temp
Cooling pPUE
SCOP
System kW DX
95º F
1.26
3.8
24.9
3.9 kW
Check Valve
4.1 kW
MC Condenser
Solenoid Valve
Refrigerant Pump
DSE
Check Valve
8.5 kW
8.7 kW
3.2 kW
Evaporator
8.5 kW
Check Valve
3.4 kW
Circuit 2
Electronic expansion valve
Circuit 1
8.7 kW
Compressor

36. Liebert DSE System: DX + Pump Operation Mode
Cooling Mode
Outdoor Temp
Cooling pPUE
SCOP
System kW DX
95º F
1.26
3.8
24.9
Partial
60º F
1.14
7.0
13.6
EconoPhase
3.9 kW
Check Valve
0.1 kW
0.3kW
MC Condenser
Solenoid Valve
Refrigerant Pump
DSE
Check Valve
9.8 kW
Evaporator
Check Valve
3.4 kW
Circuit 2
Electronic expansion valve
Circuit 1
0.0 kW
Compressor

37. Liebert DSE System: Pump Operation Mode
Cooling Mode
Outdoor Temp
Cooling pPUE
SCOP
System kW DX
95º F
1.26
3.8
24.9
Partial
60º F
1.14
7.0
13.6
Full
45º F
1.09
10.6
9.0
EconoPhase
3.9 kW
Check Valve
4.8 kW
0.4 kW
MC Condenser
0.4 kW
Solenoid Valve
Refrigerant Pump
DSE
Check Valve
0.0 kW
Evaporator
Check Valve
3.4 kW
Circuit 2
Electronic expansion valve
Circuit 1
0.0 kW
Compressor

38. Liebert DSE System: Pump Operation Mode
Cooling Mode
Outdoor Temp
Cooling pPUE
SCOP
System kW DX
95º F
1.26
3.8
24.9
Partial
60º F
1.14
7.0
13.6
Full
45º F
1.09
10.6
9.0
30º F
1.05
20.7
4.6
EconoPhase
3.9 kW
Check Valve
0.2 kW
0.5 kW
MC Condenser
0.5 kW
Solenoid Valve
Refrigerant Pump
DSE
Check Valve
0.0 kW
Evaporator
Check Valve
3.4 kW
Circuit 2
Electronic expansion valve
Circuit 1
0.0 kW
Compressor

39. Minneapolis, MN Bin Data – EconoPhase, Partial, Compressor

40. Higher Cost Strategies 3. Economization
1MW of IT load
90°F return air; 20% + redundancy; No humidity control
Which is best? It depends on the customer drivers
First cost/capital cost
Energy savings/PUE
Total cost of ownership
Redundancy/availability
Reliability
LIEBERT® DSE

41. ROI is generally less than 3 years for these strategies
Low + Medium + Higher Cost Strategies The Results of Implementation
Applying Low + Medium + Higher Cost Strategies
Key cooling system features
Supply air control
90° F return air with dew point control
Rack level sensors
Aisle containment
Variable Speed Fans
Advanced Controls
+ Pumped Refrigerant Economizers
+ Variable Capacity Compressors
Support Power Usage = 284kW
Total Facility Power Usage = 804kW
Annualized Facility PUE = 1.55
Total cooling power usage drops from 184kW to 83kW. All CW units have been replaced with new units.
183kW
703kW
1.35
ROI is generally less than 3 years for these strategies

42. Taking It a Step Further
The annualized cooling PUE for cooling only is 1.09 for the last scenario. Why is the overall PUE 1.35?
Not implementing virtualization with the servers
Inefficiencies in the power distribution:
UPS modules
PDUs
Generators
Batteries
Switchgear
Lighting
Lack of monitoring
Not having real time data means you cannot react quickly

43. Taking It a Step Further
How can I get an even better cooling PUE?
Raise water and air temperatures even higher
Implement alternate technologies that remove or greatly reduce cooling
Improve server monitoring
RISK
PUE
AVAILABILITY
PUE
SERVER LOADS

44. Implementing the Strategies
Multiple strategies to consider
Low cost
Medium cost
Higher cost
Combination of any or all of the above
Implementing any of these strategies can be somewhat difficult
Where do I start?
What can I implement?
Can the current equipment be upgraded?
Do I have budget for equipment upgrades?
Do I need outside help?

45. Summary
You don’t have to spend a fortune to get energy savings
However, to get to a world class level, major changes generally have to be made
Total energy consumption needs to be considered along with PUE
Focusing only on PUE can increase risk and availability
Works with some data center models, but not for all
For more information on this topic, please check out the updated vendor neutral Energy Logic 2 white paper, available on the Emerson Network Power website

46. Or call Uptime Solutions Inc.
Thank you!
Questions ?
Or call Uptime Solutions Inc.