2. Electronic Cooling Solutions Inc.
Thermal management consulting company
Located in the heart of Silicon Valley
Provide solutions for thermal design problems
Use of experience, modeling & experimental methods in the design process
Clients include over 60 companies

3. Outline Introduction Need for Innovative Cooling Solutions Objective
Scope of this Presentation
Cooling Solutions – Air-based Cooling Systems
– Water-based Cooling Systems
– Refrigerant-based Cooling Systems
Comparative Analysis
Conclusion
Vendor Contacts

4. Introduction - Power Density and Heat Load Trends
Electrical Power = Waste Heat
Increase in power densities from the CPU-level to the System-level
Component level
Board-level
System-level
Rack-level
Room-level
Current power density at the rack level = 1 to 3 kilowatts
(up to 30 kilowatts per rack in two to four years - Hannemann and Chu ‘07)

5. Need for Innovative Cooling Solutions at Rack-Level
Datacenter TCO is characterized on a per rack basis
Addition of newer and higher powered equipment in existing datacenters
Hotspots in datacenter resulting from high-density servers
Hotspots resulting from unavailability of cooling air from CRAC units
Design shortcomings within the rack result in inefficient cooling

6. Objective
Cooling of equipment in rack using airflow and impact of minor design changes for better cooling
Cooling of high density equipment using water-based cooling techniques
Cooling of high density equipment using refrigeration-based cooling techniques
Provide vendor data for the above mentioned products

7. Scope of this Presentation
Limited to discussion of cooling solutions only at the rack level
Closed racks and commercially available products
Discussion on Component-level, Board-level and Room-level cooling can be personally consulted

8. Airflow-based Cooling Techniques

9. Cooling of Racks with Conditioned Air
Most easy to implement and maintain
Limitation based on cooling capacity, acoustics and power consumption
Detailed analysis could improve efficiency of air cooling
Application of Computational Fluid Dynamics (CFD)
Pressure drops and airflow patterns
Determine by-pass air
Determine areas of re-circulation
Determine failure modes of cabinet fans
Requires testing to develop confidence in models

10. Cooling of Racks with Conditioned Air
Airflow Enhancement in Racks
Based on CFD Analysis
Sample case study 1
Sample case study 2
Cabinet powered fans – airflow layouts
Best practices
Airflow enhancing products

11. Sample Case Study 1 Study done by Electronic Cooling Solutions Inc.,
42U Cabinet fitted with twenty one 2U units
15.75 kilowatts per rack
Inlet temperature of 40 deg C
Airflow – 1932 CFM
10,000 ft Altitude Conditions
Simplified cabinet shown here
Objective was to optimize the cabinet for better cooling/use of higher powered equipment

12. Sample Case Study 1 (Contd …)
Higher temperatures at the inlet sides
Inlet Temperature
Re-circulation of Flow

13. Sample Case Study 1 (Contd …)
Added vertical blockages
Open space between the rack rails and cabinet sides
Vertical blockages between the rack rails and cabinet sides

14. Sample Case Study 1 (Contd …)
Added blockage above the topmost unit of the rack
Top of 2U server
Area above the topmost 2U rack is blocked
Top of rack cover meant for passing cables from front to rear

15. Sample Case Study 1 (Contd …)

16. Comparison of Inlet Temperatures
Sample Case Study 1 (Contd …)
Comparison of Inlet Temperatures
No blockages
Blocked passages

17. Sample Case Study 1 (Contd …)
Fixed Exhaust Temperature (deg C) 55 60
No Blockages
With Blockages
Mean Inlet Temperature (deg C)
43.9
41.2
Fixed Airflow
92 CFM
Power (W) per unit
397
493
575
675
Total power of Rack (kW)
8.3
10.3
12.1
14.1
Current study shows 17 to 25 % increase in power dissipation based on exhaust temperature.
By blocking re-circulating flow, it is possible to use higher powered equipment in the rack.
Blocking can be done by employing Brush Strips.
Avoid using larger racks with rails set to lower rack width settings (Using 23” rack with rails set to hold 19” equipment)

18. Sample Case Study 2 Study done by Electronic Cooling Solutions Inc.,
Container wall
Study done by Electronic Cooling Solutions Inc.,
Racks placed in containers
Create airflow model of blowers
Evaluate alternate designs for blower module
Heat Exchanger
2U Servers
2U Servers
Blower Module
Rack Model
Symmetry Walls

19. Testing to collect data for modeling
Sample Case Study 2
Testing
Module 1 – 860 CFM
Module 2 – 1024 CFM
Apprx 20% increase in airflow with the re-designed baffles and perforated casing
Testing to collect data for modeling

20. Cooling of Racks with Conditioned Air
Airflow Enhancement in Racks
Based on CFD Analysis
Sample case study 1
Sample case study 2
Cabinet powered fans – airflow layouts
Best practices
Airflow enhancing products

21. Airflow Enhancers – Cabinet Powered Fans
Supply Air
Ceiling
Raised Floor
Front In – Top Out
Front or Footprint Inlet
Supply Air
Raised Floor
Front In – Rear Out

22. Airflow Enhancers (Cabinet Best Practices)
Supply Air
Raised Floor
Raised Floor
Supply Air

23. Airflow Enhancers (Cabinet Best Practices (Contd…))
Dropped Ceiling
Dropped Ceiling
Raised Floor
Raised Floor
Supply Air
Supply Air

24. Airflow Enhancing Products
APC 2U Rack Air Distribution
Delivers air directly from the raised floor into the rack inlet
Minimizes top-bottom inlet temperature distribution
Allows rack loads up to 3.5 kilowatts per rack
Bottom-Top Rack Air Distribution Unit
Side Rack Air Distribution Unit
Roof Air Removal Unit
Application View
Images: APC (

25. Airflow Enhancing Products (Contd…)
Rittal Side Breathing Air Baffle System
Rittal Enclosure Blower
Images: Rittal (
Higher density rear door rack air removal unit
Allows rack loads up to 16.5 kilowatts /14 kilowatts per rack
Challenges in obtaining flow through tiles in the datacenter
Liebert XDA
APC
Images: and

26. Water-based Cooling Techniques

27. Water-based Cooling Basis: QLOAD = mCp DT = rVCp DT
(Water has 3000 times higher heat carrying capacity than air)
Chilled water from building supply
Cooling high density servers up to 70 kilowatts per rack
Lower energy cost as some of the CRAC units can be removed
Avoid hotspots due to high power-density equipment
Possible to have redundant systems (Chillers, pumps, piping, and power supply) to avoid downtime
Importance of CDU
Electrically conductive, corrosiveness and high flow rates

28. Pure Water-based Cooling
Combination of Air and Water Cooling
Pure Water-based Cooling
Heat Exchanger
Heat Exchanger
Raised Floor
Raised Floor

29. Water-based Cooling
Cooling system design by Naissus Thermal Management Solutions
Heat removal of 20+ kilowatts
Closed liquid loop with bottom mounted fin and tube heat exchanger
Thermal test done with 5 blade servers
Water from Chiller
Water sent to Chiller

30. Temperature Distribution inside the Rack
Water-based Cooling
Temperature Distribution inside the Rack

31. Water-based Cooling Cooling system design by Vette Corp.
Heat removal of up to 30 kilowatts
Rear door closed loop liquid heat exchanger designed by IBM
Currently available only for IBM Enterprise Rack
Available from Rittal for retro-fit designs
IBM Rear Door Heat Exchanger
Images:
Pressure drop across the heat exchanger for a typical 1U fan setup

32. Heat Exchanger and Fan Assembly
Water-based Cooling
Cooling system design by APC.
Heat removal of up to 70 kilowatts
Controlled in-row cooling
Row air containment
Modularity
Similar designs from HP (35 kilowatts)
Similar concepts available from Rittal (30 kilowatts)
Similar concepts available from Liebert (8 kilowatts and 17 kilowatts)
Heat Exchanger and Fan Assembly
Front View
Rear View
Images:

33. Refrigerant-based Cooling Techniques

34. Refrigerant-based Cooling
Phase change (latent heat transfer)
Electronics-safe
Low flow rates and non-corrosive
Some systems are stand-alone and hence flexible
CRAC units are the most common ones
Chilled water from building supply may be used for supplemental cooling
Expensive ( comparable to water+ additives)

35. Refrigerant-based Cooling
Cooling system design by Liebert
XDF- Cooling capacity of 14 kilowatts
Stand-alone unit
Liebert XDF Self Contained Unit
Images:

36. Refrigerant-based Cooling
Cooling system design by APC.
Heat removal of up to 43 kilowatts
Modularity
Rack air containment
APC In-Row Cooling

37. Refrigerant-based Cooling
Cooling system design by Liebert
XDV- Rack mount air conditioners (10 kilowatts) - Almost no floor space required
XDH-Rack Cooling capacity up to 30 kilowatts
Also available from Rittal
Liebert Roof Mount Cooling
Liebert In-Row Cooling
Rittal Rear Door Hx
Images:

38. Liquid Based Touch Cooling

39. Water/Refrigerant-based Touch Cooling
Direct contact cooling combined with chip cooling
Remove heat at the source
Available from:
Clustered Systems
Rittal (Power electronics)
SprayCool (20 to 30KW)
Cold plate with Liquid Cooling
Liquid Cooling of Boards
Images:
Images:
Spray Cooling
Images:

40. Refrigerant-based Touch Cooling
Cooling system design by Thermal Form and Function
Pumped liquid multiphase cooling
Heat removal of up to 10 kilowatts
per evaporator (Modular)
Designed for retro-fit applications
Air/Water cooled condenser unit can be used
Thermal Form and Function Refrigeration Unit
Images:
Two Phase Flow

41. Comparison of Cooling Techniques

42. Comparative Analysis Study by Hannemann and Chu – Interpack ’07
Comparative study of cooling technologies with a model datacenter
Capital Expenditure of Cooling Equipment
Area required for Cooling Equipment
Power Consumption of Cooling Equipment

43. Conclusion
Reviewed innovative and commercially available technologies for cooling racks
Discussed design approaches with use of CFD to maximize performance of air cooling
Reviewed products and techniques for enhancing airflow within a rack
Reviewed cooling of high density equipment using chilled water and refrigerant
Selection of cooling strategy will depend on the specific requirements of the client

44. Vendor Contacts APC – Morrison, Harold Wells Associates (925-355-9900)
Rittal – Sales ( )
Liebert – Frank Stone ( )
Spray Cool – Sales ( )
Clustered Systems – Phil Hughes ( )
Trox AITCS - Thomas Hudgens ( )
Thermal Form and Function – Joe Marsala ( )
Vette Corp - Skye Emerson ( )
Naissus Thermal Management Solutions - Mirko Stevanovic ( )

45. References Product websites and communication with vendors
Hannemann, R and Chu, H., (2007), “Analysis of Alternative Data Center Cooling Approaches”, ASME Interpack 1176, Vancouver, BC.

46. Acknowledgement Speck Design
Our colleagues at Electronic Cooling Solutions:
Khyati Varma
Ceferino Sanchez
Adriana Romero
Sridevi Iyengar