The Economics of Data Center Air Flow Containment

Course Description This course will explain: The acceptable range of equipment inlet conditions The value of supply air set points versus return air set points The value for maximizing the separation between supply air streams and return air streams. The benefits for maintaining air separation including: elimination of equipment hot spots elimination of bypass and re-circulation waste reduction in fan energy improvements in cooling coil efficiency ability to support trends toward higher rack heat density improvement in chiller plant efficiency access to increased hours of economization cooling increased overall design flexibility

Learning Objectives At the end of this program, you will be able to: Discuss the three basic methods for achieving air separation through hot aisle containment, cold aisle containment, and vertical exhaust ducts. State the relative advantages and disadvantages for the different separation topologies along with application appropriateness. Quantify the cost-benefits for a well-designed air flow management. Explain the overall value of good air flow management, and how it can be achieved using design elements that can enable or compromise best results.

What is the source of your pain? Hot spots? Cost for energy? Growth and/or technology refresh constrained? High cost of eliminating hot spots or reducing energy costs or positioning for growth?

Uptime Institute 2012 Data Center Survey

Limitations of Hot Aisle – Cold Aisle BACK FRONT BACK FRONT BACK 15000 CFM per air handler 300 CFM for 2kW cabinet 1500 CFM for 10kW cabinet 6000 CFM for 2 rows of 10 2kW cabinets 30000 CFM for 2 rows of 10 10kW cabinets Hot air recirculation from the hot aisle into the cold aisle Bypass airflow under the cabinets

Recommended Environmental Envelope What is the target? The New ASHRAE Recommended Environmental Envelope for Class 1 Electronics 2004 Version 2008 Version Low End Temperature 20°C (68°F) 18°C (64.4°F) High End Temperature 25°C (77°F) 27°C (80.6°F) Low End Moisture 40% RH 5.5°C DP (41.9°F) High End Moisture 55% RH 60% RH & 15°C DP (59°F DP) 6

Typical Set Point Management 72°F (22°C) 55°F (13°C) 14

Typical Set Point Management Mixing of supply air and return air in typical data center 15

Typical Set Point Management 72°F (22°C) 55°F (13°C) Hot spots Return air could cool an entire data center

All the hot return air is isolated from the supply air Supply Set Point Management All the hot return air is isolated from the supply air No hot spots. No concern about return air temperature. 17

What is Hot Air Isolation -Completely preventing hot and cold air from mixing -Completely capturing either hot or cold air -100% is probably unrealistic -leakage is important -many design trade-offs -Several design strategies

Benefits of Hot Air Isolation Eliminates hot spots Supports 4x higher heat and power densities (6 kW to 30+kW)! Effectively utilizes 100% of supplied air and reduces chilled air waste! Transfers server equipment ΔT directly back to the HVAC unit! Increase chiller efficiency through higher set points! Can increase economization through higher set-points! Allows supply air delivery from ceiling, wall or floor.

Elimination of Hot Spots 86 83 81 78 75 72 70 67 64 62 59 Temp is set based on hot spots causing great inefficiencies.

Benefits of Hot Air Isolation Eliminates hot spots Supports 4x higher heat and power densities (6 kW to 30+kW)! Effectively utilizes 100% of supplied air and reduces chilled air waste! Transfers server equipment ΔT directly back to the HVAC unit! Increase chiller efficiency through higher set points! Can increase economization through higher set-points! Allows supply air delivery from ceiling, wall or floor.

High Density Cooling 4 IBM H 8852 Blade Servers 25kW actual heat load We have contracted third party CFD testing for high density loads with IBM and HP blade servers. These reports show pressure distribution and the low pressure readings inside the vertical exhaust ducts indicate the direction of flow out of the cabinets and illustrate why fans are not required in the CPI Passive Cooling Solution. We have successfully tested up to 32kW in a cabinet in our thermal lab and a 45kW cabinet was successfully tested in a KyotoCooling data center in The Netherlands. Today, the solution is capable of dissipating more heat than equipment can actually be fit into a 45U cabinet. 4 IBM H 8852 Blade Servers 25kW actual heat load 4 HP c7000 Blade Servers 22kW actual heat load

Benefits of Hot Air Isolation Eliminates hot spots Supports 4x higher heat and power densities (6 kW to 30+kW)! Effectively utilizes 100% of supplied air and reduces chilled air waste! Transfers server equipment ΔT directly back to the HVAC unit! Increase chiller efficiency through higher set points! Can increase economization through higher set-points! Allows supply air delivery from ceiling, wall or floor.

6 Stulz CCD-1800-CWE CRAH units (N+1) Elimination of Excess Supply Air 600kW IT load 6 Stulz CCD-1800-CWE CRAH units (N+1) Standard Hot Aisle – Cold Aisle Data Center Air handlers operate @ 83% capacity Maximum inlet temperature range = 58.6°F – 79.7°F CRAH fan energy = 32.1 kW or 280,937 kW hours/year Data Center with Hot and Cold Isolation Air handlers operate at 73% capacity Maximum inlet temperature range = 75°F – 76°F CRAH fan energy = 21.6kW or 189,216 kW hours/year 91,721 kW hours energy saved by eliminating need to produce excess bypass supply air These figures are from a case study showing the difference between a very well-designed and well-managed hot aisle – cold aisle data center and a data center with a good hot and cold isolation scheme, with a 600kW IT load and N+1 cooling units. The 10% reduction in air volume produces a 32% reduction in fan energy

Benefits of Hot Air Isolation Eliminates hot spots Supports 4x higher heat and power densities (6 kW to 30+kW)! Effectively utilizes 100% of supplied air and reduces chilled air waste! Transfers server equipment ΔT directly back to the HVAC unit! Increase chiller efficiency through higher set points! Can increase economization through higher set-points! Allows supply air delivery from ceiling, wall or floor.

Supply Air Temperature Return Air Temperature CRAC Efficiency with Supply/Return Isolation Cooling Unit Supply Air Temperature Return Air Temperature Cooling Capacity Stulz CCD-600-CWE Labeled as 10 Tons 60˚ F 70˚ F 7.8 tons 90˚ F 15.5 tons 105˚ F 20.7 tons Stulz CCD-1800-CWE 30 Tons 23.0 tons 46.0 tons 61.3 tons Another effect of ducted return air systems is a much higher return air temperature. This is not necessarily a bad thing. As you can see here, the higher return air temperatures actually bumps up the performance ratings of the cooling units. This actually only works with chilled water units. DX units have a more or less fixed temperature difference that they work with so higher return temperatures will actually drive up the supply temperatures. With chilled water CRACs, however, actually cooling performance efficiency increases with higher return air temperatures. In addition, if you recall the sensible cooling equation from the beginning of the presentation, the temperature difference is the divisor, so as that temperature rise increases, the amount of air required for the same amount of cooling decreases. We have seen data centers go from experiencing significant over-heating problems to actually putting half of their cooling unit capacity in reserve merely be effectively managing these variables. Almost a 3x improvement in cooling unit efficiency!

Benefits of Hot Air Isolation Eliminates hot spots Supports 4x higher heat and power densities (6 kW to 30+kW)! Effectively utilizes 100% of supplied air and reduces chilled air waste! Transfers server equipment ΔT directly back to the HVAC unit! Increase chiller efficiency through higher set points! Can increase economization through higher set-points! Allows supply air delivery from ceiling, wall or floor.

38% savings = $168,521 Chiller Efficiency at Higher Set Points 38% Energy Savings 38% savings = $168,521 1MW example Chart from ASHRAE Data Center Design and Operation Book#6: Best Practices for Datacom Facility Energy Efficiency

Benefits of Hot Air Isolation Eliminates hot spots Supports 4x higher heat and power densities (6 kW to 30+kW)! Effectively utilizes 100% of supplied air and reduces chilled air waste! Transfers server equipment ΔT directly back to the HVAC unit! Increase chiller efficiency through higher set points! Can increase economization through higher set-points! Allows supply air delivery from ceiling, wall or floor.

Increase Economization Through Better Separation Between Supply and Return Air 538.5kW Load

Increase Economization Through Better Separation Between Supply and Return Air 538.5kW Load

Increase Economization Through Better Separation Between Supply and Return Air 538.5kW Load

Benefits of Hot Air Isolation Eliminates hot spots Supports 4x higher heat and power densities (6 kW to 30+kW)! Effectively utilizes 100% of supplied air and reduces chilled air waste! Transfers server equipment ΔT directly back to the HVAC unit! Increase chiller efficiency through higher set points! Can increase economization through higher set-points! Allows supply air delivery from ceiling, wall or floor.

Design Flexibility Return Supply

Strategies for Air Isolation Hot Aisle Containment (HAC) Cold Aisle Containment (CAC Hot air isolation (chimney or VED) Now we can move thermostat from return air to supply air and supply a number we want the air to be.

Containment Starts Inside the Cabinet Floor Tile Cut-out Grommets An Uptime Institute survey of data centers around the country found the average data center had 2.7 times the cooling capacity needed to cool the heat load, but still had numerous hot spots. That is primarily the result of bypass air – chilled air that is bypassing the computer equipment because improperly located floor openings and unsealed openings, such as cable access cut-outs. The photographs show various cut-out sealing products, but there are all kinds of ways to address bypass air, including foam inserts, plenum pillows and cardboard and duct tape. I worked with a data center out in west Texas that was frying blade PCs. We showed them where they were losing their cooling capacity and when they completed their corrective actions, they eliminated all their hot spots and turned off one of two 18 ton CRACs to hold it in reserve for back-up and maintenance. The duct tape and cardboard worked fine for these folks because they were using blade PCs, so the vast majority of the moves, adds and changes were restricted to just swapping blades, minimal or no churn on the cabling. 50% to 80% bypass air observed in many sites has three impacts—all unwanted Hot spots Cooling unit incapacity Incapacity: Any condition that prevents cooling unit from delivering rated capacity—dirty filters, fouled surfaces, leaks, latent cooling, etc. Latent cooling penalty Sealing off cable access ports in floor tiles is recommended by TIA-942 and will be a best practices requirement of the pending BICSI data center standard. Available solutions include brush grommets, sliding metal panels, rubber grommets, plenum pillows, and even duct tape and cardboard. One small data center out in Odessa, Texas used the cardboard and duct tape approach so effectively that they relegated one of their two CRACs strictly to back-up redundancy, after previously experiencing heat-related equipment failures. Application was blade PCs, so MAC activity was centered on blades rather than cabling, so cardboard and duct tape was effective. Obviously, in a high churn environment for cabling, you want something a little more flexible. Cabinet Air Dams Blanking Panels

Cabinet Air Re-Circulation CFD Model Thermal Imaging Photograph Here’s what I was talking about with hot air coming around the sides of equipment. Hot Air re-circulates around the side of the cabinet also. [trace airflow] Hot air mixes with cold air increasing the temperature of air at the server inlet. That’s why blank filler panels by themselves are not sufficient. This CFD model, by the way, was produced for us by the folks from Fluent – we play no favorites – from data we collected in our own test lab. I’ll be showing you several graphics from this test, so you should know some of the basics from the test. We had server replicator heat loads of just over 9kW. The heat loads consumed around 1600 CFM. We had AC pumped under the access floor and got out a little over 500 CFM through the perf tile in front of the cabinet. The structure around the cabinet was designed to replicate a cabinet in the middle of a row of similar cabinets with another row of cabinets both in front and behind it. [next slide] 1” rail set-back = 5+ RMU open space between hot area and cold area

Cold Aisle Containment 538.5kW Load

Hot Aisle Containment

with Cabinet Vertical Exhaust Duct Hot Air Isolation with Cabinet Vertical Exhaust Duct 538.5kW Load

Hybrid Containment with Chimney Cabinets and Hot Aisle Containment 538.5kW Load

What Is the Best Containment? DataCenter 2020: Hot-aisle and Cold Aisle Containment Efficiencies Reveal No Significant Differences Copyright © 2011 Intel Corporation. The DataCenter 2020 is a joint T-Systems and Intel data center test laboratory in the Munich-based Euroindustriepark. www.datacenter2020.com

Hot Aisle Containment Easy integration of fire suppression Hot aisle working environment @ 90 – 130⁰F Flexible supply air delivery architectures Better low density thermal ride-through Raised floor not required Pod deployment Compatible with row-based cooling

Cold Aisle Containment Data Center working environment @ 90 - 130⁰F Better high density thermal ride-through Easier retrofit of working data center Raised floor (usually) Separate “volume” for fire suppression Pod deployment Possibly sensitive to over-pressure Compatible with row-based cooling

Cabinet Containment Working environment @ 74 - 80⁰F 16’ pitch may impact fire suppression Raised floor not required Thermal ride-through (variable dependent) Cabinet deployment 3050 CFM density threshold Flexible supply air delivery architectures

Case Study 300kW IT load 2000 square feet 150 watts per square foot 60 cabinets 1,025,567 BTU/hr Case Files

Cooling Cost Roll-up 12kW to operate 5 cooling units @ 64% capacity 41.8kW to operate chiller @ 65°F H2O leaving temperature 0.06 kW for fresh air (N+1) 0.56kW for electrical room (N+1) 0.06kW for battery room (N+1) 15kW for substation (N+1, but not simultaneously running redundancy 608,119 kW hours per year $60,812 cooling budget at $0.10 per kW hour (1.23 COOLING ONLY PUE)

Savings Comparison 44% cooling energy savings over original data center 35.8% reduction in chiller plant energy 73.7% reduction in CRAH fan energy 30% cooling energy savings over optimized hot aisle – cold aisle data center 30.8% reduction in chiller plant energy 52% reduction in CRAH fan energy No economizer in any scenario

Benefits of Airflow Management by Containment Eliminates hot spots Suitable for low, medium and high density Eliminates wasted capacity and saves fan energy Improves cooling unit efficiency Supports supply air set points Reduced chiller plant energy use Increased “free cooling” utilization Facilitates greater design and deployment flexibility Now we can move thermostat from return air to supply air and supply a number we want the air to be.

Questions? Thank You For Your Time This concludes The American Institute of Architects Continuing Education Systems Course Glen ter Horst (708) 990-0798 gterhorst@chatsworth.com