Thermal Design Project Final Report John Wallerich Principal Engineer Wallerich Group, LLC

Agenda 1.Introduction 2.Executive Summary 3.Project Summary 4.Recommendations 5.CFD models Wallerich Group, LLC

1. Introduction 3 statistical analysis actual experiences in building and testing data centers CFD modeling [customer] Wallerich Group, LLC Datacenter design standards Most efficient cooling design Develop and Provide Project tasks Wallerich Group, LLC

3. Project Summary Computer Room layout Rack Optimization Horizontal cabling distribution Airflow design Rack Power Distribution CRAC design Raised Floor design Diffusers (perforated floor tiles) 4 Electro-mechanical infrastructure UPS Generator PDU Rack power distribution systems Chilled water generation Chilled water distribution system Network infrastructure Infrastructure management Building structural design In-Scope Out-of-Scope Wallerich Group, LLC

Design Components Analysis results 5 The letter/number designations and solution consensus came from the initial analysis session documents and are maintained for the sake of document continuity. These design decisions were agreed to by consensus and are incorporated into the CFD modeling analysis Wallerich Group, LLC

A1:Airflow Design DiagramOptionPros and Cons RoomMerit: Assure N+1 configuration. Room level cooling was adequate for mainframe environments, Demerit: but is not cost effective to cool high density equipment cost effectively. Very inefficient and will not deliver sufficient cooling for high density environments. Row Merit: Assure N+1 configuration. Optimize capital investment and available space. Reduce maintenance time and facilitate maintenance work. Quantify the cooling system performance. Maximize the operational efficiency by matching capacity to load. Maximizes efficiency of cooling system infrastructure Rack Merit: Assure N+1 configuration. Merit: Rack level cooling devices are much more expensive than Row type cooling, and lower revenue potential. N+1 configuration reduces # of productive racks Wallerich Group, LLC

A2:Cooling Architecture DiagramOptionPros and Cons Primary cooling Merit: Centralized chilled water system ensures maximum efficiency and redundancy that supports Tier 3. Demerit: Must support entire critical load with full redundancy (chiller, plumbing, CRAC) Primary Cooling & Supplemental Cooling Merit: Supplemental cooling is designed to provide options for datacenters who have exhausted their primary cooling capacity. Demerit: Full critical load must be supported with supplemental option disabled, or supplemental cooling design must also support critical load without use of primary cooling system.. Supplemental cooling is designed to provide options for datacenters who have exhausted their primary cooling capacity and voids Tier 3 capability. It is more costly and may reduce cabinet count. 7 Primary cooling system Supplemental cooling system Wallerich Group, LLC

A3:Primary cooling Architecture DiagramOptionPros and Cons Air Conditioner Demerit: This design does not provide N+1 redundancy. Instead of assessing heat rise characteristics there are other issues regarding using AC units that need to be decided first. One of those assumption is that racks will never exceed 2.5KW over the life of facility (.10 to 12 years). Initial investment is less than chilled water system, but may not support Tier 3 for the life of the facility (2.5kW may increase) Chilled Water/AHU Merit: Supports Tier 3 requirements for all computing models. Provides high availability option when using excess chilled water storage and achieves required cooling capacity for all rooms. Provides most cost effective solution while supporting Tier 3 specification. Allows for use of outside cold air to reduce energy consumption. 8 Chiller CRAC Cooling Medium Cooling Tower CRAC Chilled Water Wallerich Group, LLC

A4:Supplemental cooling DiagramOptionPros and Cons XD Merit: XD is used when primary cooling system is exhausted and provides cold air to a very small area Demerit: XD is a Liebert solution that does not support Tier 3 availability. More costly to operate and provides cold air to a very small area. Cannot provide total cooling and is used when primary cooling system is exhausted. Water to Rack Merit: Cools hot air as it exits equipment Demerit: Tier 3 capability assumes redundant chilled water system. May be required for very high density, but at higher risk and costs.. Makes accessing equipment more difficult, and is typically vendor specific. Rack Cooling Merit: Cools hot air as it exits rack. Demerit: Rack cooling devices must support entire room if primary CRAC failed, however they can not support this requirement. Rack cooling replaces a revenue generating rack and replaces it with a device that only cools racks on either side. Not cost effective. Outside Air Merit: No risk to Tier 3. Can have a significant impact on energy consumption during cold days. Options are being evaluated. 9 Rack Water Chilled Water Cooling Tower Rack inside Room outside Wallerich Group, LLC

A5:CRAC Grouping DiagramOptionPros and Cons No Grouping Merit: Assure “Concurrently Maintainable” and “Redundancy N+1” configuration. Reduce maintenance time and facilitate maintenance work Grouping Merit: Assure “Concurrently Maintainable” and “Redundancy N+1” configuration. Demerit: Recommended when using single chilled water supply (not recommended) 10 Group A Group B Group C Wallerich Group, LLC

B1:Cold Air Supply DiagramOptionPros and Cons Under Floor Merit: Supports Tier 3 model. Optimize cooling capacity by supplying cold air efficiently. Allows flexibility of air delivery and cable plenum. Over head Merit: Supports Tier 3 model Demerit: Less flexible to change. More cost to install. Use when building height is limited. Side Feed Merit: Supports Tier 3 model Demerit: Interrupts controlled airflow. Used only as supplemental cooling when unable to used under-floor plenum Combination Merit: Supports Tier 3 model Demerit: No advantage, and would be used when primary cooling design is inadequate. 11 CRAC rack CRAC rack Wallerich Group, LLC

B2:Raised Floor Height DiagramOptionPros and Cons 600mm /24inch Demerit: Volume of air required for high density and uneven pressure distribution inhibits high efficiency cooling model. With underfloor power cabling, obstacles have more impact on airflow and cooling delivery. 750mm /30inch Demerit: Typical Raised Floor Height is 30-36inch which is most directly correlated to power density. May need custom design to control air pressure distribution and avoid hot spots. 900mm /36inch Merit: Typical Raised Floor Height is 30-36inch which is most directly correlated to power density. Provides more even pressure distribution and higher volume. 12 Rack 600mm 750mm 900mm Wallerich Group, LLC

B3:Perforated Tile type DiagramOptionPros and Cons Center cut Demerit: Only for specific rack (ex: Verari) Fixed Merit: Reduce maintenance time Prevent human error Variable Damper Demerot: Increase possibility of human error that could damage equipment and increases costs (each tile costs more) Wallerich Group, LLC

B4:Perforated Tile opening DiagramOptionPros and Cons 20-50% open Merit: Perforated tile openings will be determined by CFD modeling. Range likely to be 40-50% open. >50% Open Merit: Perforated tile openings will be determined by CFD modeling <20% open Merit: Perforated tile openings will be determined by CFD modeling Wallerich Group, LLC

B5:Rack Configuration DiagramOptionPros and Cons Random Very poor control of hot air, increased likelihood of overheating equipment and higher energy costs Hot Aisle / Cold Aisle Maximize efficiency of cooling system through isolation of cold and hot air Wallerich Group, LLC

B6:Rack Air Control DiagramOptionPros and Cons Blank Panels Merit: Maximize CRAC cooling capability. Required to isolate hot air and cold air. Side Panels Merit: Maximize CRAC cooling capability and required for cabinet level security 16 X X Rack side Blank Panels Rack side Side Panels Wallerich Group, LLC

B7:Perforated Tile location DiagramOptionPros and Cons Cold Aisle Merit: Required for efficient delivery of cold air to equipment. Under rack Demerit: Only required by specific rack manufacturer. Other Demerit: May be required for room air supply if cold aisle enclosure selected 17 Rack Wallerich Group, LLC

B8:Return Air Design DiagramOptionPros and Cons Open Merit: Unclosing return air flow is used when floor height is low to obtain plenum area. Demerit: Hot and cold air are allowed to mix lowering the thermal load that room can support and increases costs of operation. Plenum Merit: Maximize CRAC cooling capability and increase thermal load that room can support. 18 CRAC rack CRAC rack Wallerich Group, LLC

B9:Plenum Design DiagramOptionPros and Cons Cold Aisle Merit: Improves cooling efficiency and load that can be supported. Demerit: No as efficient as hot aisle enclosure. Hot Aisle Merit: Recommended by Intel. Internal tests show as being most efficient and effective airflow design. No enclosure Demerit: Not Recommended Hot and Cold Aisle (20KW Only) Demerit: Not recommended: blocks access to network infrastructure, adds costs with no real benefit. 19 Rack Enclosure ラック Rack Enclosure Rack ラック Rack ラック Rack Enclosure Rack Wallerich Group, LLC

B10:Rack Doors (Front & Back) DiagramOptionPros and Cons No Door Merit: Maximize cooling efficiency. Demerit: no server access control. May be OK for some racks. <50% Demerit: There is no necessity to recommend % open Demerit: There is no necessity to recommend. 80% open Merit: Accelerate cooling efficiency with server access control Prevent human error 20 Rack Wallerich Group, LLC

B11:CRAC Position DiagramOptionPros and Cons Opposing Merit: Maximize CRAC cooling efficiency for typical rectangular room Multi-Wall Demerit: Can be used, but only if total capacity requires it. Mid-Floor Demerit: Not recommended as it introduces water to the RMF area, exposing equipment to possible water contamination 21 CRAC Wallerich Group, LLC

B12:CRAC Location DiagramOptionPros and Cons Inside RMF Demerit: Not recommended due to noise and security requirements. Outside RMF Merit: Reduce maintenance time and facilitate maintenance work with server access control Prevent human error Overhead Demerit: Not recommended due to building height limitations. 22 CRAC Rack CRAC Rack Wallerich Group, LLC

B13:CRAC Design DiagramOptionPros and Cons Off the Shelf Merit: to be determined by CFD tool and available CRAC designs in Japan. Custom Merit: May be required due to high air volume. Likely to be more expensive Wallerich Group, LLC

B14:Tile Size DiagramOptionPros and Cons 600mm sq Merit:: Widely distributed in Japan and Easy to purchase 700mm sq Demerit: Odd Size, no value. 800mm sq Demerit: Odd size, no value mm 700mm 800mm Wallerich Group, LLC

B15:Network Cable Location DiagramOptionPros and Cons Overhead Merit: Reduce maintenance time and facilitate maintenance work. Maximize CRAC cooling efficiency by avoiding under- floor air blockage Under Floor Demerit: Twisted-pair cable (network cable) become obstacle for efficient airflow 25 Rack Wallerich Group, LLC

B16:Power cable location DiagramOptionPros and Cons Overhead Merit: Separated distance from network cable must be mm Under Floor Merit: Reduce maintenance time and facilitate maintenance work to keep distance from network cable as mm *3 Thin bus power cable affect a little to airflow 26 Rack Wallerich Group, LLC

B17:Rack Width DiagramOptionPros and Cons 600mm / 24” Merit: Utilizing cable mgmt system, revenue target requires maximum rack count. Increase number of racks per room. 700mm / 30” Demerit: No clear advantage, and reduces overall rack count. Can be used with 600mm if needed. Reduce number of racks per room than 600mm 800mm / 36” Demerit: No clear advantage, and reduces overall rack count. Can be used with 600mm if needed. Reduce number of racks per room than 600mm mm 700mm Rack 800mm Rack Wallerich Group, LLC

B18:Rack Depth DiagramOptionPros and Cons 1m Merit: Should be less than twice tile size for maintenance access 1.2m Merit: Should be less than twice of tile size for maintenance efficiency > 1.2m Demerit: Not recommended – rack will overlap tile edge making access impossible. 28 Rack 1m Rack 1.2m Rack 2m Wallerich Group, LLC

B19:Rack Height DiagramOptionPros and Cons 1.8m Merit: Lower rack can reduce number of servers per rack 2m Merit: Typical rack size, creates no op issues. Will support up to 42 1U servers 2.2m Merit: Higher rack can increase number of servers per rack Demerit: Will limit space available for overhead network cabling 29 Rack 1.8m Rack 2m Rack 2.2m Wallerich Group, LLC

B20:Rack-Ceiling Distance DiagramOptionPros and Cons <460mm Demerit: A minimum clearance between ceiling to rack shall be 460mm >460mm Merit: A minimum clearance between ceiling to rack shall be 460mm *4 30 Rack <460mm >460mm Wallerich Group, LLC

Design Elements Diagram (1/4) 31 CRAC Air Flow=ROW CRAC Grouping=No Grouping Supplemental Cooling Design Primary Cooling Architecture=Chilled Water Cold Air Supply= Under Floor Raised Material Floor (RMT) Dropped Ceiling Raised Floor Height=900mm (36inch) Rack Cooling Architecture=Primary Cooling Wallerich Group, LLC

Design Elements Diagram (2/4) 32 CRAC Rack Perforated Tile Perforated Tile type=Fixed Perforated Tile opening= >50%open Rack Configuration= Hot Aisle & Cold Aisle Rack Air Control= Blank Panels & Side Panels Return Air Design = Plenum Plenum Design= Hot Aisle Tile Size= 600mm Perforated Tile location= Cold Aisle Wallerich Group, LLC

Design Elements Diagram (3/4) 33 Rack CRAC CRAC Position= Opposing CRAC Location= Outside RMF Dry Wall CRAC Design= Off the Shelf/Custom Wallerich Group, LLC

Design Elements Diagram (4/4) 34 Rack Building Desgin Building Design Raised Floor Height = 900mm(36inch) Rack Height = 2m Rack-Ceiling Distance= >460mm Rack Rack Width = 600mm Rack Depth = 1.2m Network Cable Location=Overhead Power Cable Location=Under Floor Wallerich Group, LLC

Wallerich Group, LLC