Data Center Construction Project Updates and Experiences Gary Stiehr May 6, 2008 HEPiX Spring 2008

Overview Existing data center filled and power and cooling at their limits Started construction approximately 10 months ago on new data center Originally expected four phases to get to 1,800 kW of redundant power/cooling Expanded Phase 1 to 750 kW from 450 kW Anticipate just two more phases

Clarification Clarification from data center talk at HEPiX Fall 2007 in St. Louis: 16,000 sq. ft. figure mentioned includes all real estate space inside and outside and partial second floor 3,200 sq. ft. is usable data center space, about 96 sq. ft. used by Remote Power Panels. Many sites not calculating usage of external real estate space; only percentage of raised floor space used by infrastructure

Current Status Phase 1 is mostly complete: power and cooling for 750 kW Installing seismic bases, racks and structured cabling this week and next week Delay from manufacturer for 5kV switch gear –No generator backup until mid June –Temporary solution in place; no downtime to fix

Equipment Racks Standardizing racks, especially height –Height: 47U 40U for equipment 7U for patch panels and other (e.g., switches, monitoring devices, grounding bus bar, etc.) –Width: 24” (610 mm) for servers, 32” (813 mm) for network –Depth: 48” to accommodate more power without taking away from cable management space Standardization: versatile, consistent deployment, easier to enclose cold aisles

Equipment Racks: Considerations Check that doorways are tall enough throughout delivery path to accommodate taller racks Fire codes/regulations require a minimum clearance from ceiling to top of racks for fire suppression system Isolating seismic vibration Weight limits: increasing density and weight Baying together helps with tipping hazards?

Power Delivery 3-phase, 30 A, 208 V to the rack: Need to balance between phases in rack? Use 60 A circuits for racks with higher loads? –Fewer power whips to rack –Fewer PDUs in the rack –More expensive to construct power whips? Does anything need 120 V any more? –L21-30R vs. L15-30R receptacles

Floor Layout Remote Power Panel (RPP) cabinets –Larger than originally specified –Placed such that row starts with a half tile –One at each end of row so racks confined between them Considerations for seismic isolation –St. Louis is in a 2A seismic zone –Bracing to concrete floor thought to pass vibration to equipment, which may cause damage –Using ISO-Base™ Seismic Isolation Platform Requires 8.5” (215.9 mm) clearance on all sides Plus additional 2” (50.8 mm) at ends if leveling feet are used on racks

Floor Layout (2) Racks may set on multiple tiles due to wider racks, RPP clearances and seismic base clearances: –Raised floor cutouts may cross two tiles –Implications on how to adjust cooling for a particular rack –Rack labeling scheme needs to take this into account if it is based off of raised floor tile grid system

Structured Cabling Patch panels in each rack (EDA), running to core patch cabinets (MDA) Category 6a (a=augmented) standards recently ratified –10 Gb/s over copper over longer distances More 10 Gb/s copper-based devices? –Comparing costs with fiber and SFPs –For cost, use cat5e/6 patch cables for management networks?

Safety 3-foot raise floor: block off work areas as usual ServerLift for loading equipment racks ECARO-25 clean agent fire suppression Alarms, controls: –White light and alarm for external fire/smoke –Amber light and alarm for ECARO-25 stage 1 fire/smoke inside server room –Bell ringing for stage 2; 30-seconds to release EPO (Emergency Power Off) button ECARO-25 temporary abort button ECARO-25 forced activation button

Building Services Do you know who will provide these: Maintenance on electrical/HVAC systems Custodial/Housekeeping services inside server room, office, mechanical rooms, etc. Staffing receiving dock (and when). Access control (e.g., swipe card system) Telecommunications (e.g., phone lines, fax) Cardboard recycling, disposal of other packaging?

Future Work Establishing more formal monitoring of temperature, air flow and electrical usage Phase 2 planning is beginning –Increase redundant power/cooling capacity to 1350 kW –Potentially enclose cold aisles but may wait for next and final phase to gather enough data

Lessons/Advice If you are new to data center planning: Start thinking about the considerations/options early Consider dedicating most of an FTE to this planning and narrowing options Continually verify measurements and placements within server room.

Questions?