BayCare Health System Designs Next Generation Data Center August 8, 2007 Doug Lauterbach Data Center Director, Bay Care Health System Matt Schuster Principal Consultant, BT INS

In todays presentation… About BayCare Challenges that necessitated a new data center Planning the new facility A look at the new data center Your questions

About BayCare 9 hospitals in Tampa Bay area Largest full service community-based health system in the region 17,000 team members 2,707 Beds 355,523 ER visits 6.3 million lab tests

Challenges that necessitated a new facility

Business needs driving IT In the process of a 7 year electronic medical record project (moving towards paperless) Constant growth of digital imaging Increasing compliance regulations - JCAHO concerns about information management and availability Increased clinical reliance on IT systems requires high availability for quality patient care –Pharmacy –Lab –Radiology –Food and Nutrition –Bed tracking –Scheduling

Business Drivers of the new Facility Best Practices Business Continuity Business Expansion Capacity Planning or Performance Analysis Change in Application Architecture, Providers or Vendors Compliance Disaster Recovery Expense Management Increased Capacity Operational Optimization Regulatory Reliability Risk Management and Mitigation Technology Growth or Consolidation Technology Refresh

Existing facility challenge: space Short on space –8 separate spaces for IT equipment –4,700 sq. ft data center split into 4 spaces 30% of data center space used for circulation Large command center using a large portion of raised floor space Buildings physical resiliency not up to hurricane standards Design aspect to 40 watts/sf Shallow cabinets limit the ability to expand and distribute power and low-voltage cabling Structure cabling unable to support: –Blades –VoIP –Anticipated SAN growth

Existing facility challenge: power load Design didnt support heat and power load Downtime related to power design and capacity –circuit capacity lacking –Distribution: not dual bus all the way to equipment Lack of available power threatening to limit new technology adoption No standard power configurations No means or method to manage/measure power at the equipment racks

Existing facility challenge: cooling Cooling was inefficient –Not using hot-aisle/cold-aisle design –Uneven temperatures through space –Hot spots creeping up Equipment lifespan likely shortened due to heat 13-year old design, not optimized for blade configurations Not easily expandable Low 12 raised floor and under-floor low-voltage cabling restricting air supply No available physical space for additional cooling Legacy CRACs were not communicating to efficiently cool the space.

Existing facility challenge: visibility & control Monitoring – currently focuses on circuits that are tripped (reactive), not designed for proactive data Temperature monitoring handled at the return air of the precision air conditioner (not looking at hotspots) Little remote control for servers Lights-out management (iLO) Existing Liebert SiteScan equipment was not managing non-Liebert equipment No temperature sensors at equipment location or around the room No branch circuit monitoring 100% of lighting on 24x7

Existing facility challenge: managing change Provisioning is not standardized, automated or managed System deployments are based on: –looking through racks –personal knowledge –gut feeling –where uneducated users want placement Not managing capacity of: –space –circuits –power –weight –heat loads –network switch ports

Planning the New Facility

Objectives Provision for planned growth – addition of 100 servers per year Gain complete visibility and control and improve administration Improve system availability through full redundancy of the power and cooling systems Improve fire detection and suppression Implement hot-aisle/cold-aisle design for maximum capacity and energy efficiency Improve planning and provisioning Improve cable management Increased Mechanical and Electrical efficiencies Reduced operational costs Operate as a nearly lights-out facility Remote access into all systems

Building a Team Doug Lauterbach Imaging Services Director (serving as Data Center Project Director), BayCare Health System Chris Jenkins, Director Technology, BayCare Health System Matt Schuster, Principal Consultant, BT INS Thomas Gooch, Local Liebert Representative, Emerson Network Power Paul Carastro, Carastro & Assoc, MEP Engineers Mike Montecalvo, General Contractor, Solutions, Inc. Paul Schnitzlein, Architect, Harvard Jolly

Documenting Existing Equipment Document existing devices, power supplies and port counts per device at the rack level Determine/Graph device averages per rack to determine cabinet profiles Each piece of equipment assigned a number and quantities and loads were considered. Classifications included: –Legacy –Low –Medium –High –Blade –IBM RISC –Network

Planning Process: Modeling Mapped watts per square foot via CFD several times, tweaking each time to determine placement of precision air conditioners Room designed for 800-1,000 kW; CFD proves capacity 24-inch raised floor, multiple precision air conditioning units White space to grow into: 40 to 50 percent allows for doubling of capacity Rerun CFD every three to six months as changes are required using actual values collected from thermistors, branch circuit monitoring and other tools

Existing facility: Servers per Cabinet

Planning Process: Profile Planning Cabinet Profiles

Best Practice: Technology Infrastructure 44U, 45 deep cabinets to allow for distribution of power and low-voltage cabling Metered Power Strips –Every cabinet has at least two strips, one for each bus –Some cabinets with additional strip behind a static transfer switch –Each strip has colored tape to quickly determine which bus KVM over IP in each cabinet –Centralized –Decentralized iLO switch in each cabinet ($38/Ethernet port)

Best Practice: Power System Design

Best Practice: Dual-Bus EPO Systems

Best Practice: Fire Suppression Very Early Smoke Detection Apparatus (VESDA) Clean Agent Suppression system –People-friendly –Environmentally friendly –Lower quantity of agent vs. FM-200 –Over 3,000 pounds of FE-25 agent Four zones Over 150+ detection points (above and below floor) Sprinklers were required, therefore pre-action with double-interlock was installed

Best Practice: Remote Management Integrated Lights Out – allows remote control of servers from anywhere in the world People-free facility benefits: –Removed NOC Command Center from raised floor –Minimizes human error disruptions –Allows managers to monitor and control environment remotely

Best Practice: Proactive Monitoring Followed some of ASHRAEs TC9.9 recommendations for monitoring temperature in racks/cabinets Temperature and Relative Humidity Branch Circuit Monitoring SiteScan I/O-32 and additional units Leak Detection VESDA Fire Suppression Fire Detection Liebert Environmental Control Systems ICOMs Control Network

Best Practice: Labeling Mechanical equipment Electrical equipment Electrical receptacles Grid Coordinate Racks and Cabinets Structured Cabling

Tools: Rackwise/DCM Used tool to document existing inventories Tracking relocation from one facility to another Manage asset location by rack unit Provisioning limitations per cabinet Reporting capabilities –Space and Assets –Power and Cooling –Cost and Capacity Planning –Rack Assembly with Cabling –Service & Warranty Contracts –Department & Customer Information Long term objective: to integrate a change/configuration management tool to track and manage assets

Tools: Rackwise/DCM

Tools: Computational Fluid Dynamics Allowed modeling the complete room for the initial 6 CRACs and full deployment of 12

Tools: Computational Fluid Dynamics Initial assessment indicates potential problems related to placement of precision cooling units… Low Pressure Zone CRACs

Tools: Computational Fluid Dynamics CFD Program allows for virtual change - validating proposed solution prior to installation. CRAC Moved To New Location Low Pressure Zone Removed

Tools: Computational Fluid Dynamics Follow-on Modeling Provides Proof of Operational Success Assuming Full Load – 735 kW. Clouds Per Fahrenheit Degree

Tools: Computational Fluid Dynamics Modeling allows for proof of design concept at full load – prior to implementation Pathlines from CRACs Temperature ( o F) in 1 foot increments

CRAC Network Communications

New Data Center – Construction

Current Data Center – Raised Floor

New Data Center – Raised Floor

Current Data Center – Operations

New Data Center – Operations

Current Data Center – Generator

Current Data Center – Network Rack

Current Data Center – Server Cabinets

Current Data Center – UPS Room

New Data Center – UPS Room

Current Data Center – Yard

New Data Center – Yard Construction

New Data Center – Yard Outside

Critical Success Factors Teamwork between BayCare, INS and Emerson Network Power/Liebert enabled success Maximization of modeling tools allowed a solid design, even before the first nail was hammered Have done virtualization study – partially executing recommendations ahead of time to make move easier Cant convert everything to dual corded, so a Liebert STS was installed to provide redundancy Space for a second generator is provisioned and an ATS is in place to accommodate expansion Move in progress!

Questions and Answers