1. Green Data Center Program
03/16/08
Green Data Center Program
Alan Crosswell
03/18/09

2. Agenda The CUIT Data Center Other Data Centers around Columbia
03/16/08
Agenda
The CUIT Data Center
Other Data Centers around Columbia
Current and Developing Data Center Best Practices
Future State Goals
Our NYSERDA Advanced Concepts Datacenter proposal
Next Steps
03/18/09

3. CUIT Data Center Architectural
03/16/08
CUIT Data Center
Architectural
Built in 1963, updated somewhat in the 1980's.
4100 sf raised floor machine room space.
1750 sf raised floor space, now offices for GSB IT.
12” raised floor
Adequate support spaces nearby
Staff
Staging
Storage
Mechanical & fire suppression
(future) UPS battery room
03/18/09

4. CUIT Data Center Electrical
03/16/08
CUIT Data Center
Electrical
Supply: 3-phase 208V from automatic transfer switch.
Distribution: 208V to wall-mounted panels; 120V to most servers.
No central UPS; lots of rack-mounted units.
Generator: 1750 kW shared with other users & at capacity.
No metering. (Spot readings every decade or so:-)‏
IT demand load tripled from
03/18/09

5. CUIT Data Center
03/18/09

6. CUIT Data Center Mechanical
03/16/08
CUIT Data Center
Mechanical
On floor CRAC units served by campus chilled water.
Also served by backup glycol dry coolers.
Supplements a central overhead chilled air system.
Heat load is shared between the overhead and CRAC.
No hot/cold aisles
Rows are in various orientations
Due to tripling of demand load, the backup (generator-powered) CRAC units lack sufficient capacity.
03/18/09

7. CUIT Data Center IT systems
03/16/08
CUIT Data Center
IT systems
A mix of mostly administrative (non- research) systems.
Most servers dual-corded 120V power input.
Many ancient servers.
Due to lack of room UPS, each rack has UPSes taking up 30-40% of the space.
Lots of spaghetti in the racks and under the floor.
03/18/09

8. Power Use Efficiency (PUE) =3.78
CUIT Data Center
Power Use Efficiency (PUE) =3.78
This is a SWAG!
03/18/09

9. LBNL Average PUE for 12 Data Centers
Power Use Efficiency (PUE) =2.17
03/18/09

10. Other Data Centers around Columbia
03/16/08
Other Data Centers around Columbia
Many school, departmental & research server rooms all over the place.
Range from huge (7,000 sf C2B2)
… to tiny (2-3 servers in a closet)
Several mid-sized (Physics, Biology, CS, APAM, CUIT backup DC, OAD, CUF, CUMCIT, Lamont, Nevis, etc.)
Most lack electrical or HVAC backup.
Many could be better used as academic space (labs, offices, classrooms).
Growth in research HPC putting increasing pressure on these server rooms.
Lots of wasted money building new server rooms for HPC clusters that are part of faculty startup packages, etc.
03/18/09

11. Making the server slice bigger, the pie smaller and green.
Reduce the PUE ratio by improving electrical & mechanical efficiency.
Google claims a PUE of 1.2
Consolidate data centers (server rooms)
Claimed more efficient when larger (prove it!)
Free up valuable space for wet labs, offices, classrooms.
Reduce the overall IT load through
Server efficiency (newer, more efficient hardware)
Server consolidation & sharing
Virtualization
Shared research clusters
Move servers to a zero-carbon data center
03/18/09

12. Data center electrical best practices
95% efficient 480V room UPS
Basement UPS battery room
vs. wasting 40% of rack space
480V distribution to PDUs at ends of rack rows
Transformed to 208/120V at PDU
Reduces copper needed, transmission losses
208V power to servers vs. 120V
More efficient (how much?)
Variable Frequency Drives for cooling fans and pumps
Motor power consumption increases as the cube of the speed.
Generator backup
03/18/09

13. Data center mechanical best practices
Air flow – reduce mixing, increase delta-T
Hot/cold or double hot aisle separation
24-36” under floor plenum
Plug up leaks in floor
Increase temperature
Perform CFD modeling
Alternative cooling technique: In-row or in-rack cooling
Reduces or eliminates hot/cold air mixing
More efficient transfer of heat (how much?)
Supports much higher power density
Water-cooled servers are making a comeback
03/18/09

14. Data center green power best practices
Locate data center near a renewable source
Hydroelectric power in Canada
Wind power – but most wind farms lack transmission capacity.
40% of power is lost in transmission. So bring the servers to the power.
Leverages our international high speed data networks
Use “free cooling” (outside air)
Stanford proposal will free cool almost always
Implement “follow the Sun” data centers
Move the compute load to wherever the greenest power is currently available.
03/18/09

15. Energy Saving Best Practices
Efficient lighting, HVAC, windows, appliances, etc.
LBNL and other nations’ 1W standby power proposals
Behavior modification
Turn off the lights!
Enable power-saving options on computers
Social experiment in Watt Hall
Co-generation
Waste heat is recycled to generate energy
Planned for Manhattanville campus
Possibly for Morningside campus
Columbia participation in PlaNYC
03/18/09

16. Barriers to implementing best practices
Capital costs
Short-term and parochial thinking
Saving electricity is not well incented as nobody is billed for their electrical use.
Distance
Synchronous writes for data replication are limited to about 30 miles
Bandwidth*delay product impact on transmission of large amounts of data
Reliability concerns
Server hugging
Staffing needs
03/18/09

17. Key Recommendations from Bruns-Pak 2008 Study
Allocate currently unused spaces for storage, UPS, etc.
Consolidate racks to recapture floor space
Generally improve redundancy of electrical & HVAC
Upgrade electrical systems
750 kVA UPS module
New 480V 1500 kVA service
Generator improvements
Upgrade HVAC systems
200-ton cooling plant
VFD pumps & fans
Advanced control system
03/18/09

18. Future State Goals – Next 5 years
03/16/08
Future State Goals – Next 5 years
Begin phased upgrades of the Data Center to Improve power and space efficiency. Overall cost ~ $20M.
Consolidate and replace pizza box servers with blades (& virtualization).
Consolidate and simplify storage systems.
Accommodate growing demand for HPC research clusters
Share clusters among researchers to be more efficient
Accommodate server needs of new Interdisciplinary Science Building.
Develop internal cloud services.
Explore external cloud services.
Stanford giving Amazon EC2 credits for faculty startup
03/18/09

19. Future State Goals – Next 5-10 years
03/16/08
Future State Goals – Next 5-10 years
Build a new data center of 10,000-15,000 sf
Perhaps cooperatively with others
Not necessarily in NYC
Possibly in Manhattanville
Consolidate many small server rooms.
Significant use of green-energy cloud computing resources.
From
03/18/09

20. 03/16/08
Our NYSERDA Grant
New York State Energy Research & Development Authority Program Opportunity Notice 1206
~$1.2M ($447K from NYSERDA awarded pending contract)‏
Improve space & power efficiency of primarily administrative servers.
Contribute to Columbia's PlaNYC carbon footprint reduction goal.
Make room for shared research computing in the existing data center.
Measure and test vendor claims of energy efficiency improvements.
03/18/09

21. Our NYSERDA Grant– Specific Tasks
03/16/08
Our NYSERDA Grant– Specific Tasks
Identify 30 old servers to replace.
Instrument server power consumption and data center heat load. “Real time” with SNMP.
Establish PUE profile (use DoE DC Pro survey tool)‏
Implement 9 racks of high-density cooling (in-row/rack).
Implement proper UPS and high-voltage distribution.
Compare old & new research clusters' power consumption for the same workload.
Implement advanced server power management and measure improvements.
Review with internal, external and research faculty advisory groups.
Communicate results.
03/18/09

22. Measuring Power Consumption
Use SNMP which enables comparison with other metrics
03/18/09

23. Measuring Power Consumption
Can measure power use with SNMP at:
Main electrical feeder, panels, subpanels, circuits.
UPS
Power strips
Some servers
HP c7000 blade chassis SNMP MIB includes
Power supply load
Cooling fan airflow
etc.
03/18/09

24. Measuring Heat Rejection
Data Center chilled water goes through a plate heat exchanger to the campus chilled water loop.
Measure the amount of heat rejected to the campus loop with SNMP instrumented:
BTU meter &
Flow meter
03/18/09

25. Next Steps Beyond the NYSERDA Grant
Develop a “shovel ready” plan to continue moving forward.
Includes Server & Storage replacement strategy
Identify funding needs and opportunities
ISB Servers
Manhattanville Phase 1 Servers
Shared research cluster expansion
Possible grants:
NIH Extramural Research Facilities and Improvement Program ($1-10M)
DoE Information and Communication Facility Energy Efficiency (to be announced this month)
Additional forthcoming stimulus (ARRA) grants
03/18/09

26. Thanks…
CU Facilities: Steve Poller, Frank Mastromauro, Dave Carlson, Dominick Chirico, Clem Olivio, Frank Martin, Wil Elmes, Matt Early, Dave Forbes, Geoff Wiener, Joe Ienuso, Joe Mannino, Fran Fitzgerald, Frances Huppert, David Greenberg
Office of the EVP for Research: Greg Culler, Victoria Hamilton, David Hirsh, Marie Tracy, Patty Valencia-Ferguson, Mario Reyes
Bruns-Pak, Dell, HP, IBM
NYSERNet: Bob Bloom, Tim Lance
NYSERDA: Joe Borowiec
CUIT/SAS: Halayn Hescock, Rich Hall, Victor Warren, Ryan Abrecea, Tony Cirillo, all of Systems & Network Engineering, Candy Fleming, Donna Sadlon, Jeff Scott, John Milnes
External visiting committee: Marilyn McMillan (NYU), Vace Kundakci (CCNY), Lauri Kerr (NYC), Tim Lance (NYSERNet),
Internal advisory group: Wil Elmes, Art Langer, Nilda Mesa, Scott Norum, Len Peters
Research faculty user group: Liam Paninski, Lei Cong, Greg Bryan, Kathryn Johnston, Mary Putman
This work is supported in part by the New York State Energy Research and Development Authority.
03/18/09