Energy Savings in CERN’s Main Data Centre Vincent Doré & Wayne Salter IT Computing Facilities Group Mauro Nonis & Sabri Masrie EN Cooling and Ventilation Group

Overview of CERN’s Data Centre Cooling system for main room Summary Overview of CERN’s Data Centre Cooling system for main room Situation before 2008 Improvements since 2008 Foreseen further improvements Energy savings Conclusions Energy Savings in CERN’s Main Data Centre - 2

CERN’s Main Data Centre Data centre built in early 1970s Has been refurbished/upgraded several times Has three main computer rooms: 1440m2 main room with air cooling 1240m2 basement room mixture of air/water cooling 225m2 room for critical servers only with water cooling Data B/U in a separate building 3.5 MW available for computing equipment Energy Savings in CERN’s Main Data Centre - 3

Main Room Cooling System 3+1 chillers (each 1.45 MW) 9+1 AHUs just under roof Cold air passes down through large ducts and under the false floor Enters cold aisle through perforated tiles Server fans draw the air though the servers Exiting hot air sucked out via the false ceiling Energy Savings in CERN’s Main Data Centre - 4

Situation before 2008 From 70s to 90s mainframes Early 00s tower PCs In 2007 started with rack mounted PCs No hot/cold aisle separation Inefficient cooling High energy consumption Not possible to properly cool in the summer Energy Savings in CERN’s Main Data Centre - 5

Installation of cold corridors (2008) Enclosed cold aisles Doors from rack supplier Roof made at CERN Strict hot/cold air separation All servers comfortably cooled even during very hot weather Unfortunately energy savings from this are not known Energy Savings in CERN’s Main Data Centre - 6

Optimisation of air distribution (2010/2011) Analysing and modifying the operation of AHUs to increase the use of free cooling by improving measurement and implementing full automation of air intake and mixing reducing the intake of hot exhaust air modifying the anti-freeze protection Maximum server inlet temperature increased from 25 ºC to 27 ºC (ASHRAE) Set points increased temperature of air produced by AHUs increased in steps from 14 ºC to 22 ºC temperature of chilled water circuit increased from 5 ºC to 8 ºC Thermal analysis Energy Savings in CERN’s Main Data Centre - 7

Other measures since 2011 Improved monitoring Identified and fixed leaks between the hot and cold plenums Now able to run with 4 from 10 AHUs switched off Investigate the possibility to change the fixed speed motors of the AHUs to variable speed motors Allow better control of the flow rate to meet actual needs Introduction of water cooled racks; first active and now passive Increasing power density per rack Energy Savings in CERN’s Main Data Centre - 8

What are the energy savings? Not known exactly! Missing important monitoring data before 2010 Chillers are used not only for main room but also for other rooms Nevertheless we estimate for the changes made in 2010/2011 for the main room With DC load of 2.6 MW and 1.9 MW in main room Annual power consumption of chillers decreased from 6.5 GWh to 1.4 GWh PUE for the entire DC reduced from 1.7 to 1.5 Switching off 3 AHUs (40 kW each) saves an additional 1.1 GWh Chiller Energy Consumption For Main Computer Room July Nov Feb Jan Energy Savings in CERN’s Main Data Centre - 9

Thanks to the CERN EN/CV Group. Conclusion Data centre built during mainframe era Has needed to be adapted to changing computing technologies Major improvements have been made: Installation of hot/cold aisles More complete use of free cooling More aggressive temperature environment Impressive energy savings achieved through fairly simple measures Further measurements are continually being sought Thanks to the CERN EN/CV Group. Energy Savings in CERN’s Main Data Centre - 10