1. One Small Step Toward a Green Datacenter Quan Nguyen Network and Communications Services McGill University CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

2. Todays Talk Power consumption trends in the datacenter Our cooling problem Our solution The result Lessons learned: –Take time to understand air flow and heat extraction –Find one design and stick to it –Ad-hoc fixes can adversely affect the solution CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

3. Power consumption trends in the datacenter Jonathan Koomey[1] @ LNBL and Stanford: –Power consumption by servers in the US grows by 14% each year –In 2005, datacenters in the US burned about 45 billion kWh, 1.2% of 2005 retail electricity sales or about $2.7 billion –Worldwide electricity use for volume servers (OEM) estimated at 1.7% –Every kWh of electricity used for processing IT loads translates into another kWh of electricity for the supporting physical infrastructure (UPSes, PDUs, HVAC equipment) CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

4. [Nick Carr] [2] An avataran online representation of a humanon Linden Labs virtual reality online game Second Life consumes: 1,752 kWh per year (about $100/year of electricity) as much power as the average Brazilian over the course of one year CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

5. More trends on power in datacenters 2007 Gartner report[3]: datacenters are responsible for 0.5% of global CO 2 emissions. The whole IT and Telecom industry accounts for 2%, equivalent to that of the aviation industry A medium-sized server has roughly same carbon footprint as an SUV doing 15 mpg [4] AFCOM (Association for Computer Operation Managers) and InterUnity Group 2005 survey: –48% felt that they would be incapable of supporting business requirements due to capacity constraints Demand for new application growth will continue to outstrip the processing power increases that flow from Moores Law CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

6. Our Cooling Problem Many hot spots (not Wi-Fi hot spots!). Servers near the top are not cooled sufficiently Average room temperature rose from 72F to 77F when one specific cabinet was put in service Frequent CRAC (computer room air- conditioner) failures Our cold aisles are hot and our hot aisles are hot too CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

7. CAR Hot aisle CRAC UPS Cold aisle Hot aisle Cold aisle Cool air Hot air Airflow direction CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

8. CRAC UPS Cool air Hot air Raised Floor False Ceiling Concrete Floor Side view of the datacenter CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

9. We soon realized… Recommendations [6] from a reputable engineering firm that designs our building HVAC systems… brute force approach - add another CRAC unit But –It will cost a lot (and a lot) to add more cooling capacity –We still dont know how to solve the hot spot problem –We still dont know how to properly set up hot aisle/cold aisle –Not more efficient than current system –CRAC still fight with CRACs for humidity control CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

10. We asked for a second opinion We talked to a firm[7] specializing in datacenter build-out and soon learned: –We can make our CRAC units working more efficiently instead of the brute force approach of adding more cooling power (costs much less) –There are some things that technology just cant solve. We need to update our rack and server placement and ad-hoc installation methodology –We need a technology to support the new higher density racks. CRACs just cant cool these CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

12. A detailed analysis of our problem Our CRAC units are supposed to be able to adequately cool our existing heat load. They simply didnt work efficiently They were designed for low power density racks and cant keep up with higher density servers They were designed to cool the room and not necessarily the hot spots CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

13. … detailed analysis of our problem Heat dissipated by the servers could not be absorbed completely back to the CRACs Non-captured heat got stuck just below the ceiling creating a heat fog phenomena Non-captured heat got mixed back with the cold air making the situation worst CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

14. … detailed analysis of our problem Air is not cold enough when it gets to the top of the rack Our CRAC units are working too hard: –8% more current drawn than normal –Increased rate of failures –More refrigerant/compressor seizures CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

15. … detailed analysis of our problem Some equipment is placed in the wrong direction blowing heat into the wrong place Too much heat turbulence Air is still very cold under the floor –Indication that we still have enough cooling capacity Too many or not enough perforated tiles? (More on this later) CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

16. So what is the ideal situation? CRAC units should work as the specs say within their optimal curves. Intake temperature should be as high as possible New concept is to remove heat and not pump cold Cold aisles stay cold, hot aisles stay hot Servers at top of cabinets receive adequate cooling Which means: –No heat fog –CRAC units must draw less current –No heat turbulence CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

17. Too many or not enough perforated tiles? The big engineering firm said that we didnt have enough Based on the theoretical limit of our CRAC units, they are supplying more CFM (cubic feet per minute) than the existing perforated tiles allow So we must add more perforated tiles It didnt help! … more in next slides CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

18. Why adding more perforated tiles didnt help? There is not enough static pressure under the raised floor. –We were able to fix the problem and raise the static pressure by 38% after the work described later The placement of the perforated tiles and the type (50%, 25% perforation) is important. Not all racks have the same heat load. Add those tiles only in higher- density rack area CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

19. Why adding more perforated tiles didnt help? We are already poking too many holes in the floor mostly under the racks for cable runs. –It was equivalent to have 25 floor tiles permanently removed –The cold air was released into the hot aisles creating more heat turbulence and starving the racks of cool air Cable runs under the raised floor block the air circulation in some areas CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

20. The fix A series of coordinated efforts recommended by the specialized firm We did spend a bit extra to measure the static pressure and efficiency of our CRAC units before and after the work All the gains were obtained as expected CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

21. 1 – Clean out sub floor Remove all unused cables (from the mainframe days) Tidy up cable runs (or rerun some) Make sure cable paths run in parallel with the airflow from CRAC units CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

22. 2 – Seal the floor Seal almost 100 Sq-feet (260 floor openings) KodLok® brush system CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

23. 2 – Seal the floor Plenaform adhesive sheets CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

24. 3 – Use the false ceiling for hot air return 3A – Install ductwork onto the return air intakes of the CRAC units so that 100% of the return air would be drawn from the false ceiling plenum 3B – Install egg crate type return air grills in the false ceiling and position them strategically (mostly in hot aisles and hot spots) CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

25. 3 – Use the false ceiling for hot air return CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

26. CRAC UPS Cool air Hot air Raised Floor False Ceiling Concrete Floor New ductwork 3 – Use the false ceiling for hot air return CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

27. 4 – Prevent heat leakage between rows Do not leave openings in the racks. Install blanking panels Do not leave openings in between cabinets Do not have cabinets of varying heights in the same row CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

28. From APC White Paper #42 CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

29. 5 – Reposition cabinets Avoid blowing hot air into cold aisle or the intake of other equipment (surprising that we have made that mistake) Make sure home-made fans are not mounted in reverse or blowing hot air into cold aisle Avoid fans in any case. They stir up dust CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

30. Measured results Static pressure increased by 38% Average room temperature unchanged but the temperature of the hot air intake into the CRAC units increases by 2.6% Total CFM of cold air coming from the floor increased by 23% (34,400 CFM 42,500 CFM) Current drawn by the CRAC units dropped by 8% CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

31. Estimated Results The return air temperature into the CRAC units increased, resulting in higher cooling capacity and efficiency. We have regained our CRAC efficiency. We estimated that we have regained about 20 tons of cooling capacity (for about 70kW of heat load) Total work cost $50K versus $150K+ for adding equivalent cooling capacity CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

32. In Summary 1.Clean out sub floor 2.Seal the floor 3.Use the false ceiling for hot air return (install ducting from the CRAC into the ceiling) 4.Prevent heat leakage between rows 5.Reposition cabinets CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

33. How to stay green Save energy (and money too) by running efficiently –Carefully implement the hot-aisle, cold-aisle scheme –Run the CRAC at its optimal performance curve and set the temperature and humidity at the right mid-point (74 F and 50% humidity) –Run air conditioners and UPS units at close capacity to be energy-efficient (at least more than 65%). Do not overbuy. Use modular systems CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary

34. End Notes [1] Estimating total power consumption by servers in the U.S. and the world http://enterprise.amd.com/Downloads/svrpwrusecompletefinal.pdf http://enterprise.amd.com/us-en/AMD-Business/Technology-Home/Power- Management.aspx http://enterprise.amd.com/Downloads/svrpwrusecompletefinal.pdf http://enterprise.amd.com/us-en/AMD-Business/Technology-Home/Power- Management.aspx [2] Avatars consume as much electricity as Brazilians http://www.roughtype.com/archives/2006/12/avatars_consume.php http://www.roughtype.com/archives/2006/12/avatars_consume.php [3] KUMAR, Rakesh, et Lars MIERITZ. Conceptualizing 'Green' IT and Data Center Power and Cooling Issues, Gartner, 7 September 2007 [4] An Inefficient Truth, Global Action Plan http://www.globalactionplan.org.uk/research.aspx http://www.globalactionplan.org.uk/research.aspx [5] The green datacenter: Energy efficient computing in the 21 st century http://searchdatacenter.techtarget.com/generic/0,295582,sid80_gci1273283,00.html http://searchdatacenter.techtarget.com/generic/0,295582,sid80_gci1273283,00.html [6] Project 05-19-022 – Infrastructure study Data Centre – Burnside Hall, McGill University-restricted, May 4, 2006 [7] Report on the improvement of the Main Server Room conditions Lot 1 – Air distribution, McGill University-restricted, September 4, 2007 CANHEIT | On the EDGE | June 15-18, 2008 | University of Calgary