1. A water-cooling solution for PC-racks of the LHC experiments  The cooling concept  Rack Model  Cooling Test  Rack Performance  Failure Tests  Future rack-cooler prototype  Conclusions JCOV- 18Mar04 Nuno Alexandre ELIAS (PH-ATI)

2. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 2 Why test a water-cooling solution ? All LHC experiments will all have a cluster consisting of 1000-2000 rack-mounted PC’s for data-taking (Event Filter Farm). CPU heat dissipation is increasing with clock speed. Assume: 250-300W/PC. Current air-cooling best practice for industrial data-centers of max 1.0-1.2kW per m2, would be very expensive. Water-cooling: can handle high-density heat loads. can profit from available water-cooling infrastructure for electronics. CONCEPT: H orizontally rack-mounted PC’s are cooled with a commercially available, vertically retrofitted heat exchanger, called rack-cooler.

3. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 3 Rack Simplified Model Horizontal mounted PC’s Rack Cooler –Heat Exchanger –Fans Air stream removes heat from PC’s Rack-cooler extracts heat from air Thermodynamic approach Heat Exchanger

4. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 4 Cooling Test-bed in ESS lab CERN 56U 19”- rack (LEP) Test a Liebert rack-mounted cooler / 8 kW nominal capacity. Measurement station running PVSS ( reading temperatures, loads, water flow and power consumption) Configurations with increasing number of PCs. Rack Configuration Melrow mono CPU Elonex mono CPU Elonex dual CPU 30 PCs2550 40 PCs25510 48 PCs25518

5. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 5 Cooling Test-bed

6. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 6 Rack-mounted 1U-PCs MELROW ELONEX ModelsPower consumption OffIdle100% Loaded Elonex mono(5.0 ± 0.5) W(55.0 ± 0.5) W(92.0 ± 0.5) W 40 VA100 VA121 VA Melrow mono(5.0 ± 0.5) W(59.0 ± 0.5) W(96.0 ± 3.0) W -67 VA103 VA Elonex dual(9.0 ± 0.5) W(89.0 ± 2.0) W(167.0 ± 5.0) W 24 VA99 VA180 VA PC configuration of the mono CPU models: single 2.4 GHz CPU, 512MB RAM, 1 Gbit NIC + 1 Fast Ethernet NIC, 40GB disk, PWS 250W (PFC). Perforated front panel serves as air inlet to cool CPU + additional air blower in box. Experience: –Packaging varies: different physical dimensions. –Internal layout is vendor dependent: position of PWS + blower + … 0% => 100% load:  Power= about +35 Watt per CPU

7. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 7 CPU operating temperatures Check operating conditions from point of view of single PC: will CPUs run at lower/higher T in a closed rack ? Good operating conditions for the PCs The rack-cooler removes the heat out of the racks. In rack with rack-cooler: CPUs operate at a slightly lower temperature than without rack-cooler (lower air flow through PC’s). Open rack in air-cooled room - T(room)=23ºC Load 0%Load 100% Elonex mono32 ± 145 ± 2 Melrow mono40 ± 352 ± 3 Elonex dual30 ± 239 ± 2 Closed rack with rackcooler - T(room)=22ºC Elonex mono29 ± 142 ± 1 Melrow mono40 ± 352 ± 3 Elonex dual28 ± 138 ± 1

8. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 8 Air flow measurements Measurements of the airflows for rack with 40 PCs. Measurement systematic error of about 15-20% for all measurements. Observations: –Air flow through the PCs when rack-cooler off is 25% lower than the normal air flow => need to test CPU heating in case of rack-cooler fan failure. –Some “holes and gaps” in a closed rack through which air escapes / is taken in: no problem when limited. –Rack-cooler enhances air flow through PCs by ~20% (consistent with CPU temperature results).

9. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 9 Air temperature mapping PC Load 0% - 48 PCs FrontMiddleBack 21.3 ± 0.327.3 ± 0.320.0 ± 0.3 21.5 ± 0.328.5 ± 0.320.1 ± 0.3 21.4 ± 0.328.2 ± 0.320.8 ± 0.3 21.1 ± 0.327.2 ±0.320.0 ± 0.3 Front AverageMiddle AverageBack Average 21.3 ± 0.227.8 ± 0.620.2 ± 0.4 PC Load 100% - 48 PCs FrontMiddleBack 22.3 ± 0.334.9 ± 0.322.5 ± 0.3 22.6 ± 0.333.5 ± 0.322.2 ± 0.3 22.6 ± 0.333.3 ± 0.322.6 ± 0.3 22.1 ± 0.334.1 ± 0.322.1 ± 0.3 Front AverageMiddle AverageBack Average 22.4 ± 0.334.0 ± 0.722.3 ± 0.2 Efficient and homogeneous cooling of the heated air, back to room temperature (or below). Cooling also for PCs who are above/under direct rack-cooler surface. Advantage over inhomogeneous vertical airflow cooling (vertical temperature gradient). Temperature in the middle increases proportionally with the heat load. Depending on the load the air can exit the rack at a temperature higher/lower than the room.

10. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 10 Dissipated power measurements Difference between Power Dissipated by PC’s and Power removed on rack-cooler increases/decreases room temperature. Heat Loss: P LOSS = P AIR + P SIDE –Air (returned to room at higher/lower temperature) –Side Panels (natural convection + radiation) Highest point: P loss = P in - P out = (0.4±0.2) kW ~ (6±4)% of P in

11. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 11 Rack-cooler Performance P IN vs. P OUT Temperature of the inlet water (14.5ºC, 16.5ºC, 17.5ºC) Room temperature (20.8ºC>>24.1ºC) What’s the trend line? Can we predict other situations? Is it possible to parameterize?

12. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 12 Rack-cooler Performance P IN vs. P OUT (vs.  T*) It’s possible to know what’s the “optimal cooling point” for given conditions Results are ‘co-planar’ In the future the conditions will change only between ‘100% load’ and “idle mode’

13. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 13 Failure Tests Observations: –T_middle , reaches plateau after about 2 hours. –T_side  : heat flow through sides of the rack. Slow increase in room temperature (T_front). –  T(water): still about 50% of P(in) extracted by rack-cooler. –T of CPUs reaches plateau about 8-15 ºC above normal. Recommendations: –Remaining airflow induced by PC fans is enough to keep CPUs running: no immediate action required on PC level. –Fan failure signal would allow for failure handling by DCS. Rack-cooler fans stop (4/4 worst scenario) Central idea: In addition to cooling performance, one needs to check cooling behavior in failure scenarios similar to the ones expected at the experimental sites.

14. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 14 Failure Tests Observations: –T_middle , reaches plateau after about 2 hours. –T_side  : heat flow through sides of the rack. Slow increase in room temperature (T_front). –T(water): goes to room temperature. –T of CPUs rise slowly to about 6-9 ºC above normal after 2 hours. Recommendations: –Unchanged airflow is enough to keep CPUs running: no immediate action required on PC level. –Slow increase in room temperature: warming up of complete rack. Failure will be detected by water flow monitoring in DCS / barrack temperature in DSS. Cooling water stop

15. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 15 New Rack-cooler prototype New 10kW prototype rack-cooler from CIAT with integrated door to close the back of the rack. Thermo-dynamical parameters optimized for our cooling requirements. Delivery in the next weeks To be tested in new DAQ test-bed in 157

16. JCOV Meeting /Mar-04 Nuno Alexandre ELIAS [PH - ATI] 16 Rack Cooler Performance CONCLUSIONS: –Good agreement between theory and measurements –Design parameters play major role on performance –Results show that PC racks can be cooled efficiently with a retro-fitted rack- mounted heat exchanger –Estimation for 10kW was presented using actual data. –Tests on the farm allow us to understand the behavior of the system, and to set up basic parameters for future power dissipation. –A CIAT 10kW rack-cooler was purchased and new tests will start soon. –More detailed results in technical note (draft circulating) http://alicedcs.web.cern.ch/AliceDCS/pcrackcool/Doc/pc_rack_cooling_draftv2.pdf