1. PIC port d’informació científica First operational experience from a compact, highly energy efficient data center module V. Acín, R. Cruz, M. Delfino, F. Martínez, M. Rodríguez, P. Tallada

2. PIC port d’informació científica Index  PIC Datacenter  Known limitations  BOX2010 project  BOX2010 results  BOX2010 realities  Monitoring and instrumentation  Future Plans 2

3. PIC port d’informació científica PIC Datacenter 3 Building D Campus UAB: Total electrical power: 2000KVA Common cooling infrastructure for computer room and offices:1230 KW cooling UAB business services in another room of the same building 150 m 2 machine room + 25 m 2 module for PIC 200 KVA IT UPS + 100 KVA IT UPS Module ~ 2500 CPU cores Cluster ~ 3,5 Petabytes disk ~ 3,4 Petabytes tape, STK-8500 and IBM-3584 libraries

4. PIC port d’informació científica Needed to allocate 50KW IT CPU for 2010 Known limitations – Power Usage Effectiveness (PUE) estimated at 2.3 We would need 65 KW extra for cooling – Obsolete cooling system with no capacity to increase power Options: Change cooling infrastructure: too expensive Provide a more efficent Data Center Module 4

5. PIC port d’informació científica BOX 2010 Project INITIAL IDEA : Build an “air conditioner that computes” using standard “commodity” components installed in basement storage space SOLUTION: 25 m 2 x 2,2 m height AST SMART-SHELTER 2 standard gas expansion A/C 40 KW cooling (redundant, no UPS) 1 x100 KVA UPS for IT Power Smart Electrical Panel – Power Meter on each Circuit – Isolate completely cold and hot aisles, making it possible to increase operational temperature – Heavily instrumented: collect environmental data to study temperature distribution and cooling behaviour 5

6. PIC port d’informació científica BOX 2010 Realities 6

7. PIC port d’informació científica BOX 2010 Realities IT Power: 100 KVA UPS = 80 KW UPS = 65 KW IT Complete isolation is not possible with normal racks Varying A/C temperature set point not allowed by manufacturer !!! Erratic behaviour of humidity still under investigation 7

8. PIC port d’informació científica Monitoring and instrumentation Monitoring and instrumentation from a single management console using CENTREON: * Module hosts 8 industrial sensors, output over ethernet SNMP 8 UPSSNMPManaging TRAPS from CENTREON CRAC (i.e. A/C units)HTTPPython scripts ELECTRIC CONSUMPTION SCADASelfmade programs translate ModBus to TCP/IP TEMPERATURESSNMPPrototype* designed by PIC and IFAE engineers HUMITIDYSNMPPrototype designed by PIC and IFAE engineers SYSTEM POWER CONSUMPTION TCP/IPGetting information from ILOM

9. PIC port d’informació científica Monitoring and instrumentation 9 Using ModBus Python libraries electrical information is integrated on a Centreon console

10. PIC port d’informació científica Monitoring and instrumentation 10 Electrical meters allow us to calculate instantaneous PUE Humidity and temperature sensors inform Centreon via TCP/IP about air distribution status

11. PIC port d’informació científica Monitoring and instrumentation 11 Single BladeCenter : System operating temperatures and power consumption

12. PIC port d’informació científica Future plans Integrate information in order to be able to automatically react on systems when an infrastructure problem is detected Improve isolation between cold and hot aisles Increase A/C temperature setpoint and measure PUE as a function of it Compare behaviour to CFD simulations Study schemes to conduct hot air directly to A/C intakes Design and build the next module ???? 12

13. PIC port d’informació científica Future plans 13