1. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 1 INFN-GRID-WP2.4: Computing Fabric & Mass Storage LNL L. Berti M. Biasotto M. Gulmini G. Maron N. Toniolo Padova S. Balsamo M. Bellato F. Costa R. Ferrari M. Michelotto I. Saccarola S. Ventura CNAF A. Chierici L. DellAgnello F. Giacomini P. Matteuzzi C. Vistoli S. Zani Lecce G. Aloisio M. Cafaro M. Cafaro Z. Zzzz Z. Zzzz L. Depaolis L. Depaolis S. Campeggio S. Campeggio E. Fasanelli E. Fasanelli Torino A. Forte Genova G. Chiola G. Ciaccio Roma 1 D. Anzellotti C. Battista M. De Rossi F. Marzano S. Falciano A. Spanu E. Valente Bologna G.P. Siroli P. Mazzanti Catania C. Rocca E. Cangiano Tecnologo

2. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 2 Terminology PC LAN PC network + midleware = PC Farm/Fabric PC Cluster PC Cluster LA N PC Farm network PC Farm network + midleware = GRID

3. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 3 Why this WP? Commodity components like PCs and LANs are mature to form inexpensive and powerful Computing Fabric Commodity components like PCs and LANs are mature to form inexpensive and powerful Computing Fabric Computing Fabrics located in different sites are integrating to form a Computational/Data GRID Computing Fabrics located in different sites are integrating to form a Computational/Data GRID But But –How to design a fabric of 1000s nodes balancing computing power and efficient storage access? –How to control and monitor the basic system components? –How to publish the monitored values to the GRID monitor system?

4. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 4 Computing Fabric WP This WP wants to address the mentioned problems adding a technological tracking task to follow and to test, with real use case, the evolution of the basic constituents of a fabric. This WP wants to address the mentioned problems adding a technological tracking task to follow and to test, with real use case, the evolution of the basic constituents of a fabric. WP Break Down WP Break Down –Fabric Design (INFNGrid Wp 2.4) Overall architecture and fabric setup Overall architecture and fabric setup LAN and SAN (System Area Network) technologies LAN and SAN (System Area Network) technologies Communication protocols for high speed network fabrics Communication protocols for high speed network fabrics Storage Systems Storage Systems Microprocessor Technology Microprocessor Technology –Fabric Management (DataGrid WP4) Configuration management and automatic software installation Configuration management and automatic software installation System monitoring System monitoring Dynamic System Partition Dynamic System Partition Problem management Problem management

5. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 5 Fabric Design - INFNGrid WP 2.4 - Institutions Dipartimento di Scienze dellInformazione - Universita di Venezia Dipartimento di Scienze dellInformazione - Universita di Venezia Dipartimento di Scienze dellinformazione - Universita di Genova Dipartimento di Scienze dellinformazione - Universita di Genova Ingegneria Informatica – Universita di Lecce Ingegneria Informatica – Universita di Lecce 8 sezioni INFN, 1 Laboratorio Nazionale 8 sezioni INFN, 1 Laboratorio Nazionale

6. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 6 Fabric Management - DataGrid WP4 - Institutions CERN CERN Konrad Zuse Zentrum (Berlin) Konrad Zuse Zentrum (Berlin) Kirchhoff Institute (Heidelberg) Kirchhoff Institute (Heidelberg) IN2P3 (Lyon) IN2P3 (Lyon) INFN INFN Nikhef Nikhef RAL RAL

7. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 7 Fabric Design Detailed Program (2000-2001) Fabric Architecture Fabric Architecture –Network topologies –Data Server connections and network file systems –System break down Interconnection Networks Interconnection Networks –100/1000 Ethernet –Myrinet –Infiniband Communication protocols for high speed network fabrics Communication protocols for high speed network fabrics Storage Systems Storage Systems –Ultra SCSI (160/320/…) –Ultra and Serial ATA –Storage Area Network (SAN) Fibre Channel Fibre Channel Microprocessor Technology Microprocessor Technology –Dual slim processors –IA64

8. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 8 Fabric Design Deliverables

9. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 9 Fabric Design Milestone

10. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 10 DATA GRID Fabric Management WP4 DATA GRID Fabric Management WP4 Automatic Software Installation and Management (OS and Applications) Automatic Software Installation and Management (OS and Applications) Configuration Management Configuration Management System Monitoring System Monitoring Problem Management Problem Management Local Authorization Services and Grid Integration Local Authorization Services and Grid Integration

11. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 11 WP4 Fabric Managemet Deliverables Requirements document and survey of existing tools and technologies (month 6) Requirements document and survey of existing tools and technologies (month 6) A configuration and installation management demonstrated to work on a cluster of more than 100 nodes (month 12) A configuration and installation management demonstrated to work on a cluster of more than 100 nodes (month 12) A fully deployed service level monitoring system for a computer centre. Hooks to provide remote requests for meta information like policies and quality measures, to allow schedule decisions (month 24) A fully deployed service level monitoring system for a computer centre. Hooks to provide remote requests for meta information like policies and quality measures, to allow schedule decisions (month 24) A fully integrated system to accept remote resource requests in the form of tape mounts r jobs to run and provide monitoring information about progress og requests, and final accounting report back to the sender (month 36) A fully integrated system to accept remote resource requests in the form of tape mounts r jobs to run and provide monitoring information about progress og requests, and final accounting report back to the sender (month 36)

12. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 12 NFS Disk Servers Based Farm (Legnaro) Disk Server: -Dual PIII 800 MHz - Mem 512 MB - SCSI Adapters Ethernet Switch: - 10 FE - 1 Uplink GE Computational Nodes: - Dual PIII 800 MHz (40+40 SI95) -Mem 512 MB N1 SWITCHSWITCHSWITCHSWITCH N10 GE FE N2 Farm Module LNL is testing this farm module (with PIII at 450 MHz) DISKSERVER This Farm has been funded By Comm Calcolo for LNL Off-line computation (gr. 2/3) Requests for 2000: - 2 Raid SCSI Controllers 10 Ml ANISBootServer

13. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 13 LNL PC Farm (phase I – May 200) N1 SWITCHSWITCHSWITCHSWITCH N10 GE FE N2 Farm Module I DISKSERVER FrontEnd(ANIS) ASUS P3BF 440 BX chipset 1 PIII 450 MHz PCI 33/32 SuperMicro PIIIDME 840 chipset 1-2 PIII 600 MHz PCI 33/32-66/64 ExtremeNetworks Summit 48 48 Fe 2 Ge 2 ULTRA 160 SCSI ADAPTEC 39160 Problemi con Memoria SDRAM

14. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 14 DISKSERVER LNL PC Farm (phase II – Nov 2000) N1 SWITCHSWITCHSWITCHSWITCH N15 GE FE N2 Farm Module I DISKSERVER SuperMicro 370DLE ServerWorks III LE chipset 2 PIII 800 MHz 512 Mbyte PCI 33/32-66/64 Case minitower ExtremeNetworks Summit 48 48 Fe 2 Ge Enclosure Compaq 4354R 14 dischi 36GB SCSI U3 500 Gbyte Storage Array Raid Controller U3 Compaq SmartArray 5300 Case Test: SuperMicro SC810

15. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 15 Low Price 1U case example: SM SC810 Low Price 1U case example: SM SC810

16. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 16 CMS Fabric at LNL (2001) N1 N10 FE N2 Farm Module N1 N10 FE N2 N1 N10 FE N2 N1 N10 FE N2 DISKSERVER DISKSERVER DISKSERVER DISKSERVER GigaEthernet Switch SWITCH SWITCH SWITCH SWITCH HP2524 HP8000FastIron 4000

17. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 17 LNL Farm 32 PCs, PIII 600 MHz, 3 Farm modules, 120 Gbps GE/Cu switch, 500 Gbyte disk Used for: - Sadirc/CMS Event Builder prototypes - Data Analysis Production for LNL exps - NFS and topological tests

18. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 18 NFS Distributed Disks based Farm (CNAF) SWITCHSWITCHSWITCHSWITCH N10 FE N2 Farm Module ANISBootServer N1 - Topology tests - dual slim processors (rack mounted) tests - remote file systems (nfs, ams, etc. ) tests Requests for 2000: - 10 dual slim processors 90 Ml - Fast ethernet switch 10 Ml - 10 SCSI disks 36/72 GB 20 Ml ------ ------ 120 Ml

19. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 19 Storage Systems: FC and SCSI road map SCSIUltra3/Ultra160 160 MB/sec Ultra2 80 MB/sec Wide Ultra 40 MB/sec Ultra320 320 MB/sec Ultra640 640 MB/sec 2004200320022001200019991998199719961995 Fibre Channel 1Gb 100 MB/sec 2Gb 200 MB/sec 400MB/sec (full duplex) 8/10Gb 1000 MB/sec

20. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 20 Storage Systems: Fiber Channel/SCSI evaluation (PD) N1 FCSWTFCSWTFCSWTFCSWT N4 FC N2 N3 N1 Arbitrated loop (AL) FC Switched FC Requests for 2000 - 4 Disk Servers30 Ml - 1 8 ports FC Switch 20 Ml - 4 FC adapters 15 Ml - 2 Raid Arrays 30 Ml - 4 Ultra/160 SCSI disks 8 Ml SCSI disks 8 Ml ------ ------ 103 Ml N1 SCSI

21. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 21 Storage Systems: Serial ATA (Padova) N1 SerialATA Requests for 2001 - 4 ATA Cntr.5 Ml

22. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 22 Interconnection Networks: Gigabit Ethernet (Genova) Communication Protocols for high speed networks: - Communication System GAMMA - Use of Programmable GigaEthernet NICs - 1000 Base T GigaEthernet (on copper) reliability evaluation - Implementation of efficient parallel/distributed RAID systems Requests for 2000 - 8 PC24 Ml - 1 12 ports Geth Switch 30 Ml - 8 GEth NICs16 Ml - ------ TOTAL70 Ml

23. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 23 Interconnection Networks: Myrinet Full-duplex 2.5+2.5 Gigabit/second links, switch ports, and interface ports Full-duplex 2.5+2.5 Gigabit/second links, switch ports, and interface ports Flow control, error control, and heartbeat continuity monitoring on every link Flow control, error control, and heartbeat continuity monitoring on every link Low latency, cut-through crossbar switches Low latency, cut-through crossbar switches

24. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 24 Interconnection Networks: Myrinet (Lecce) N1 MYRINET N10 N2 DISKSERVER MYRINET N10 N2 N1 Requests for 2000 - 10 Biproc Comp. Nodes50 Ml - 1 16 ports Myrinet Switch 8 Ml - 10 Myrinet NICs35 Ml - 10 disks 72 GB 35 Ml - 1 disk server10 Ml - Rack + cables 5 Ml - ------ TOTAL143 Ml Myrinet based farm - Disk Server based - Distributed disks based

25. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 25 Interconnection Networks: InfiniBand (I) TODAY

26. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 26 InfiniBand (II) Legacy host architecture The Infiniband Model

27. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 27 InfiniBand (III) What? What? –Initial single link signaling rate of 2.5Gbaud Means unidirectional transfer rate of 250MB/sec with a theoretical full duplex rate of 500MB/sec Means unidirectional transfer rate of 250MB/sec with a theoretical full duplex rate of 500MB/sec –Initial support for single, 4, and 12 wide link widths –Point to point switched fabric –Message based with multicasting support MultiStageSwitch Links TCA I/O Controller Memory Controller PCI-X Host Bridges CPUCPU CPUCPU HCA Link Fibre Channel SCSI Gig. Ethernet Memory HCA- Host Channel Adapter TCA - Target Channel Adapter

28. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 28 Interconnection Networks: InfiniBand (LNL) Simple test system (4 servers + Storafe Area Network + Network) for 2001 is possible Simple test system (4 servers + Storafe Area Network + Network) for 2001 is possible Early access to the products Early access to the products Test Beds Test Beds Requests 2001 (valuations) Comm V (Sadirc2000) Requests 2001 (valuations) Comm V (Sadirc2000) 4 servers50 Ml 4 servers50 Ml 1 IBA Switch20 Ml 1 IBA Switch20 Ml IBA Adapters10 Ml IBA Adapters10 Ml

29. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 29 Microprocessor Technology: Intel Itanium (IA64) Requests for 2001 Requests for 2001 –Padova: Investigation IA64 4 PC IA64 40 Ml 4 PC IA64 40 Ml –Lecce: IA64 in Myrinet 10 Dual IA64100 Ml 10 Dual IA64100 Ml 1 Disk Server + 1 Disk Server + disks 16 Ml disks 16 Ml

30. LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre 2000 30 Resources SitesPersonsFTEMaterials(Ml) 2000 2001 Ital. Trav. (Ml) 2000 2001 Inter.Trav(Ml) Legnaro52.61080 Padova72.110345 CNAF61.7120- Bologna20.6-- Lecce61.4143116 Torino10.2-- Genova21.070- Catania30.7-- Roma 1 71.5** TOTAL3911.8446161241