LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre INFN-GRID-WP2.4: Computing Fabric & Mass Storage LNL L. Berti M. Biasotto M. Gulmini G. Maron N. Toniolo Padova S. Balsamo M. Bellato F. Costa R. Ferrari M. Michelotto I. Saccarola S. Ventura CNAF A. Chierici L. DellAgnello F. Giacomini P. Matteuzzi C. Vistoli S. Zani Lecce G. Aloisio M. Cafaro M. Cafaro Z. Zzzz Z. Zzzz L. Depaolis L. Depaolis S. Campeggio S. Campeggio E. Fasanelli E. Fasanelli Torino A. Forte Genova G. Chiola G. Ciaccio Roma 1 D. Anzellotti C. Battista M. De Rossi F. Marzano S. Falciano A. Spanu E. Valente Bologna G.P. Siroli P. Mazzanti Catania C. Rocca E. Cangiano Tecnologo

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Terminology PC LAN PC network + midleware = PC Farm/Fabric PC Cluster PC Cluster LA N PC Farm network PC Farm network + midleware = GRID

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Why this WP? Commodity components like PCs and LANs are mature to form inexpensive and powerful Computing Fabric Commodity components like PCs and LANs are mature to form inexpensive and powerful Computing Fabric Computing Fabrics located in different sites are integrating to form a Computational/Data GRID Computing Fabrics located in different sites are integrating to form a Computational/Data GRID But But –How to design a fabric of 1000s nodes balancing computing power and efficient storage access? –How to control and monitor the basic system components? –How to publish the monitored values to the GRID monitor system?

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Computing Fabric WP This WP wants to address the mentioned problems adding a technological tracking task to follow and to test, with real use case, the evolution of the basic constituents of a fabric. This WP wants to address the mentioned problems adding a technological tracking task to follow and to test, with real use case, the evolution of the basic constituents of a fabric. WP Break Down WP Break Down –Fabric Design (INFNGrid Wp 2.4) Overall architecture and fabric setup Overall architecture and fabric setup LAN and SAN (System Area Network) technologies LAN and SAN (System Area Network) technologies Communication protocols for high speed network fabrics Communication protocols for high speed network fabrics Storage Systems Storage Systems Microprocessor Technology Microprocessor Technology –Fabric Management (DataGrid WP4) Configuration management and automatic software installation Configuration management and automatic software installation System monitoring System monitoring Dynamic System Partition Dynamic System Partition Problem management Problem management

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Fabric Design - INFNGrid WP Institutions Dipartimento di Scienze dellInformazione - Universita di Venezia Dipartimento di Scienze dellInformazione - Universita di Venezia Dipartimento di Scienze dellinformazione - Universita di Genova Dipartimento di Scienze dellinformazione - Universita di Genova Ingegneria Informatica – Universita di Lecce Ingegneria Informatica – Universita di Lecce 8 sezioni INFN, 1 Laboratorio Nazionale 8 sezioni INFN, 1 Laboratorio Nazionale

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Fabric Management - DataGrid WP4 - Institutions CERN CERN Konrad Zuse Zentrum (Berlin) Konrad Zuse Zentrum (Berlin) Kirchhoff Institute (Heidelberg) Kirchhoff Institute (Heidelberg) IN2P3 (Lyon) IN2P3 (Lyon) INFN INFN Nikhef Nikhef RAL RAL

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Fabric Design Detailed Program ( ) Fabric Architecture Fabric Architecture –Network topologies –Data Server connections and network file systems –System break down Interconnection Networks Interconnection Networks –100/1000 Ethernet –Myrinet –Infiniband Communication protocols for high speed network fabrics Communication protocols for high speed network fabrics Storage Systems Storage Systems –Ultra SCSI (160/320/…) –Ultra and Serial ATA –Storage Area Network (SAN) Fibre Channel Fibre Channel Microprocessor Technology Microprocessor Technology –Dual slim processors –IA64

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Fabric Design Deliverables

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Fabric Design Milestone

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre DATA GRID Fabric Management WP4 DATA GRID Fabric Management WP4 Automatic Software Installation and Management (OS and Applications) Automatic Software Installation and Management (OS and Applications) Configuration Management Configuration Management System Monitoring System Monitoring Problem Management Problem Management Local Authorization Services and Grid Integration Local Authorization Services and Grid Integration

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre WP4 Fabric Managemet Deliverables Requirements document and survey of existing tools and technologies (month 6) Requirements document and survey of existing tools and technologies (month 6) A configuration and installation management demonstrated to work on a cluster of more than 100 nodes (month 12) A configuration and installation management demonstrated to work on a cluster of more than 100 nodes (month 12) A fully deployed service level monitoring system for a computer centre. Hooks to provide remote requests for meta information like policies and quality measures, to allow schedule decisions (month 24) A fully deployed service level monitoring system for a computer centre. Hooks to provide remote requests for meta information like policies and quality measures, to allow schedule decisions (month 24) A fully integrated system to accept remote resource requests in the form of tape mounts r jobs to run and provide monitoring information about progress og requests, and final accounting report back to the sender (month 36) A fully integrated system to accept remote resource requests in the form of tape mounts r jobs to run and provide monitoring information about progress og requests, and final accounting report back to the sender (month 36)

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre NFS Disk Servers Based Farm (Legnaro) Disk Server: -Dual PIII 800 MHz - Mem 512 MB - SCSI Adapters Ethernet Switch: - 10 FE - 1 Uplink GE Computational Nodes: - Dual PIII 800 MHz (40+40 SI95) -Mem 512 MB N1 SWITCHSWITCHSWITCHSWITCH N10 GE FE N2 Farm Module LNL is testing this farm module (with PIII at 450 MHz) DISKSERVER This Farm has been funded By Comm Calcolo for LNL Off-line computation (gr. 2/3) Requests for 2000: - 2 Raid SCSI Controllers 10 Ml ANISBootServer

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre LNL PC Farm (phase I – May 200) N1 SWITCHSWITCHSWITCHSWITCH N10 GE FE N2 Farm Module I DISKSERVER FrontEnd(ANIS) ASUS P3BF 440 BX chipset 1 PIII 450 MHz PCI 33/32 SuperMicro PIIIDME 840 chipset 1-2 PIII 600 MHz PCI 33/32-66/64 ExtremeNetworks Summit Fe 2 Ge 2 ULTRA 160 SCSI ADAPTEC Problemi con Memoria SDRAM

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre DISKSERVER LNL PC Farm (phase II – Nov 2000) N1 SWITCHSWITCHSWITCHSWITCH N15 GE FE N2 Farm Module I DISKSERVER SuperMicro 370DLE ServerWorks III LE chipset 2 PIII 800 MHz 512 Mbyte PCI 33/32-66/64 Case minitower ExtremeNetworks Summit Fe 2 Ge Enclosure Compaq 4354R 14 dischi 36GB SCSI U3 500 Gbyte Storage Array Raid Controller U3 Compaq SmartArray 5300 Case Test: SuperMicro SC810

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Low Price 1U case example: SM SC810 Low Price 1U case example: SM SC810

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre CMS Fabric at LNL (2001) N1 N10 FE N2 Farm Module N1 N10 FE N2 N1 N10 FE N2 N1 N10 FE N2 DISKSERVER DISKSERVER DISKSERVER DISKSERVER GigaEthernet Switch SWITCH SWITCH SWITCH SWITCH HP2524 HP8000FastIron 4000

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre LNL Farm 32 PCs, PIII 600 MHz, 3 Farm modules, 120 Gbps GE/Cu switch, 500 Gbyte disk Used for: - Sadirc/CMS Event Builder prototypes - Data Analysis Production for LNL exps - NFS and topological tests

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre NFS Distributed Disks based Farm (CNAF) SWITCHSWITCHSWITCHSWITCH N10 FE N2 Farm Module ANISBootServer N1 - Topology tests - dual slim processors (rack mounted) tests - remote file systems (nfs, ams, etc. ) tests Requests for 2000: - 10 dual slim processors 90 Ml - Fast ethernet switch 10 Ml - 10 SCSI disks 36/72 GB 20 Ml Ml

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Storage Systems: FC and SCSI road map SCSIUltra3/Ultra MB/sec Ultra2 80 MB/sec Wide Ultra 40 MB/sec Ultra MB/sec Ultra MB/sec Fibre Channel 1Gb 100 MB/sec 2Gb 200 MB/sec 400MB/sec (full duplex) 8/10Gb 1000 MB/sec

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Storage Systems: Fiber Channel/SCSI evaluation (PD) N1 FCSWTFCSWTFCSWTFCSWT N4 FC N2 N3 N1 Arbitrated loop (AL) FC Switched FC Requests for Disk Servers30 Ml ports FC Switch 20 Ml - 4 FC adapters 15 Ml - 2 Raid Arrays 30 Ml - 4 Ultra/160 SCSI disks 8 Ml SCSI disks 8 Ml Ml N1 SCSI

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Storage Systems: Serial ATA (Padova) N1 SerialATA Requests for ATA Cntr.5 Ml

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Interconnection Networks: Gigabit Ethernet (Genova) Communication Protocols for high speed networks: - Communication System GAMMA - Use of Programmable GigaEthernet NICs Base T GigaEthernet (on copper) reliability evaluation - Implementation of efficient parallel/distributed RAID systems Requests for PC24 Ml ports Geth Switch 30 Ml - 8 GEth NICs16 Ml TOTAL70 Ml

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Interconnection Networks: Myrinet Full-duplex Gigabit/second links, switch ports, and interface ports Full-duplex Gigabit/second links, switch ports, and interface ports Flow control, error control, and heartbeat continuity monitoring on every link Flow control, error control, and heartbeat continuity monitoring on every link Low latency, cut-through crossbar switches Low latency, cut-through crossbar switches

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Interconnection Networks: Myrinet (Lecce) N1 MYRINET N10 N2 DISKSERVER MYRINET N10 N2 N1 Requests for Biproc Comp. Nodes50 Ml ports Myrinet Switch 8 Ml - 10 Myrinet NICs35 Ml - 10 disks 72 GB 35 Ml - 1 disk server10 Ml - Rack + cables 5 Ml TOTAL143 Ml Myrinet based farm - Disk Server based - Distributed disks based

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Interconnection Networks: InfiniBand (I) TODAY

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre InfiniBand (II) Legacy host architecture The Infiniband Model

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre InfiniBand (III) What? What? –Initial single link signaling rate of 2.5Gbaud Means unidirectional transfer rate of 250MB/sec with a theoretical full duplex rate of 500MB/sec Means unidirectional transfer rate of 250MB/sec with a theoretical full duplex rate of 500MB/sec –Initial support for single, 4, and 12 wide link widths –Point to point switched fabric –Message based with multicasting support MultiStageSwitch Links TCA I/O Controller Memory Controller PCI-X Host Bridges CPUCPU CPUCPU HCA Link Fibre Channel SCSI Gig. Ethernet Memory HCA- Host Channel Adapter TCA - Target Channel Adapter

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Interconnection Networks: InfiniBand (LNL) Simple test system (4 servers + Storafe Area Network + Network) for 2001 is possible Simple test system (4 servers + Storafe Area Network + Network) for 2001 is possible Early access to the products Early access to the products Test Beds Test Beds Requests 2001 (valuations) Comm V (Sadirc2000) Requests 2001 (valuations) Comm V (Sadirc2000) 4 servers50 Ml 4 servers50 Ml 1 IBA Switch20 Ml 1 IBA Switch20 Ml IBA Adapters10 Ml IBA Adapters10 Ml

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Microprocessor Technology: Intel Itanium (IA64) Requests for 2001 Requests for 2001 –Padova: Investigation IA64 4 PC IA64 40 Ml 4 PC IA64 40 Ml –Lecce: IA64 in Myrinet 10 Dual IA64100 Ml 10 Dual IA64100 Ml 1 Disk Server + 1 Disk Server + disks 16 Ml disks 16 Ml

LNL G. Maron, INFN-GRID, Comm. I INFN, Cagliari, 13 settembre Resources SitesPersonsFTEMaterials(Ml) Ital. Trav. (Ml) Inter.Trav(Ml) Legnaro Padova CNAF Bologna Lecce Torino Genova Catania Roma ** TOTAL