CMS week, June 2002, CERN 1 First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro

CMS week, June 2002, CERN 2 Infiniband in brief Same network to transport low latency ipc, storage I/O and network I/O Internet Intranet Link speed 1x 2.5 Gbps 4x 10.0 Gbps. 12x 30 Gbps Router xCA Link CPU Mem Cntlr HCA CPU Mem Cntlr HCA CPU Mem Cntlr HCA Link CPU Mem Cntlr HCA Switch Link TCA N/W Target TCA Storage Target Link n Channel based message passing 1000’s node per subnet 1 x Link 4 x Link 12 x Link

CMS week, June 2002, CERN 3 Infiniband Transport Protocols  IBA has been developed with Virtual Interface in mind. VIPL 2.0 includes IBA extensions and RDMA operations.  SCSI RDMA Protocol (SRP). It is a T10 standard. –SRP defines mapping to IBA architecture –it is the transport protocol over IBA –SRP is based on VI  Direct Access Files System (DAFS)  Direct Access Socket (DAS) –TCP/IP functionality over VI/IB IBA Host Channel Adapter Virtual Interface over IB DAS DAFS SRP fast, low latency TCP/IP sockets File Access Block Access fast, low latency network storage

CMS week, June 2002, CERN 4 LNL Infiniband Test Bed Leaf Switch 32 1x (2.5 Gbps) ports in 1 U chassis PCI-X (max ~ 380 Mbyte/s) Supermicro P4DL6 IBA Host Channel Adapter

CMS week, June 2002, CERN 5 Status of Infiniband test bed  All the hardware has been provided by Infiniswitch (1 switch + 4 HCA)  All the hardware is up and running  First p2p measurements have been performed  Software –Virtual Interface Library (VIPL) as provided by Infiniswitch  Send/Receive over reliable connections  RDMA over reliable connections –Sourceforge has a infiniband project over Linux  VIPL source is available. Compiled and works!  Performance as the infiniswitch VIPL (probably they are the same code)  Results –Round trip time small buffers ~ 40  sec (latency 20  sec) –P2p ~ 80% link saturation

CMS week, June 2002, CERN 6 First IB p2p tests 220 Link Saturation Mbyte/s Buffer Size (Bytes) ~ 40 Mbyte/s

CMS week, June 2002, CERN 7 Work Plan  2 months IB hardware evaluation (started end of May)  Mainly focus on functionality tests both of the hardware and software available: –SRP – SCSI remote protocol – already got, to be tested –DAS and DAFS if early version will be available  Contacts started with major companies (IBM, Dell, Fujitsu/Siemens) to understand their IB products road map