Recent experience with PCI-X 2.0 and PCI-E network interfaces and emerging server systems Yang Xia Caltech US LHC Network Working Group October 23, 2006

Wire speed 10GbE cards Neterion’s Xframe II 10 Gigabit Ethernet PCI-X 2.0 –PCI-X MHz/64bit. Compatible with PCI-X 1.0 bus. –32MB Rx Frame Buffer. –Offload engine: TCP/UDP Checksum, LSO, LRO. –Support of PCI Message Signaled Interrupt (MSI) and PCI Extended Message Interrupt (MSI-X) –Fixed Optics (LR or SR). –System support PCI-X 2.0 bus: IBM x366 series –Native PCIe interface card will be available in 2007 Myricom’s 10GbE PCIe –PCIe 1.1 and PCIe 1.0a x8 (8 lane) –Optical XFP transceiver (SR & LR) & CX4 –Standard and half-height PCI faceplate –2MB local memory for firmware execution and for buffering packet headers.

10GbE Capable System – AMD Opteron Processor Opteron motherboards with PCIe support: Tyan: S2891 (1U), S2892, S4881 –AMD-8131 chipset for PCI-X 1.0 slots –nVidia nForce Pro 2200 chipset to support PCI Express x4, x16 slots Supermicro: H8DAR-T, H8DCi –nVidia nForce Pro AMD 8132 to support PCI Express x4, x16 slots –ServerWorks HT HT1000 –AMD-8132 chipset for PCI-X 1.0 slots Integrated memory controller. NUMA support: allows CPU to access local RAM without using the HyperTransport bus.

10GbE Capable System – Intel Xeon Processor IBM x366 system: Xeon MP 3.6GHz, 667MHz system bus, X3 Architechture, 6 available PCI-X MHz slots. Intel woodcrest processor motherboards: –e.g. Supermicro X7DBR-E (1U) & X7DBE Dual Intel Xeon GHz woodcrest processor (4 cores/system) Much improved Northbridge speed: Front-side bus is 64-bit wide, 1333MHz data bus that transfers data at 10.7 GB/sec. Unlike AMD's Opteron processors, none of Intel's forthcoming chips comes with an integrated memory controller. In future chips, it may integrate memory controllers, which let the processor get data from memory. DDR2 1GB 667MHZ ECC Fully buffered memory. PCIe x8 and PCI-X interface slots. 4 SATA disks in 1U form factor: - read ~220MB/s – write ~150MB/s (from FDT) 24 SATA disks on 2 Areca 1230 PCIe controllers - read ~650MB/s - write ~450MB/s

10GbE Capable System – Intel Xeon Processor Woodcrest will consume a maximum of 80 watts, previous Xeon processor use110-watt. SPECint power rating for woodcrest processor is ~17% less than Opteron 285 processor: –SPECint: 281.1watts vs 327.7watts. –Actual measurement: 220watts vs 259watts (~18% power reduction). CMSSW overall factor of 2 compare to Xeon and 1.5 compare to Opteron 248.

Block Diagram of the Intel 5000P Chipset PCIe x8 FSB 1333MHz

Neterion Xframe II WAN performance on USN  Caltech-Starlight-Caltech 155ms RTT network  Small 64MB TCP window size  8 TCP streams from Iperf  1hr average 9.2 Gbits/s  9.9Gbits/s performance in LAN connection.

Myri-10GbE PCIe LAN performance  Intel woodcrest 5160 system  Cisco GbE port with flowcontrol on.  Single TCP stream from Iperf  1hr average 9.88 Gbits/s

Myri-10GbE PCIe WAN performance on USN Single TCP Stream Multiple TCP Streams RHEL AS4.4 x86_64, Kernel ELsmp

Caltech/CERN SC’06 Demos Plan to use new Intel woodcrest systems in the booth from HP. Storage systems: Bluearc (IP), Data Direct Network (4Gbits/s FC)and Nimbus (iSCSI). Support from Cisco: chassis, new GbE module. Greater use of storage-to-storage transfers with PCIe NICs to match 10 Gbps links Disk-to-disk transfers of REAL scientific data. FDT: –A scalable Streaming Data Transfer application: coupled to VINCI dynamic circuit provisioning services; Intend to be integrated with dcache/SRM and other production storage systems. bbcp. Gridftp. Fully utilizes 10 Gig paths from Tampa to Caltech, CERN, Starlight and International sites (Korea via Gloriad, Brazil via Ampath).

SC’06 Available Network Resources Note: NLR circuits and A-wave need to be scheduled. NLR circuits are guaranteed to have dedicated 10 GE access for 1 hour at a time for any application.