ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 1 Multi-Gigabit Trials on GEANT Collaboration with Dante. Richard Hughes-Jones The University of Manchester then “Talks”

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 2 Outline uWhat is GÉANT2 u What is it interesting? 10 Gigabit Ethernet UDP memory-2-memory flows uOptions using GÉANT Development Network 10 Gbit SDH Network uOptions Using the GÉANT LightPath Service PoP Location for Network tests uPCs and Current 10 Gbit Tests PC Servers Some Test results

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 3 GÉANT2 Topology

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 4 GÉANT2: The Convergence Solution NREN Access Existing IP Router Existing IP Router GÉANT2 POP B GÉANT2 POP A Managed Lambda’s 1626 LM L2 Matrix L2 TDM Matrix TDM 1678 MCC Dark Fiber EXPReS PC 10 GE EXPReS PC 10 GE

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 5 From PoS to Ethernet Connect. Communicate. Collaborate More Economical Architecture Highest Overall Network Availability Flexibility (VLAN management) Highest Network Performance (Latency) Router IP Links 1/10 Gigabit Ethernet VC-4-nv Channels L2 Matrix TDM Matrix 1678 MCC Transport Node VLANs

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 6 What do we want to do? uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in their PoPs with 10 Gigabit NICs uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems uDante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP when using BW limited Lightpaths uNeed A Collaboration Agreement

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 7 Options Using the GÉANT Development Network u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: London Amsterdam Paris Prague Frankfurt uCan do traffic routing so make long rtt paths uAvailable Dec/Jan 07 uLess Pressure for long term tests

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 8 Options Using the GÉANT LightPaths uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in Dante PoPs u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna uCan do traffic routing so make long rtt paths uIdeal: London Copenhagen

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 9 4 Gigabit GÉANT LightPath uExample of a 4 Gigabit Lightpath Between GÉANT PoPs PCs in Dante PoPs 26 * VC-4s 4180 Mbit/s

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 10 PCs and Current Tests

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 11 Test PCs Have Arrived u Boston/Supermicro X7DBE u Two Dual Core Intel Xeon Woodcrest GHz Independent 1.33GHz FSBuses u 530 MHz FD Memory (serial) uChipsets: Intel 5000P MCH – PCIe & Memory ESB2 – PCI-X GE etc. u PCI 3 8 lane PCIe buses 3* 133 MHz PCI-X u 2 Gigabit Ethernet u SATA

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 12 Bandwidth Challenge wins Hat Trick uThe maximum aggregate bandwidth was >151 Gbits/s 130 DVD movies in a minute serve 10,000 MPEG2 HDTV movies in real-time u22 10Gigabit Ethernet waves Caltech & SLAC/FERMI booths In 2 hours transferred TByte 24 hours moved ~ 475 TBytes uShowed real-time particle event analysis uSLAC Fermi UK Booth: 1 10 Gbit Ethernet to UK NLR&UKLight: transatlantic HEP disk to disk VLBI streaming 2 10 Gbit Links to SALC: rootd low-latency file access application for clusters Fibre Channel StorCloud 4 10 Gbit links to Fermi Dcache data transfers SC Gbit/s In to booth Out of booth

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 13 SC|05 Seattle-SLAC 10 Gigabit Ethernet u2 Lightpaths: Routed over ESnet Layer 2 over Ultra Science Net u6 Sun V20Z systems per λ udcache remote disk data access 100 processes per node Node sends or receives One data stream Mbit/s uUsed Neteion NICs & Chelsio TOE uData also sent to StorCloud using fibre channel links uTraffic on the 10 GE link for 2 nodes: 3-4 Gbit per nodes Gbit on Trunk

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 14 Lab Tests 10 Gigabit Ethernet u10 Gigabit Test Lab being set up in Manchester Cisco 7600 Cross Campus λ <1ms Server quality PCs Neterion NICs Chelsio being purchased uB2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing uPropose to install Fedora Core5 Kernel on the new Intel dual-core PCs

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester Gigabit Ethernet: TCP Data transfer on PCI-X uSun V20z 1.8GHz to 2.6 GHz Dual Opterons uConnect via 6509 uXFrame II NIC uPCI-X mmrbc 4096 bytes 66 MHz uTwo 9000 byte packets b2b uAve Rate 2.87 Gbit/s uBurst of packets length us uGap between bursts 343 us u2 Interrupts / burst CSR Access Data Transfer

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester Gigabit Ethernet: UDP Data transfer on PCI-X uSun V20z 1.8GHz to 2.6 GHz Dual Opterons uConnect via 6509 uXFrame II NIC uPCI-X mmrbc 2048 bytes 66 MHz uOne 8000 byte packets 2.8us for CSRs 24.2 us data transfer effective rate 2.6 Gbit/s u2000 byte packet, wait 0us ~200ms pauses u8000 byte packet, wait 0us ~15ms between data blocks CSR Access 2.8us Data Transfer

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 17 Any Questions?

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 18 More Information Some URLs 1 uUKLight web site: uMB-NG project web site: uDataTAG project web site: uUDPmon / TCPmon kit + writeup: uMotherboard and NIC Tests: & “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue uTCP tuning information may be found at: & uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet uDante PERT

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 19 uLectures, tutorials etc. on TCP/IP: uEncylopaedia uTCP/IP Resources uUnderstanding IP addresses uConfiguring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt uAssigned protocols, ports etc (RFC 1010) & /etc/protocols More Information Some URLs 2

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 20 Backup Slides

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 21 Research Activity Policy Middleware Network Resource Mgr Bandwidth on Demand Our Long-Term Vision Ethernet Applications e.g. GRID Ethernet 1678 MCC Applications e.g. GRID Bandwidth Request Bandwidth Request UNI-C Command GMPLS

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 22 SuperComputing

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester Gigabit Ethernet: UDP Throughput u1500 byte MTU gives ~ 2 Gbit/s uUsed byte MTU max user length uDataTAG Supermicro PCs uDual 2.2 GHz Xenon CPU FSB 400 MHz uPCI-X mmrbc 512 bytes uwire rate throughput of 2.9 Gbit/s uCERN OpenLab HP Itanium PCs uDual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.7 Gbit/s uSLAC Dell PCs giving a uDual 3.0 GHz Xenon CPU FSB 533 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.4 Gbit/s

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester Gigabit Ethernet: Tuning PCI-X u16080 byte packets every 200 µs uIntel PRO/10GbE LR Adapter uPCI-X bus occupancy vs mmrbc Measured times Times based on PCI-X times from the logic analyser Expected throughput ~7 Gbit/s Measured 5.7 Gbit/s mmrbc 1024 bytes mmrbc 2048 bytes mmrbc 4096 bytes 5.7Gbit/s mmrbc 512 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update