Presentation is loading. Please wait.

Presentation is loading. Please wait.

ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 1 Multi-Gigabit Trials on GEANT Collaboration with Dante. Richard Hughes-Jones The.

Published byHugh Hicks Modified over 8 years ago

Similar presentations

Presentation on theme: "ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 1 Multi-Gigabit Trials on GEANT Collaboration with Dante. Richard Hughes-Jones The."— Presentation transcript:

1 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 1 Multi-Gigabit Trials on GEANT Collaboration with Dante. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 2 Outline uWhat is GÉANT2 u What is it interesting? 10 Gigabit Ethernet UDP memory-2-memory flows uOptions using GÉANT Development Network 10 Gbit SDH Network uOptions Using the GÉANT LightPath Service PoP Location for Network tests uPCs and Current 10 Gbit Tests PC Servers Some Test results

3 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 3 GÉANT2 Topology

4 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 4 GÉANT2: The Convergence Solution NREN Access Existing IP Router Existing IP Router GÉANT2 POP B GÉANT2 POP A Managed Lambda’s 1626 LM L2 Matrix L2 TDM Matrix TDM 1678 MCC Dark Fiber EXPReS PC 10 GE EXPReS PC 10 GE

5 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 5 From PoS to Ethernet Connect. Communicate. Collaborate More Economical Architecture Highest Overall Network Availability Flexibility (VLAN management) Highest Network Performance (Latency) Router IP Links 1/10 Gigabit Ethernet VC-4-nv Channels L2 Matrix TDM Matrix 1678 MCC Transport Node VLANs

6 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 6 What do we want to do? uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in their PoPs with 10 Gigabit NICs uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems uDante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP when using BW limited Lightpaths uNeed A Collaboration Agreement

7 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 7 Options Using the GÉANT Development Network u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: London Amsterdam Paris Prague Frankfurt uCan do traffic routing so make long rtt paths uAvailable Dec/Jan 07 uLess Pressure for long term tests

8 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 8 Options Using the GÉANT LightPaths uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in Dante PoPs u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna uCan do traffic routing so make long rtt paths uIdeal: London Copenhagen

9 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 9 4 Gigabit GÉANT LightPath uExample of a 4 Gigabit Lightpath Between GÉANT PoPs PCs in Dante PoPs 26 * VC-4s 4180 Mbit/s

10 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 10 PCs and Current Tests

11 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 11 Test PCs Have Arrived u Boston/Supermicro X7DBE u Two Dual Core Intel Xeon Woodcrest 5130 2 GHz Independent 1.33GHz FSBuses u 530 MHz FD Memory (serial) uChipsets: Intel 5000P MCH – PCIe & Memory ESB2 – PCI-X GE etc. u PCI 3 8 lane PCIe buses 3* 133 MHz PCI-X u 2 Gigabit Ethernet u SATA

12 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 12 Bandwidth Challenge wins Hat Trick uThe maximum aggregate bandwidth was >151 Gbits/s 130 DVD movies in a minute serve 10,000 MPEG2 HDTV movies in real-time u22 10Gigabit Ethernet waves Caltech & SLAC/FERMI booths In 2 hours transferred 95.37 TByte 24 hours moved ~ 475 TBytes uShowed real-time particle event analysis uSLAC Fermi UK Booth: 1 10 Gbit Ethernet to UK NLR&UKLight: transatlantic HEP disk to disk VLBI streaming 2 10 Gbit Links to SALC: rootd low-latency file access application for clusters Fibre Channel StorCloud 4 10 Gbit links to Fermi Dcache data transfers SC2004 101 Gbit/s In to booth Out of booth

13 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 13 SC|05 Seattle-SLAC 10 Gigabit Ethernet u2 Lightpaths: Routed over ESnet Layer 2 over Ultra Science Net u6 Sun V20Z systems per λ udcache remote disk data access 100 processes per node Node sends or receives One data stream 20-30 Mbit/s uUsed Neteion NICs & Chelsio TOE uData also sent to StorCloud using fibre channel links uTraffic on the 10 GE link for 2 nodes: 3-4 Gbit per nodes 8.5-9 Gbit on Trunk

14 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 14 Lab Tests 10 Gigabit Ethernet u10 Gigabit Test Lab being set up in Manchester Cisco 7600 Cross Campus λ <1ms Server quality PCs Neterion NICs Chelsio being purchased uB2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing uPropose to install Fedora Core5 Kernel 2.6.17 on the new Intel dual-core PCs

15 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 15 10 Gigabit Ethernet: TCP Data transfer on PCI-X uSun V20z 1.8GHz to 2.6 GHz Dual Opterons uConnect via 6509 uXFrame II NIC uPCI-X mmrbc 4096 bytes 66 MHz uTwo 9000 byte packets b2b uAve Rate 2.87 Gbit/s uBurst of packets length 646.8 us uGap between bursts 343 us u2 Interrupts / burst CSR Access Data Transfer

16 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 16 10 Gigabit Ethernet: UDP Data transfer on PCI-X uSun V20z 1.8GHz to 2.6 GHz Dual Opterons uConnect via 6509 uXFrame II NIC uPCI-X mmrbc 2048 bytes 66 MHz uOne 8000 byte packets 2.8us for CSRs 24.2 us data transfer effective rate 2.6 Gbit/s u2000 byte packet, wait 0us ~200ms pauses u8000 byte packet, wait 0us ~15ms between data blocks CSR Access 2.8us Data Transfer

17 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 17 Any Questions?

18 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 18 More Information Some URLs 1 uUKLight web site: http://www.uklight.ac.uk uMB-NG project web site: http://www.mb-ng.net/ uDataTAG project web site: http://www.datatag.org/ uUDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net uMotherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt & http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ uTCP tuning information may be found at: http://www.ncne.nlanr.net/documentation/faq/performance.html & http://www.psc.edu/networking/perf_tune.html uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ uDante PERT http://www.geant2.net/server/show/nav.00d00h002

19 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 19 uLectures, tutorials etc. on TCP/IP: www.nv.cc.va.us/home/joney/tcp_ip.htm www.cs.pdx.edu/~jrb/tcpip.lectures.html www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm www.cis.ohio-state.edu/htbin/rfc/rfc1180.html www.jbmelectronics.com/tcp.htm uEncylopaedia http://www.freesoft.org/CIE/index.htm uTCP/IP Resources www.private.org.il/tcpip_rl.html uUnderstanding IP addresses http://www.3com.com/solutions/en_US/ncs/501302.html uConfiguring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt uAssigned protocols, ports etc (RFC 1010) http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols http://www.es.net/pub/rfcs/rfc1010.txt More Information Some URLs 2

20 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 20 Backup Slides

21 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 21 Research Activity Policy Middleware Network Resource Mgr Bandwidth on Demand Our Long-Term Vision Ethernet Applications e.g. GRID Ethernet 1678 MCC Applications e.g. GRID Bandwidth Request Bandwidth Request UNI-C Command GMPLS

22 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 22 SuperComputing

23 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 23 10 Gigabit Ethernet: UDP Throughput u1500 byte MTU gives ~ 2 Gbit/s uUsed 16144 byte MTU max user length 16080 uDataTAG Supermicro PCs uDual 2.2 GHz Xenon CPU FSB 400 MHz uPCI-X mmrbc 512 bytes uwire rate throughput of 2.9 Gbit/s uCERN OpenLab HP Itanium PCs uDual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.7 Gbit/s uSLAC Dell PCs giving a uDual 3.0 GHz Xenon CPU FSB 533 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.4 Gbit/s

24 ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 24 10 Gigabit Ethernet: Tuning PCI-X u16080 byte packets every 200 µs uIntel PRO/10GbE LR Adapter uPCI-X bus occupancy vs mmrbc Measured times Times based on PCI-X times from the logic analyser Expected throughput ~7 Gbit/s Measured 5.7 Gbit/s mmrbc 1024 bytes mmrbc 2048 bytes mmrbc 4096 bytes 5.7Gbit/s mmrbc 512 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update

Download ppt "ESLEA-FABRIC Technical Meeting, 1 Sep 2006, R. Hughes-Jones Manchester 1 Multi-Gigabit Trials on GEANT Collaboration with Dante. Richard Hughes-Jones The."

Similar presentations

MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.

MB - NG MB-NG Technical Meeting 03 May 02 R. Hughes-Jones Manchester 1 Task2 Traffic Generation and Measurement Definitions Pass-1.
DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)

DataTAG CERN Oct 2002 R. Hughes-Jones Manchester Initial Performance Measurements With DataTAG PCs Gigabit Ethernet NICs (Work in progress Oct 02)
CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/

CALICE, Mar 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University of Manchester
JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.

JIVE VLBI Network Meeting 15 Jan 2003 R. Hughes-Jones Manchester The EVN-NREN Project Richard Hughes-Jones The University of Manchester.
GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.

GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 End-2-End Network Monitoring What do we do ? What do we use it for? Richard Hughes-Jones Many people.
Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 1 e-Science work ESLEA & EXPReS vlbi_udp Multiple Flow Tests DCCP Tests EXPReS-Dante Collaboration.

Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 1 e-Science work ESLEA & EXPReS vlbi_udp Multiple Flow Tests DCCP Tests EXPReS-Dante Collaboration.
ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.

ESLEA Closing Conference, Edinburgh, March 2007, R. Hughes-Jones Manchester 1 Protocols Working with 10 Gigabit Ethernet Richard Hughes-Jones The University.
Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.

Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004 R. Hughes-Jones Manchester Networking for ATLAS Remote Farms Richard Hughes-Jones The University.
Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.

Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.
GridPP meeting Feb 03 R. Hughes-Jones Manchester WP7 Networking Richard Hughes-Jones.

GridPP meeting Feb 03 R. Hughes-Jones Manchester WP7 Networking Richard Hughes-Jones.
CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.

CdL was here DataTAG/WP7 Amsterdam June 2002 R. Hughes-Jones Manchester 1 EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester.
ESLEA Technical Collaboration Meeting, 20-21 Jun 2006, R. Hughes-Jones Manchester 1 Protocols Recent and Current Work. Richard Hughes-Jones The University.

ESLEA Technical Collaboration Meeting, Jun 2006, R. Hughes-Jones Manchester 1 Protocols Recent and Current Work. Richard Hughes-Jones The University.
PFLDnet, Nara, Japan 2-3 Feb 2006, R. Hughes-Jones Manchester 1 Transport Benchmarking Panel Discussion Richard Hughes-Jones The University of Manchester.

PFLDnet, Nara, Japan 2-3 Feb 2006, R. Hughes-Jones Manchester 1 Transport Benchmarking Panel Discussion Richard Hughes-Jones The University of Manchester.
5 Annual e-VLBI Workshop, 17-20 September 2006, Haystack Observatory R. Hughes-Jones Manchester 1 The Network Transport layer and the Application or TCP/IP.

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 1 The Network Transport layer and the Application or TCP/IP.
Slide: 1 Richard Hughes-Jones PFLDnet2005 Lyon Feb 05 R. Hughes-Jones Manchester 1 Investigating the interaction between high-performance network and disk.

Slide: 1 Richard Hughes-Jones PFLDnet2005 Lyon Feb 05 R. Hughes-Jones Manchester 1 Investigating the interaction between high-performance network and disk.
DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)

DataGrid WP7 Meeting CERN April 2002 R. Hughes-Jones Manchester Some Measurements on the SuperJANET 4 Production Network (UK Work in progress)
JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University.
T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then.

T2UK RAL 15 Mar 2006, R. Hughes-Jones Manchester 1 ATLAS Networking & T2UK Richard Hughes-Jones The University of Manchester then.
CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones.

CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones.
GEANT2 Network Performance Workshop, 11-12 Jan 200, R. Hughes-Jones Manchester 1 TCP/IP Masterclass or So TCP works … but still the users ask: Where is.

GEANT2 Network Performance Workshop, Jan 200, R. Hughes-Jones Manchester 1 TCP/IP Masterclass or So TCP works … but still the users ask: Where is.

Similar presentations

Ads by Google