EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 1 The Performance of High Throughput Data Flows for e-VLBI in Europe Multi-Gigabit over GÉANT2 Richard Hughes-Jones The University of Manchester then “Talks”

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 2 But will 10 Gigabit Ethernet work on a PC?

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 3 High-end Server PCs for 10 Gigabit u Boston/Supermicro X7DBE u Two Dual Core Intel Xeon Woodcrest GHz Independent 1.33GHz FSBuses u 530 MHz FD Memory (serial) Parallel access to 4 banks uChipsets: Intel 5000P MCH – PCIe & Memory ESB2 – PCI-X GE etc. u PCI 3 8 lane PCIe buses 3* 133 MHz PCI-X u 2 Gigabit Ethernet u SATA

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 4 10 GigE Back2Back: UDP Latency uMotherboard: Supermicro X7DBE uChipset: Intel 5000P MCH uCPU: 2 Dual Intel Xeon GHz with 4096k L2 cache uMem bus: 2 independent 1.33 GHz uPCI-e 8 lane uLinux Kernel web100_pktd-plus uMyricom NIC 10G-PCIE-8A-R Fibre umyri10ge v firmware v rx-usecs=0 Coalescence OFF MSI=1 Checksums ON tx_boundary=4096 uMTU 9000 bytes uLatency 22 µs & very well behaved uLatency Slope µs/byte uB2B Expect: µs/byte Mem PCI-e GigE PCI-e Mem uHistogram FWHM ~1-2 us

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 5 10 GigE Back2Back: UDP Throughput uKernel web100_pktd-plus uMyricom 10G-PCIE-8A-R Fibre rx-usecs=25 Coalescence ON uMTU 9000 bytes uMax throughput 9.4 Gbit/s uNotice rate for 8972 byte packet u~0.002% packet loss in 10M packets in receiving host uSending host, 3 CPUs idle uFor 90% in kernel mode inc ~10% soft int uReceiving host 3 CPUs idle uFor <8 µs packets, 1 CPU is 70-80% in kernel mode inc ~15% soft int

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 6 10 GigE UDP Throughput vs packet size uMotherboard: Supermicro X7DBE uLinux Kernel web100_ pktd-plus uMyricom NIC 10G-PCIE-8A-R Fibre umyri10ge v firmware v rx-usecs=0 Coalescence ON MSI=1 Checksums ON tx_boundary=4096 uSteps at 4060 and 8160 bytes within 36 bytes of 2 n boundaries uModel data transfer time as t= C + m*Bytes C includes the time to set up transfers Fit reasonable C= 1.67 µs m= 5.4 e4 µs/byte Steps consistent with C increasing by 0.6 µs uThe Myricom driver segments the transfers, limiting the DMA to 4096 bytes – PCI-e chipset dependent!

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 7 10 GigE X7DBE  X7DBE: TCP iperf uNo packet loss uMTU 9000 uTCP buffer 256k BDP=~330k uCwnd SlowStart then slow growth Limited by sender ! uDuplicate ACKs One event of 3 DupACKs uPackets Re-Transmitted uIperf TCP throughput 7.77 Gbit/s Web100 plots of TCP parameters

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 8 OK so it works !!! UDP Performance on the 4Gbit Light Path

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 9 ESLEA-FABRIC:4 Gbit flows over GÉANT2 uSet up 4 Gigabit Lightpath Between GÉANT2 PoPs Collaboration with DANTE GÉANT2 Testbed London – Prague – London PCs in the DANTE London PoP with 10 Gigabit NICs uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and udpmon Multi-Gigabit TCP performance with current kernels Multi-Gigabit CBR over TCP/IP Experience for FPGA Ethernet packet systems uDANTE Interests: Multi-Gigabit TCP performance The effect of (Alcatel 1678 MCC 10GE port) buffer size on bursty TCP using BW limited Lightpaths

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 10 The GÉANT2 Testbed u10 Gigabit SDH backbone uAlcatel 1678 MCCs uGE and 10GE client interfaces uNode location: London Amsterdam Paris Prague Frankfurt uCan do lightpath routing so make paths of different RTT uLocate the PCs in London

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 11 Photos at The PoP 10 GE Test-bed SDH Production SDH Optical Transport Production Router

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 12 Provisioning the lightpath on ALCATEL MCCs uSome jiggery-pokery needed with the NMS to force a “looped back” lightpath London-Prague-London uManual XCs (using element manager) possible but hard work 196 needed + other operations! uInstead used RM to create two parallel VC-4-28v (single- ended) Ethernet private line (EPL) paths Constrained to transit DE uThen manually joined paths in CZ Only 28 manually created XCs required !!

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 13 Provisioning the lightpath on ALCATEL MCCs uPaths come up u(Transient) alarms clear uResult: provisioned a path of 28 virtually concatenated VC-4s UK-NL-DE-NL-UK uOptical path ~4150 km uWith dispersion compensation ~4900 km uRTT 46.7 ms

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 14 4 Gig Flows on GÉANT: UDP Throughput uKernel web100_pktd- plus uMyricom 10G-PCIE-8A-R Fibre rx-usecs=25 Coalescence ON uMTU 9000 bytes uMax throughput Gbit/s uSending host, 3 CPUs idle uFor 90% in kernel mode inc ~10% soft int uReceiving host 3 CPUs idle uFor <8 µs packets, 1 CPU is ~37% in kernel mode inc ~9% soft int

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 15 4 Gig Flows on GÉANT: 1-way delay uKernel web100_pktd-plus uMyricom 10G-PCIE-8A-R Fibre Coalescence OFF u1-way delay stable at µs uPeak separation 86 µs u~40 µs extra delay uLab Tests: uPeak separation 86 µs u~40 µs extra delay uLightpath adds no unwanted effects

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 16 4 Gig Flows on GÉANT: Jitter histogram uKernel web100_pktd-plus uMyricom 10G-PCIE-8A-R Fibre Coalescence OFF uPeak separation ~36 µs uFactor 100 smaller Packet separation 300 µs Packet separation 100 µs Lab Tests: Lightpath adds no effects

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 17 4 Gig Flows on GÉANT: UDP Flow Stability uKernel web100_pktd- plus uMyricom 10G-PCIE-8A-R Fibre Coalescence OFF uMTU 9000 bytes uPacket spacing 18 us uTrials send 10 M packets uRan for 26 Hours uThroughput very stable Gbit/s uOccasional trials have packet loss ~40 in 10M – due to loopback in Prague uOur thanks go to all our collaborators uDANTE really provided “Bandwidth on Demand” uA record 6 hours ! including Driving to the PoP Installing the PCs Provisioning the Light-path

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 18 Buffer size on Alcatel 1678 MCC

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 19 uClassic Bottleneck u10 Gbit/s input 4 Gbit/s output uUse udpmon to send a stream of spaced UDP packets uMeasure packet number of first lost frame as function of w packet spacing Alcatel Buffer size: Method

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 20 Alcatel Buffer size: Plots uSlope gives buffer size u~57 kBytes

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 21 TCP Performance

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 22 4 Gig Flows on GÉANT: TCP iperf uTCP Throughput from iperf uPath Lon-Ams_FF-Prague-Paris-Lon uRtt 55.5 ms uWindow for 1 Gbit/s 6.94 Mbytes uRate: 449 Mbit/s

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 23 TCP iperf with Web100 uTCP Throughput from iperf uPath Lon-Ams_FF-Prague-Paris-Lon uRtt 55.5 ms uWindow for 1 Gbit/s 6.94 Mbytes u1 st second with slow start uBy 3 rd round trip: TCP sent 25 packets lost some transmitted packets Dup ACK 5 uBy 4 th round trip: TCP sent 49 packets lost more transmitted packets Dup ACK 31

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 24 Use UDP to emulate TCP slowstart uudpmon sends bursts of spaced packets: 32 packets Jumbo 8000 bytes back2back 4 ms between bursts uPath Lon-Ams_FF-Prague-Paris-Lon uRtt 55.5 ms uSee 13 packets then loose 1 in 3 uConfirm the TCP problem!

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester Gigabit Ethernet from the iBoB Slides from Jonathan’s talk

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 26 iBOB Under Test

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 27 RS232JTAG Optional second CX4 CX4 10Gbps up to 15m iBOB Configured as network testing device Network PC Or Switch Local PC Download FPGA firmware over JTAG Local ‘TinySH’ control over RS232 Removed when firmware is stable 10/100/1000 Ethernet Remote PC Remote login to network PC to run tests from JBO, Manchester or elsewhere iBOB as a Network Testing Device

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 28 Network testing device: Simulink design

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 29 UDP Throughput vs. Packet Spacing uPC uKernel web100_pktd-plus uMyricom 10G-PCIE-8A-R CX4 rx-usecs=25 Coalescence ON uMTU 9000 bytes uUDP Packets uMax throughput 9.4 Gbit/s uiBoB uPacket 8234 Data: Header: 42 u100 MHz clock uMax rate 6.6 Gbit/s uSee 6.44Gbit/s

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 30 Any Questions?

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 31 What is EXPReS? uEXPReS = Express Production Real-time e-VLBI Service uThree year project, started March 2006, funded by the European Commission (DG-INFSO), Sixth Framework Programme, Contract # uObjective: to create a distributed, large-scale astronomical instrument of continental and inter-continental dimensions uMeans: high-speed communication networks operating in real-time and connecting some of the largest and most sensitive radio telescopes on the planet uAdditional Information only one “s”] Introduction

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 32 EXPReS Partners Radio Astronomy Institutes uJoint Institute for VLBI in Europe (Coordinator), The Netherlands uArecibo Observatory, National Astronomy and Ionosphere Center, Cornell University, USA uAustralia Telescope National Facility, a Division of CSIRO, Australia uInstitute of Radioastronomy, National Institute for Astrophysics (INAF), Italy uJodrell Bank Observatory, University of Manchester, United Kingdom uMax Planck Institute for Radio Astronomy (MPIfR), Germany uMetsähovi Radio Observatory, Helsinki University of Technology (TKK), Finland uNational Center of Geographical Information, National Geographic Institute (CNIG-IGN), Spain uHartebeesthoek Radio Astronomy Observatory, National Research Foundation, South Africa uNetherlands Foundation for Research in Astronomy (ASTRON), NWO, The Netherlands uOnsala Space Observatory, Chalmers University of Technology, Sweden uShanghai Astronomical Observatory, Chinese Academy of Sciences, China uTorun Centre for Astronomy, Nicolaus Copernicus University, Poland uTransportable Integrated Geodetic Observatory (TIGO), University of Concepción, Chile uVentspils International Radio Astronomy Center, Ventspils University College, Latvia National Research Networks uAARNet, Australia uDANTE, United Kingdom uPoznan Supercomputing and Networking Center, Poland uSURFnet, The Netherlands Introduction

EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 33 Participating EXPReS Telescopes Introduction