1. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 1 The Performance of High Throughput Data Flows for e-VLBI in Europe Multi-Gigabit over GÉANT2 Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 2 But will 10 Gigabit Ethernet work on a PC?

3. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 3 High-end Server PCs for 10 Gigabit u Boston/Supermicro X7DBE u Two Dual Core Intel Xeon Woodcrest 5130 2 GHz Independent 1.33GHz FSBuses u 530 MHz FD Memory (serial) Parallel access to 4 banks uChipsets: Intel 5000P MCH – PCIe & Memory ESB2 – PCI-X GE etc. u PCI 3 8 lane PCIe buses 3* 133 MHz PCI-X u 2 Gigabit Ethernet u SATA

4. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 4 10 GigE Back2Back: UDP Latency uMotherboard: Supermicro X7DBE uChipset: Intel 5000P MCH uCPU: 2 Dual Intel Xeon 5130 2 GHz with 4096k L2 cache uMem bus: 2 independent 1.33 GHz uPCI-e 8 lane uLinux Kernel 2.6.20-web100_pktd-plus uMyricom NIC 10G-PCIE-8A-R Fibre umyri10ge v1.2.0 + firmware v1.4.10 rx-usecs=0 Coalescence OFF MSI=1 Checksums ON tx_boundary=4096 uMTU 9000 bytes uLatency 22 µs & very well behaved uLatency Slope 0.0028 µs/byte uB2B Expect: 0.00268 µs/byte Mem0.0004 PCI-e0.00054 10GigE0.0008 PCI-e0.00054 Mem0.0004 uHistogram FWHM ~1-2 us

5. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 5 10 GigE Back2Back: UDP Throughput uKernel 2.6.20-web100_pktd-plus uMyricom 10G-PCIE-8A-R Fibre rx-usecs=25 Coalescence ON uMTU 9000 bytes uMax throughput 9.4 Gbit/s uNotice rate for 8972 byte packet u~0.002% packet loss in 10M packets in receiving host uSending host, 3 CPUs idle uFor 90% in kernel mode inc ~10% soft int uReceiving host 3 CPUs idle uFor <8 µs packets, 1 CPU is 70-80% in kernel mode inc ~15% soft int

6. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 6 10 GigE UDP Throughput vs packet size uMotherboard: Supermicro X7DBE uLinux Kernel 2.6.20-web100_ pktd-plus uMyricom NIC 10G-PCIE-8A-R Fibre umyri10ge v1.2.0 + firmware v1.4.10 rx-usecs=0 Coalescence ON MSI=1 Checksums ON tx_boundary=4096 uSteps at 4060 and 8160 bytes within 36 bytes of 2 n boundaries uModel data transfer time as t= C + m*Bytes C includes the time to set up transfers Fit reasonable C= 1.67 µs m= 5.4 e4 µs/byte Steps consistent with C increasing by 0.6 µs uThe Myricom driver segments the transfers, limiting the DMA to 4096 bytes – PCI-e chipset dependent!

7. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 7 10 GigE X7DBE  X7DBE: TCP iperf uNo packet loss uMTU 9000 uTCP buffer 256k BDP=~330k uCwnd SlowStart then slow growth Limited by sender ! uDuplicate ACKs One event of 3 DupACKs uPackets Re-Transmitted uIperf TCP throughput 7.77 Gbit/s Web100 plots of TCP parameters

8. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 8 OK so it works !!! UDP Performance on the 4Gbit Light Path

9. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 9 ESLEA-FABRIC:4 Gbit flows over GÉANT2 uSet up 4 Gigabit Lightpath Between GÉANT2 PoPs Collaboration with DANTE GÉANT2 Testbed London – Prague – London PCs in the DANTE London PoP with 10 Gigabit NICs uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and udpmon Multi-Gigabit TCP performance with current kernels Multi-Gigabit CBR over TCP/IP Experience for FPGA Ethernet packet systems uDANTE Interests: Multi-Gigabit TCP performance The effect of (Alcatel 1678 MCC 10GE port) buffer size on bursty TCP using BW limited Lightpaths

10. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 10 The GÉANT2 Testbed u10 Gigabit SDH backbone uAlcatel 1678 MCCs uGE and 10GE client interfaces uNode location: London Amsterdam Paris Prague Frankfurt uCan do lightpath routing so make paths of different RTT uLocate the PCs in London

11. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 11 Photos at The PoP 10 GE Test-bed SDH Production SDH Optical Transport Production Router

12. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 12 Provisioning the lightpath on ALCATEL MCCs uSome jiggery-pokery needed with the NMS to force a “looped back” lightpath London-Prague-London uManual XCs (using element manager) possible but hard work 196 needed + other operations! uInstead used RM to create two parallel VC-4-28v (single- ended) Ethernet private line (EPL) paths Constrained to transit DE uThen manually joined paths in CZ Only 28 manually created XCs required !!

13. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 13 Provisioning the lightpath on ALCATEL MCCs uPaths come up u(Transient) alarms clear uResult: provisioned a path of 28 virtually concatenated VC-4s UK-NL-DE-NL-UK uOptical path ~4150 km uWith dispersion compensation ~4900 km uRTT 46.7 ms

14. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 14 4 Gig Flows on GÉANT: UDP Throughput uKernel 2.6.20-web100_pktd- plus uMyricom 10G-PCIE-8A-R Fibre rx-usecs=25 Coalescence ON uMTU 9000 bytes uMax throughput 4.199 Gbit/s uSending host, 3 CPUs idle uFor 90% in kernel mode inc ~10% soft int uReceiving host 3 CPUs idle uFor <8 µs packets, 1 CPU is ~37% in kernel mode inc ~9% soft int

15. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 15 4 Gig Flows on GÉANT: 1-way delay uKernel 2.6.20-web100_pktd-plus uMyricom 10G-PCIE-8A-R Fibre Coalescence OFF u1-way delay stable at 23.435 µs uPeak separation 86 µs u~40 µs extra delay uLab Tests: uPeak separation 86 µs u~40 µs extra delay uLightpath adds no unwanted effects

16. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 16 4 Gig Flows on GÉANT: Jitter histogram uKernel 2.6.20-web100_pktd-plus uMyricom 10G-PCIE-8A-R Fibre Coalescence OFF uPeak separation ~36 µs uFactor 100 smaller Packet separation 300 µs Packet separation 100 µs Lab Tests: Lightpath adds no effects

17. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 17 4 Gig Flows on GÉANT: UDP Flow Stability uKernel 2.6.20-web100_pktd- plus uMyricom 10G-PCIE-8A-R Fibre Coalescence OFF uMTU 9000 bytes uPacket spacing 18 us uTrials send 10 M packets uRan for 26 Hours uThroughput very stable 3.9795 Gbit/s uOccasional trials have packet loss ~40 in 10M – due to loopback in Prague uOur thanks go to all our collaborators uDANTE really provided “Bandwidth on Demand” uA record 6 hours ! including Driving to the PoP Installing the PCs Provisioning the Light-path

18. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 18 Buffer size on Alcatel 1678 MCC

19. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 19 uClassic Bottleneck u10 Gbit/s input 4 Gbit/s output uUse udpmon to send a stream of spaced UDP packets uMeasure packet number of first lost frame as function of w packet spacing Alcatel Buffer size: Method

20. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 20 Alcatel Buffer size: Plots uSlope gives buffer size u~57 kBytes

21. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 21 TCP Performance

22. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 22 4 Gig Flows on GÉANT: TCP iperf uTCP Throughput from iperf uPath Lon-Ams_FF-Prague-Paris-Lon uRtt 55.5 ms uWindow for 1 Gbit/s 6.94 Mbytes uRate: 449 Mbit/s

23. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 23 TCP iperf with Web100 uTCP Throughput from iperf uPath Lon-Ams_FF-Prague-Paris-Lon uRtt 55.5 ms uWindow for 1 Gbit/s 6.94 Mbytes u1 st second with slow start uBy 3 rd round trip: TCP sent 25 packets lost some transmitted packets Dup ACK 5 uBy 4 th round trip: TCP sent 49 packets lost more transmitted packets Dup ACK 31

24. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 24 Use UDP to emulate TCP slowstart uudpmon sends bursts of spaced packets: 32 packets Jumbo 8000 bytes back2back 4 ms between bursts uPath Lon-Ams_FF-Prague-Paris-Lon uRtt 55.5 ms uSee 13 packets then loose 1 in 3 uConfirm the TCP problem!

25. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 25 10 Gigabit Ethernet from the iBoB Slides from Jonathan’s talk

26. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 26 iBOB Under Test

27. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 27 RS232JTAG Optional second CX4 CX4 10Gbps up to 15m iBOB Configured as network testing device Network PC Or Switch Local PC Download FPGA firmware over JTAG Local ‘TinySH’ control over RS232 Removed when firmware is stable 10/100/1000 Ethernet Remote PC Remote login to network PC to run tests from JBO, Manchester or elsewhere iBOB as a Network Testing Device

28. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 28 Network testing device: Simulink design

29. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 29 UDP Throughput vs. Packet Spacing uPC uKernel 2.6.20-web100_pktd-plus uMyricom 10G-PCIE-8A-R CX4 rx-usecs=25 Coalescence ON uMTU 9000 bytes uUDP Packets uMax throughput 9.4 Gbit/s uiBoB uPacket 8234 Data: 8192+ Header: 42 u100 MHz clock uMax rate 6.6 Gbit/s uSee 6.44Gbit/s

30. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 30 Any Questions?

31. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 31 What is EXPReS? uEXPReS = Express Production Real-time e-VLBI Service uThree year project, started March 2006, funded by the European Commission (DG-INFSO), Sixth Framework Programme, Contract #026642 uObjective: to create a distributed, large-scale astronomical instrument of continental and inter-continental dimensions uMeans: high-speed communication networks operating in real-time and connecting some of the largest and most sensitive radio telescopes on the planet uAdditional Information http://expres-eu.org/[note: only one “s”] http://www.jive.nl Introduction

32. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 32 EXPReS Partners Radio Astronomy Institutes uJoint Institute for VLBI in Europe (Coordinator), The Netherlands uArecibo Observatory, National Astronomy and Ionosphere Center, Cornell University, USA uAustralia Telescope National Facility, a Division of CSIRO, Australia uInstitute of Radioastronomy, National Institute for Astrophysics (INAF), Italy uJodrell Bank Observatory, University of Manchester, United Kingdom uMax Planck Institute for Radio Astronomy (MPIfR), Germany uMetsähovi Radio Observatory, Helsinki University of Technology (TKK), Finland uNational Center of Geographical Information, National Geographic Institute (CNIG-IGN), Spain uHartebeesthoek Radio Astronomy Observatory, National Research Foundation, South Africa uNetherlands Foundation for Research in Astronomy (ASTRON), NWO, The Netherlands uOnsala Space Observatory, Chalmers University of Technology, Sweden uShanghai Astronomical Observatory, Chinese Academy of Sciences, China uTorun Centre for Astronomy, Nicolaus Copernicus University, Poland uTransportable Integrated Geodetic Observatory (TIGO), University of Concepción, Chile uVentspils International Radio Astronomy Center, Ventspils University College, Latvia National Research Networks uAARNet, Australia uDANTE, United Kingdom uPoznan Supercomputing and Networking Center, Poland uSURFnet, The Netherlands Introduction

33. EVN-NREN Meeting 19 September 2007, R. Hughes-Jones Manchester 33 Participating EXPReS Telescopes Introduction