1. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 1 e-Science work ESLEA & EXPReS vlbi_udp Multiple Flow Tests DCCP Tests EXPReS-Dante Collaboration Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 2 ESLEA and UKLight uExploiting Switched Lightpaths for e-Science Applications uEPSRC e-Science project £1.1M 11.5 FTE Core Technologies: Protocols Control plane HEP data transfers – ATLAS and D0 e-VLBI Medical Applications High Performance Computing uInvestigate how well the protocol implementations work UDP flows, TCP advanced stacks, DCCP (developed by UCL partners) Also examine how the Applications “use” the protocols Also the effect of the transport protocol on what the Application intended! ue-VLBI Develop real-time UDP transport for e-VLBI – vlbi_udp uHEP ATLAS Investigate the performance of a distributed dcache between Lancaster and Manchester linked with UKLight uStephen Kershaw RA Joint with EXPReS

3. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 3 EXPReS & FABRIC uExpress Production Real-time e-VLBI Service uEU Project (DG-INFSO), Sixth Framework Programme, Contract #026642 uAims: to realise the current potential of eVLBI and investigate the Next Generation capabilities. uSSA Use of GRID Farms for distributed correlation. Linking Merlin telescopes to JIVE (present correlator) 4 * 1 Gigabit from Jodrellto JIVE (NL) Links to Service Challenge. Interface to eMERLIN – data input rates at 30 Gbit/s per telescope uJRA - FABRIC Investigate use of different IP Protocols 10 Gigabit from Onsala to Jodrell Links to 10 Gbit HEP work. Investigate 4 Gigabit flows over GEANT2 Switched Lightpaths UDP and TCP Links to Remote Compute Farm HEP work. Develop 1 and 10 Gbit Ethernet end systems using FPGAs Links to CALICE HEP work.

4. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 4 Onsala Sweden Gbit link Jodrell Bank UK Dwingeloo Medicina Italy Chalmers University of Technology, Gothenburg Torun Poland Gbit link Metsähovi vlbi_udp: Multi-site Streams Dedicated DWDM Gbit links

5. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 5 uiGrid2002 monolithic code uConvert to use pthreads control Data input Data output uWork on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do uMulti-flow Network performance – being set up Nov/Dec06 3 Sites to JIVE: manc UKLight; Manc production; Bologna GEANT PoP Measure: throughput, packet loss, re-ordering, 1-way delay vlbi_udp: UDP on the WAN

6. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 6 Multiple vlbi_udp Flows uGig7  Huygens UKLight 15 us spacing 816 Mbit/s del <1Mbit/s step 1 Mbit/s Zero packet loss Zero re-ordering uGig8  mark623 Academic Internet 20 us spacing 612 Mbit/s 0.6 falling to 0.05% packet loss 0.02 % re-ordering uBologna  mark620 Academic Internet 30 us spacing 396 Mbit/s 0.02 % packet loss 0 % re-ordering

7. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 7 Multiple vlbi_udp Flows uGig7  Huygens UKLight 15 us spacing 800 Mbit/s uGig8  mark623 Academic Internet 20 us spacing 600 Mbit/s uBologna  mark620 Academic Internet 30 us spacing 400 Mbit/s SURFnet Access link SJ5 Access link GARR Access link

8. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 8 DCCP: Datagram Congestion Control Protocol uUnreliable No re-transmissions uHas modular congestion control Can detect congestion and take avoiding action Different algorithms can be selected – ccid TCP-like TCP Friendly Rate Control uDCCP is like UDP with congestion control uDCCP is like TCP without reliability uApplication uses Multi-media – send new data instead of re-sending useless old data Applications that can choose data encoding & transmission rate VLBI – discussing a special ccid

9. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 9 DCCP: The Application View uStephen & Richard with help from Andrea uPorted udpmon to dccpmon Some system calls don’t work getsockopt(*soc, SOL_DCCP, DCCP_SOCKOPT_CHANGE_L, &dccp_features, &len); uHad problems with Fedora Core 6 using kernel 2.6.19-rc1 DCCP data packets never reached the receiving TSAP ! Verify with tcpdump Using development “patches” kernel 2.6.19-rc5-g73fd2531-dirty udccpmon tests Plateau ~990 Mbit/s wire rate No packet Loss Receive system crashed! uIperf tests 940Mbps, back-to-back uNeed more instrumentation in DCCP Eg a line in /proc/sys/snmp

10. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 10 DCCP: Latest Kernel uKernel 2.6.19_pktd-plus - ~2 weeks old udccpmon tests Receive system crashed even faster ! Just 1 or 2 1000,000 packet tests uIperf tests OK short runs 940Mbps, back-to-back Hangs longer runs

11. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 11 ESLEA-FABRIC:4 Gbit flows over GÉANT uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante GÉANT Development Network London – Amsterdam and GÉANT Lightpath service CERN – Poznan PCs in their PoPs with 10 Gigabit NICs uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems uDante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP using BW limited Lightpaths

12. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 12 Options Using the GÉANT LightPaths uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in Dante PoPs u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna uCan do traffic routing so make long rtt paths uIdeal: London Copenhagen

13. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 13 EXPReS-Dante Collaboration Doc u1. INTRODUCTION 1.1. Executive Summary 1.2. Time Scales 1.3. The Roles of Dante, The NRNs and EXPReS-FABRIC 1.4. Application area u2. INTRODUCTION TO THE COLLABORATION u3. DISCUSSION OF THE TESTS 3.1. Characterise the end-to-end network with UDP memory-memory flows 3.1.1. Latency and Packet Jitter 3.1.2. Throughput 3.1.3. Packet loss frequency and loss pattern 3.1.4. 1-way delay estimates8 3.2. end-to-end network TCP Throughput 3.3. Operate long term multi-gigabit flows u4. REQUIREMENTS Access to PoPs / Development network Provision of 4 Gigabit Lightpath from VC4s u5. CONCLUSIONS

14. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 14 Any Questions?

15. Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 15 Backup Slides