1. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 1 VLBI & Protocols vlbi_udp Multiple Flow Tests Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 2 Setting up the Tests uThank you to the people at all the sites for your help and making facilities available uIncluding: uOnsala uJIVE uTorun and Pioner Poznan uGARR Bologna uJodrell and Manchester Network Group uDante

3. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 3 Dedicated Gbit link Onsala Sweden Gbit link Jodrell Bank UK Dwingeloo DWDM link Medicina Italy Chalmers University of Technology, Gothenburg Torun Poland Gbit link Metsähovi vlbi_udp: Multi-site Streams

4. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 4 uiGrid2002 monolithic code uConvert to use pthreads control Data input Data output uCode branch for Simon’s file transfer tests uWork on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do uMulti-flow Network performance – being set up Nov/Dec06 3 Sites to JIVE: manc UKLight; Manc production; Bologna GEANT PoP Measure: throughput, packet loss, re-ordering, 1-way delay uTo do Add code for: MarkV card and PCEVN interface vlbi_udp: UDP on the WAN

5. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 5 vlbi_udp: Some of the Problems uJIVE made Huygens, mark524 (.54) and mark620 (.59) available uWithin minutes of Arpad leaving the Alteon NIC of mark524 lost the data network! OK used mark623 (.62) – faster CPU uFirewalls needed to allow vlbi_udp ports Aarrgg (!!!) Huygens is SUZE Linux uRouting – well this ALWAYS needs to be fixed !!! uAMD Opteron did not like sched_getaffinity() sched_setaffinity() Comment out this bit uudpmon flows Onsala to JIVE look good uudpmon flows JIVE mark623 to Onsala & Manc UKL don’t work Firewall down stops after 77 udpmon loops Firewall up udpmon cant communicate with Onsala uCPU load issues on the markV systems Don’t seem to be able to keep up with receiveing UDP flow AND emptying the ring buffer uTorun PC / Link lost during the tests

6. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 6 vlbi_udp Onsala  mark623 uIt worked ! uNo not quite !!! uAsked for a flow of 600Mbit/s (-w20 us) got ~300 Mbit/s (-w58 us) u1.8% packet loss u1.1% re-ordered (mainly by 1 or 2 packets) uOnsala  gig7 (Academic internet to Ams then UKLight) Onsala  gig8 (all Academic internet) u~4.5% lost and ~3.5% re-ordered u2.2GHz AMD Athlon uUdpmon OK but single thread uVlbi_udp uses recvmsg() & iovectors

7. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 7 Multiple vlbi_udp Flows uGig7  Huygens UKLight 15 us spacing 800 Mbit/s uGig8  mark623 Academic Internet 20 us spacing 600 Mbit/s uBologna  mark620 Academic Internet 30 us spacing 400 Mbit/s SURFnet Access link SJ5 Access link GARR Access link

8. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 8 Multiple vlbi_udp Flows uGig7  Huygens UKLight 15 us spacing 816 Mbit/s del <1Mbit/s step 1 Mbit/s Zero packet loss Zero re-ordering uGig8  mark623 Academic Internet 20 us spacing 612 Mbit/s 0.6 falling to 0.05% packet loss 0.02 % re-ordering uBologna  mark620 Academic Internet 30 us spacing 396 Mbit/s 0.02 % packet loss 0 % re-ordering

9. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 9 Any Questions?

10. ESLEA Bits&Bytes, Manchester, 7-8 Dec 2006, R. Hughes-Jones Manchester 10 Backup Slides