1. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 1 vlbi_udp Throughput Performance and Stability. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 2 vlbi_udp Flow Diagram Ring buffer Input thread Ring buffer Send thread Disk Memory Receive thread Output thread Disk Memory From Simon Casey UDP Data TCP Control Control thread Vlbi_control vlbi_send vlbi_recv

3. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 3 uiGrid2002 monolithic code uConvert to use pthreads control Data input Data output uCode branch for Simon’s file transfer tests uWork on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do – CPU used uNetwork performance – in progress History or the story so far:

4. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 4 Instability: 2 streams into JIVE, March 2006 From Simon Casey

5. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 5 L1 ; 0; 0; 0; 100; 0; 0; 0;; L2 ; 0; 0; 0; 99; 0; 0; 0;; L3 ; 0; 0; 0; 99; 0; 0; 0;; L4 ; 2; 0; 43; 53; 0; 1; 18 Early B2B UDP Tests uKernel 2.6.6 (May06) uvlbi_recv semaphore uWait 15 us; signal 200 events uUnstable throughput ~815 Mbit/s uPacket loss ~ 0.05% Lost at top of IP stack uInter-packet time uProcessing time mean 0.1166 sigma 0.3493 uCPU load:

6. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 6 Confirm with B2B UDP Tests uKernel 2.6.6 (Aug06) uvlbi_recv semaphore uWait 15 us; signal 100 events uThroughput a bit unstable ~815 Mbit/s uPacket loss ~ 0.05% Lost at top of IP stack uCPU number & % packet loss Have loss when using same CPU as that servicing interrupts uConfirmed by many udpmon tests !

7. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 7 Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.3% hi, 0.0% si Cpu1 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si Cpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si Cpu3 : 6.0% us, 17.0% sy, 0.0% ni, 40.3% id, 0.0% wa, 1.0% hi, 35.7% si B2B UDP Tests – different kernel uKernel 2.6.9 uvlbi_recv semaphore uWait 12 us; signal 200 events uStable throughput 999 Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean 0.1206 sigma 0.3274 uCPU load:

8. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 8 uCpu0 : 3.7% us, 8.0% sy, 0.0% ni, 88.4% id, 0.0% wa, 0.0% hi, 0.0% si uCpu1 : 1.0% us, 4.0% sy, 0.0% ni, 95.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu3 : 11.6% us, 22.9% sy, 0.0% ni, 41.9% id, 0.0% wa, 1.0% hi, 22.6% si B2B UDP Tests uKernel 2.6.9 uvlbi_recv sched_yield() uWait 20 us uStable throughput 612.5 Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean 0.1005 sigma 0.1438 uCPU load:

9. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 9 uCpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.3% wa, 0.0% hi, 0.0% si uCpu1 : 11.3% us, 88.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si uCpu3 : 9.3% us, 15.6% sy, 0.0% ni, 37.5% id, 0.0% wa, 1.3% hi, 36.2% si B2B UDP Tests uKernel 2.6.9 uvlbi_recv sched_yield() uWait 12 us uStable throughput 999 Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean 0.1005 sigma 0.1438 uCPU load:

10. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 10 Next Steps uFix the timing precision – again! uTest with latest 2.6.16 and 2.6.17 kernels uWhy does shed_yield() take more CPU time? Is it a problem or not? uTest on long-distance Networks Production (packet switched) UKLight uTest at 4-8 Gbit

11. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 11 Any Questions?

12. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 12 More Information Some URLs 1 uUKLight web site: http://www.uklight.ac.uk uMB-NG project web site: http://www.mb-ng.net/ uDataTAG project web site: http://www.datatag.org/ uUDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net uMotherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt & http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ uTCP tuning information may be found at: http://www.ncne.nlanr.net/documentation/faq/performance.html & http://www.psc.edu/networking/perf_tune.html uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ uDante PERT http://www.geant2.net/server/show/nav.00d00h002

13. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 13 uLectures, tutorials etc. on TCP/IP: www.nv.cc.va.us/home/joney/tcp_ip.htm www.cs.pdx.edu/~jrb/tcpip.lectures.html www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm www.cis.ohio-state.edu/htbin/rfc/rfc1180.html www.jbmelectronics.com/tcp.htm uEncylopaedia http://www.freesoft.org/CIE/index.htm uTCP/IP Resources www.private.org.il/tcpip_rl.html uUnderstanding IP addresses http://www.3com.com/solutions/en_US/ncs/501302.html uConfiguring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt uAssigned protocols, ports etc (RFC 1010) http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols http://www.es.net/pub/rfcs/rfc1010.txt More Information Some URLs 2

14. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 14 Backup Slides

15. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 15 VLBI Network Topology

16. ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 16 Geant2 Launch event, June 2005