ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 1 vlbi_udp Throughput Performance and Stability. Richard Hughes-Jones The University of Manchester then “Talks”

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 2 vlbi_udp Flow Diagram Ring buffer Input thread Ring buffer Send thread Disk Memory Receive thread Output thread Disk Memory From Simon Casey UDP Data TCP Control Control thread Vlbi_control vlbi_send vlbi_recv

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 3 uiGrid2002 monolithic code uConvert to use pthreads control Data input Data output uCode branch for Simon’s file transfer tests uWork on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do – CPU used uNetwork performance – in progress History or the story so far:

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 4 Instability: 2 streams into JIVE, March 2006 From Simon Casey

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 5 L1 ; 0; 0; 0; 100; 0; 0; 0;; L2 ; 0; 0; 0; 99; 0; 0; 0;; L3 ; 0; 0; 0; 99; 0; 0; 0;; L4 ; 2; 0; 43; 53; 0; 1; 18 Early B2B UDP Tests uKernel (May06) uvlbi_recv semaphore uWait 15 us; signal 200 events uUnstable throughput ~815 Mbit/s uPacket loss ~ 0.05% Lost at top of IP stack uInter-packet time uProcessing time mean sigma uCPU load:

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 6 Confirm with B2B UDP Tests uKernel (Aug06) uvlbi_recv semaphore uWait 15 us; signal 100 events uThroughput a bit unstable ~815 Mbit/s uPacket loss ~ 0.05% Lost at top of IP stack uCPU number & % packet loss Have loss when using same CPU as that servicing interrupts uConfirmed by many udpmon tests !

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 7 Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.3% hi, 0.0% si Cpu1 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si Cpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si Cpu3 : 6.0% us, 17.0% sy, 0.0% ni, 40.3% id, 0.0% wa, 1.0% hi, 35.7% si B2B UDP Tests – different kernel uKernel uvlbi_recv semaphore uWait 12 us; signal 200 events uStable throughput 999 Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean sigma uCPU load:

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 8 uCpu0 : 3.7% us, 8.0% sy, 0.0% ni, 88.4% id, 0.0% wa, 0.0% hi, 0.0% si uCpu1 : 1.0% us, 4.0% sy, 0.0% ni, 95.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu3 : 11.6% us, 22.9% sy, 0.0% ni, 41.9% id, 0.0% wa, 1.0% hi, 22.6% si B2B UDP Tests uKernel uvlbi_recv sched_yield() uWait 20 us uStable throughput Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean sigma uCPU load:

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 9 uCpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.3% wa, 0.0% hi, 0.0% si uCpu1 : 11.3% us, 88.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si uCpu3 : 9.3% us, 15.6% sy, 0.0% ni, 37.5% id, 0.0% wa, 1.3% hi, 36.2% si B2B UDP Tests uKernel uvlbi_recv sched_yield() uWait 12 us uStable throughput 999 Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean sigma uCPU load:

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 10 Next Steps uFix the timing precision – again! uTest with latest and kernels uWhy does shed_yield() take more CPU time? Is it a problem or not? uTest on long-distance Networks Production (packet switched) UKLight uTest at 4-8 Gbit

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 11 Any Questions?

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 12 More Information Some URLs 1 uUKLight web site: uMB-NG project web site: uDataTAG project web site: uUDPmon / TCPmon kit + writeup: uMotherboard and NIC Tests: & “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue uTCP tuning information may be found at: & uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet uDante PERT

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 13 uLectures, tutorials etc. on TCP/IP: uEncylopaedia uTCP/IP Resources uUnderstanding IP addresses uConfiguring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt uAssigned protocols, ports etc (RFC 1010) & /etc/protocols More Information Some URLs 2

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 14 Backup Slides

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 15 VLBI Network Topology

ESLEA VLBI Bits&Bytes Workshop, 31 Aug 2006, R. Hughes-Jones Manchester 16 Geant2 Launch event, June 2005