1. Experiences in Design and Implementation of a High Performance Transport Protocol Yunhong Gu, Xinwei Hong, and Robert L. Grossman National Center for Data Mining

2. Outline TCP’s inefficiency in grid applications UDT Design issues Implementations issues Conclusion and future work

3. TCP and AIMD TCP has been very successful in the Internet –AIMD (Additive Increase Multiplicative Decrease) Fair: max-min fairness Stable: globally asynchronously stable But, inefficient and not scalable –In grid networks (with high bandwidth-delay product) RTT bias

4. Efficiency of TCP 1 Gb/s link, 200ms RTT, between Tokyo and Chicago 28 minutes On 10 Gb/s link, 200ms RTT, it will take 4 hours 43 minutes to recover from a single loss. TCP’s throughput model: It needs extremely low loss rate on high bandwidth-delay product networks.

5. Fairness of TCP 100ms 1 Gb/s 1ms 1Gb/s Merge two real-time data streams From Chicago 1 to Chicago 2: 800Mbps From Amsterdam to Chicago 2: 80Mbps The throughput is limited by the slowest stream! Amsterdam Chicago 2 Chicago 1

6. UDT – UDP-based Data Transfer Protocol Application level transport protocol built above UDP Reliable data delivery End-to-end approach Bi-directional General transport API; not a (file transfer) tool. Open source

7. UDT Architecture DATA ACK ACK2 NAK Sender Recver Sender Recver  Pkt. Scheduling Timer  ACK Timer  NAK Timer  Retransmission Timer  Rate Control Timer Sender

8. UDT – Objectives Goals –Easy to install and use –Efficient for bulk data transfer –Fair –Friendly to TCP Non-goals –TCP replacement –Messaging service

9. Design Issues Reliability/Acknowledging Congestion/Flow Control Performance evaluation –Efficiency –Fairness and friendliness –Stability

10. Reliability/Acknowledging Acknowledging is expensive –Packet processing at end hosts and routers –Buffer processing Timer-based selective acknowledgement –Send acknowledgement per constant time (if there are packets to be acknowledged) Explicit negative acknowledgement

11. Congestion Control AIMD with decreasing increases Increase formula Decrease –1/9 Control interval is constant –SYN = 0.01 second

12. UDT Algorithm C (Mbps)L - C (Mbps)Increment (pkts/SYN) [0, 9000)(1000, 10000]10 [9000, 9900)(100, 1000]1 [9900, 9990)(10, 100]0.1 [9990, 9999)(1, 10]0.01 [9999, 9999.9)(0.1, 1]0.001 9999.9+<0.10.00067 L = 10 Gbps, S = 1500 bytes

13. UDT: Efficiency and Fairness Characteristics Takes 7.5 seconds to reach 90% of the link capacity, independent of BDP Satisfies max-min fairness if all the flows have the same end-to-end link capacity –Otherwise, any flow will obtain at least half of its fair share Does not take more bandwidth than concurrent TCP flow as long as

14. Efficiency UDT bandwidth utilization –960Mb/s on 1Gb/s –580Mb/s on OC-12 (622Mb/s)

15. Fairness Fair bandwidth sharing between networks with different RTTs and bottleneck capacities –330 Mb/s each for the 3 flows from Chicago to Chicago Local via 1Gb/s, Amsterdam via 1Gb/s and Ottawa via 622Mb/s

16. Fairness Fairness index –Simulation: Jain’s Fairness Index for 10 UDT and TCP flows over 100Mb/s link with different RTTs

17. RTT Fairness Fairness index of TCP flows with different RTTs –2 flows, one has 1ms RTT, the other varies from 1ms to 1000ms

18. Fairness and Friendliness 50 TCP flows and 4 UDT flows between SARA and StarLight Realtime snapshot of the throughput The 4 UDT flows have similar performance and leave enough space for TCP flows

19. TCP Friendliness Impact on short life TCP flows –500 1MB TCP flows with 1-10 bulk UDT flows, over 1Gb/s link between Chicago and Amsterdam

20. Stability Stability index of UDT and TCP –Stability: average standard deviation of throughout per unit time –10 UDT flows and 10 TCP flows with different RTTs

21. Implementations Issues Efficiency and CPU utilization Loss information processing Memory management API Conformance

22. Efficiency and CPU utilization Efficiency = Mbps/MHz Maximize throughput –Use CPU time as little as possible, so that CPU won’t be used up before network bottleneck is reached –Remove CPU burst, which can cause packet loss: even distribution of processing Minimize CPU utilization

23. Loss Processing On high BDP networks, the number of lost packets can be very large during a loss event Access to the loss information may take long time Acknowledge may take several packets

24. Loss Processing UDT loss processing –Most loss are continuous –Record loss event other than lost packets –Access time is almost constant

25. Memory Processing Memory copy avoidance Overlapped IO Data scattering/gathering Speculation of next packet Protocol Buffer User Buffer Data New Data

26. API Socket-like API Support overlapped IO File transfer API –sendfile/recvfile Thread safe Performance monitoring

27. API - Example UDTSOCKET client = UDT::socket(AF_INET, SOCK_STREAM, 0); UDT::connect(client, (sockaddr*)&serv_addr, sizeof(serv_addr)); If (UDTERROR == UDT::send(client, data, size, 0)) { //error processing } int client = socket(AF_INET, SOCK_STREAM, 0); connect(client, (sockaddr*)&serv_addr, sizeof(serv_addr)); If (-1 == send(client, data, size, 0)) { //error processing }

28. Implementation Efficiency CPU usage of UDT and TCP –UDT takes about 10% more CPU than TCP –More code optimizations are still on going

29. Conclusion TCP is not suitable for distributed data intensive applications over grid networks We introduced a new application level protocol named UDT, to overcome the shortcomings of TCP We explained the design rationale and implementations details in this paper

30. Future Work Bandwidth Estimation CPU utilization –Self-clocking –Code optimization Theoretical work

31. References More details can be found in our paper. UDT specification –Draft-gg-udt-01.txt Congestion control –Paper on Gridnets '04 workshop UDT open source project –http://udt.sf.net

32. Thank you! Questions and comments are welcome! For more information, please visit Booth 653 (UIC/NCDM) at Exhibition Floor UDT Project: http://udt.sf.net NCDM: http://www.ncdm.uic.edu