1. UDT: UDP based Data Transfer Yunhong Gu & Robert Grossman Laboratory for Advanced Computing University of Illinois at Chicago

2. 02/17/2004 PFLDnet 2004 2 Outline Background UDT Protocol UDT Congestion Control Implementation/Simulation Results Summary

3. 02/17/2004 PFLDnet 2004 3 Background Distributed data intensive applications over wide area optical networks:  Grid computing, access of bulk scientific data, data mining, high resolution video, etc. Transport protocol support:  Efficient and fair bandwidth unitization TCP does not work!

4. 02/17/2004 PFLDnet 2004 4 Trans-Atlantic TCP Performance Chicago -> Amsterdam, 1Gbps link capacity, 110ms RTT  TCP: 5Mbps @ default setting (64KB buffer)  TCP: 100Mbps @ 12MB buffer (=1Gbps*110ms)  Parallel TCP: 800Mbps @ 64 TCP concurrent flows, with each having 1MB buffer Two concurrent TCP flows, 1 from Chicago to Amsterdam, 1 within Chicago local networks:  2Mps vs. 940Mbps!

5. 02/17/2004 PFLDnet 2004 5 Why TCP Fails Discover/recover slow on high BDP links  Increase 1 byte per RTT Drastic decrease in sending rate Fairness bias on longer RTT links More prone to link error in high BDP links B: throughout in packets per second, p: loss rate

6. 02/17/2004 PFLDnet 2004 6 Requirements to the New Protocol FAST  High utilization of the abundant bandwidth either with single or multiplexed connections FAIR  Intra-protocol fairness, independent of RTT FRIENDLY  TCP compatibility

7. 02/17/2004 PFLDnet 2004 7 Use Scenarios Small number of sources shares abundant bandwidth Bulk data transfer  Most of the packets can be packed in maximum segment size (MSS) in a UDT session  MSS can be set up by applications and the optimal value is the path MTU

8. 02/17/2004 PFLDnet 2004 8 UDT: UDP based Data Transfer  Reliable, application level, duplex, transport protocol, over UDP with congestion control  Implementation: Open source C++ library Two orthogonal parts  The UDT protocol framework that can be implemented above UDP, with any suitable congestion control algorithms  The UDT congestion control algorithm, which can be implemented in any transport protocols such as TCP What’s UDT?

9. 02/17/2004 PFLDnet 2004 9 Packet Structure Data Packet:  Header: 1bit flag + 31bit sequence number Control Packet:  Header: 1bit flag + 3bit type + 12bit reserved + 16bit ACK seq. no. + (0 - 32n)bit control info  Type: ACK, ACK2, NAK, Handshake, Keep-alive, and Shutdown Actual size of a UDT packet can be ascertained from UDP header

10. 02/17/2004 PFLDnet 2004 10 Data Packet 0Packet Sequence Number User Data Payload  Flag Bit: 0  UDT uses 31-bit packet based sequence number, ranging from 0 and (2 31 - 1)  Sequence number may be wrapped if it exceeds the maximum available number

11. 02/17/2004 PFLDnet 2004 11 Control Packet 1typereservedACK Seq. No. Control Information Field  Flag Bit: 1  type: 3-bit  handshake (000), shutdown (101), keep-alive (001)  ACK (010), ACK2 (110), NAK (011)  UDT uses sub-sequencing: each ACK and related ACK2 are assigned a 16-bit unique ACK sequence number

12. 02/17/2004 PFLDnet 2004 12 Acknowledgements Selective acknowledgement (ACK)  Generated at every constant interval to send back largest continuously received sequence number of data packets.  The sender sends back an ACK2 to the receiver for each ACK (sub-sequencing).  Also carries RTT, packet arrival speed, and estimated link capacity. Explicit negative acknowledgement (NAK)  Generated as soon as loss is detected.  Loss information may be resent if receiver has not received the retransmission after an increasing interval.  Loss information is compressed in NAK.

13. 02/17/2004 PFLDnet 2004 13 Timing Packet Scheduling Timer  Tuned by Rate Control  High precision in CPU clock cycles Rate Control Timer: trigger rate control  RCTP = 0.01 seconds ACK Timer: trigger acknowledgement  ATP = RCTP

14. 02/17/2004 PFLDnet 2004 14 Timing (cont.) NAK Timer: trigger negative acknowledgement  NTP = RTT Retransmission Timer: trigger retransmission based on time-out and maintain connection status  RTP = (exp-count + 1) * RTT + ATP where exp-count is the number of continuous time-out

15. 02/17/2004 PFLDnet 2004 15 UDT Architecture DATA ACK ACK2 NAK Sender Recver Sender Recver  Pkt. Scheduling Timer  ACK Timer  NAK Timer  Retransmission Timer  Rate Control Timer Sender

16. 02/17/2004 PFLDnet 2004 16 Congestion Control Rate based congestion control (Rate Control)  RC tunes the packet sending period.  RC is triggered periodically at the sender side.  RC period is constant of 0.01 seconds. Window based flow control (Flow Control)  FC limits the number of unacknowledged packets.  FC is triggered on each received ACK at the sender side.

17. 02/17/2004 PFLDnet 2004 17 Rate Control AIMD: Increase parameter is related to link capacity and current sending rate; Decrease factor is 1/9, but not decrease for all loss events. Link capacity is probed by packet pair, which is sampled UDT data packets.  Every 16th data packet and it successor packet are sent back to back to form a packet pair.  The receiver uses a median filter on the interval between the arrival times of each packet pair to estimate link capacity. ……

18. 02/17/2004 PFLDnet 2004 18 Rate Control (cont.) 1. If loss rate is greater than 1%, do not increase; 2. Number of packets to be increased in next RCTP time is: where B is estimated link capacity, C is current sending rate. Both are in packets or packets per second. MSS is the packet size in bytes. β = 1.5 * 10 -6. 3. Recalculate packet sending period (STP).

19. 02/17/2004 PFLDnet 2004 19 Rate Control (cont.) C (Mbps)B - C (Mbps)Increase Param. (Pkts) [0, 9000)(1000, 10000]10 [9000, 9900)(100, 1000]1 [9900, 9990)(10, 100]0.1 [9990, 9999)(1, 10]0.01 [9999, 9999.9)(0.1, 1]0.001 9999.9+<0.10.00067 B = 10Gbps, MSS = 1500 bytes

20. 02/17/2004 PFLDnet 2004 20 Rate Control (cont.) Decrease sending rate by 1/9, (or equivalently, increase packet sending period by 1.125), only if 1. Received an NAK, whose last lost sequence number is greater than the largest sequence number when last decrease occurred; or 2. The number of loss events since last decrease has exceeded a threshold, which increases exponentially and is reset when condition 1 is satisfied. No data will be sent out for the next RCTP time if a decrease occurs.  Help to clear congestion.

21. 02/17/2004 PFLDnet 2004 21 BDP W = W*0.875 + AS*(RTT+ATP)*0.125 AS is the packets arrival speed at receiver side.  The receiver records the packet arrival intervals. AS is calculated from the average of latest 16 intervals after a median filter.  It is carried back within ACK. Flow Control

22. 02/17/2004 PFLDnet 2004 22 Slow Start Flow window starts at 2 and increases to the number of acknowledged packets, until the sender receives an NAK or reaches the maximum window size, when slow start ends. Packet sending period is 0 during slow start phase and set to the packet arrival interval at the end of the phase. Slow start only occurs at the beginning of a UDT session.

23. 02/17/2004 PFLDnet 2004 23 Implementation: Performance

24. 02/17/2004 PFLDnet 2004 24 Implementation: Intra-protocol Fairness

25. 02/17/2004 PFLDnet 2004 25 Implementation: TCP Friendliness

26. 02/17/2004 PFLDnet 2004 26 Implementation: TCP Friendliness (cont.)

27. 02/17/2004 PFLDnet 2004 27 Implementation: File Transfer ToStarLightCanarieSARA From StarLight460505560 Canarie440502- SARA441-660 CanarieStarLightSARA 1Gbps/15.9ms1Gbps/110ms Disk R: 800Mbps W: 550Mbps Disk R: 800Mbps W: 500Mbps Disk R: 1300Mbps W: 900Mbps

28. 02/17/2004 PFLDnet 2004 28 Simulation: UDT Throughput at Different Bandwidth and RTT

29. 02/17/2004 PFLDnet 2004 29 Simulation: Performance of Concurrent UDT Flows

30. 02/17/2004 PFLDnet 2004 30 Simulation: Intra-protocol Fairness

31. 02/17/2004 PFLDnet 2004 31 Simulation: RTT Independence

32. 02/17/2004 PFLDnet 2004 32 Simulation: TCP Friendliness

33. 02/17/2004 PFLDnet 2004 33 Simulation: Convergence/Stability

34. 02/17/2004 PFLDnet 2004 34 Simulation: Complex Scenario 100  5010    Flow ID123456 Throughput (Mbps) 89.390.05.1841.750.84.78 Link capacity Mbps Flow and its ID Node DropTail

35. 02/17/2004 PFLDnet 2004 35 Simulation: Multi-bottleneck A x 200 B C X0.1110204060 AB198.8189.2180.1170.9152.5137.6 AC0.0980.9799.95519.8839.4657.70 X80100120140160180 AB108.4104.6100.8101.3100.7100.3 AC73.4992.4298.4798.0498.6599.00

36. 02/17/2004 PFLDnet 2004 36 Summary UDT Protocol  Application level upon UDP  Selective acknowledgement / explicit negative acknowledgement UDT Congestion Control  Rate Control Bandwidth estimation for fast probing available bandwidth and fast recovery AIMD for fairness Constant rate control interval  Flow Control Dynamic flow window according to packet receiving speed

37. 02/17/2004 PFLDnet 2004 37 UDT Characters Good use of available bandwidth Application level - no changes in router and operating system No manual tuning Fair and Friendly: intra-protocol fairness, TCP friendliness, and RTT independence. Open source

38. Thank You! LAC: www.lac.uic.eduwww.lac.uic.edu UDT: sourceforge.net/projects/dataspacesourceforge.net/projects/dataspace Internet Draft: draft-gg-udt-01.txt