1. Congestion Control on High-Speed Networks
Injong Rhee, Lisong Xu
Computer Science Department
North Carolina State University

2. Outline
Motivation
Related Work
BIC-TCP
Future Work

3. High-Speed Networks Many high-speed networks are being developed.NL
SURFnet
GENEVA UK
SuperJANET4
ABILENE
ESNET
CALREN It
GARR-B
GEANT
NewYork Fr
Renater
STAR-TAP
STARLIGHT
Many high-speed networks are being developed.
BW: 10Gbps, and expected to be upgraded to higher speeds in the near future.

4. ESNET: Energy Science NetworkNL
SURFnet
GENEVA UK
SuperJANET4
ABILENE
ESNET
CALREN It
GARR-B
GEANT
NewYork Fr
Renater
STAR-TAP
STARLIGHT
ESNET
ESNet is funded by the Department of Energy to provide a high-speed network serving thousands of scientists worldwide

5. High-Speed Applications
Weather Simulation Video Conference Telemedicine
Transport Protocols
be able to transfer a large amount of data
over a long distance within a short amount of time
A Doctor performs operations on a patient located in another hospital - even in another country. A large amount of video data has to be quickly transferred from the patient site to the Doctor site.
High-Speed Networks

6. Cannot fully utilize the huge capacity of high-speed networks!
TCP Performance
Utilization of a link with 5 TCP connections
Cannot fully utilize the huge capacity of high-speed networks!
NS-2 Simulation (100 sec)
Link Capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps,
Drop-Tail Routers, 0.1BDP Buffer
5 TCP Connections, 100ms RTT, 1000-Byte Packet Size
Ns-2: capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps
100 sources, 100 ms round trip propagation delay, 1500-Byte Packet Size

7. TCP Congestion Control
The instantaneous throughput of TCP is controlled by a variable cwnd,
TCP transmits approximately a cwnd number of packets per RTT (Round-Trip Time).
Time (RTT)
Slow start
Congestion avoidance
Packet loss
cwnd
TCP
cwnd = cwnd + 1
cwnd = cwnd * (1-1/2)
First, TCP increases the congestion window, cwnd, exponentially to quickly find the available bandwidth. This exponential increase is known as “slow start”. After the first packet loss, TCP decreases cwnd to the half point, and then switches from slow start to congestion avoidance. In congestion avoidance, TCP increases cwnd linearly with an increment of 1 packet per RTT until the next packet loss. After the second packet loss, TCP decreases cwnd to the half point again, and then increase cwnd linearly again until the next packet loss. TCP repeats this process again and again until the end of the connection.

8. TCP over High-Speed Networks
A TCP connection with 1250-Byte packet size and 100ms RTT is running over a 10Gbps link (assuming no other connections, and no buffers at routers)
slow
increase
1.4 hours
Packet loss
Packet loss
Packet loss
Packet loss
TCP
cwnd
big
decrease
100,000
10Gbps
Now let’s look at an example of TCP over a high-speed network. We have the same parameters as the previous one, except that the link capacity is 10Gbps.
This time, TCP has packet losses at 10Gbps, which corresponds to a cwnd of After a packet loss, TCP decreases its transmission rate to 5Gbps, which corresponds to a cwnd of Since RTT is 100ms, it takes about 1.7hours for TCP to increase its transmission rate from 5Gbps to 10Gbps.
We can see that, if a TCP connection is a short connection with a short duration, then TCP may finish the transmission, even before it reaches to 10Gbps. In that case, we will have a low link utilization.
50,000
5Gbps
Slow start
Congestion avoidance
Time (RTT)

9. Response Function of TCP
Response function of TCP is the average throughput of a TCP connection in terms of the packet loss probability, the packet size, and the round-trip time.
R : Average Throughput
MSS: Packet Size
RTT: Round-Trip Time
P : Packet Loss Probability
Response Function of TCP is :
J. Padhye, V. Firoio, D. Towsley, J. Kurose, "Modeling TCP Throughput: a Simple Model and its Empirical Validation", Proceedings of SIGCOMM 98
Usually, we study the performance of TCP using its response function, which is defined as the average throughput of TCP in terms of packet loss probability.
Here is the response function of TCP. In this function, MSS is the packet size, RTT is the round trip time, and p is the packet loss probability. From this function, we can see that the throughput of TCP depends on the packet size, the round-trip time, and the packet loss probability. In order to increase the throughput of TCP, we can either increase the packet size, or decrease the round trip time, or decrease the packet loss probability. Usually for a TCP connection, the packet size and round-trip time are fixed. Therefore, the throughput of a TCP connection mainly depends on the packet loss probability.

10. Response Function of TCP
10Gbps requires a packet loss rate of 10-10, or correspondingly a link bit error rate of at most 10-14, which is an unrealistic (or at least hard) requirement for current networks
In this figure, we show how the throughput changes as we vary the packet loss probability. Note that the figure is shown in log-log scale. We can see that as the packet loss probability decreases, the TCP throughput increases. Assume the packet size is 1000 Bytes, and the RTT is 100ms, then TCP achieves the 10Gbps throughput when the packet loss probability is around which corresponds to a link BER of which is an unrealistic requirement for current networks. That is, in the current networks, it is impossible for TCP to achieve 10Gbps throughput.
Assuming 1250-Byte packet size, and 100ms RTT

11. Outline Motivation Related Work BIC-TCP Future Work
Since TCP has really a poor performance over high-speed networks, recently some new congestion control protocols have been proposed to achieve high performance over high-speed networks. Next, I am going to describe some of them.

12. Proposed High-Speed Protocols
Window-Based Protocols
AIMD (Additive Increase Multiplicative Decrease)
Jumbo Frame, GridFTP, PFTP, PSockets
HSTCP (High-Speed TCP) by Sally Floyd at ICIR, Berkeley
STCP (Scalable TCP) by Tom Kelly at Cambridge University
FAST (Fast AQM Scalable TCP) by Steven Low at California Institute of Technology
Rate-Based Protocols
SABUL (Simple Available Bandwidth Utilization Library ) by Robert Grossman at University of Illinois at Chicago
window-based protocols are known for safer incremental deployment.
D. Bansal, H. Balakrishnan, S. Floyd, and S. Shenker, "Dynamic behavior of slowly responsive congestion controls", In Proceedings of SIGCOMM 2001, San Diego, California.

13. AIMD (Additive Increase Multiplicative Decrease)
AIMD increases cwnd by a larger number, say 32, instead of 1 per RTT.
After a packet loss, AIMD decreases cwnd by 1/8, instead of 1/2
cwnd = cwnd + 1
cwnd = cwnd + 32
cwnd = cwnd * (1-1/2)
cwnd = cwnd * (1-1/8)
Packet loss
Packet loss
Packet loss
Packet loss
TCP
cwnd
Slow start
Congestion avoidance
Time (RTT)

14. Response Function of AIMD
TCP:
AIMD:
The throughput of AIMD is always about 13 times larger than that of TCP
Now let’s look at the response function of the AIMD protocol. The response function of AIMD is very similar to that of the TCP. The only difference is the constant. The constant of TCP is 1.2, and the constant of AIMD is Therefore, the throughput of AIMD is always about 13 times larger than the throughput of TCP.

15. Properties of AIMD Bandwidth Scalability TCP-Friendliness
Bandwidth Scalable
Bandwidth Scalability
The ability to achieve 10Gbps with a reasonable packet loss probability
NOT TCP Friendly
TCP-Friendliness
The ability to share bandwidth with TCP connections on low-speed networks
Recall that TCP achieves 10Gbps throughput when the packet loss probability is 10-10, here, AIMD achieves the same throughput when the packet loss probability is about 10-7, which is a more realistic requirement for the current networks. Therefore, we say that AIMD is a scalable protocol.
However, as we described before, AIMD always has about 13 times larger throughput than TCP, even in low-speed networks. That is, AIMD can not share the bandwidth with TCP in low-speed networks, or AIMD is not a TCP-friendly protocol. This is not a desired property.

16. STCP (Scalable TCP)
STCP adaptively increases cwnd, and decreases cwnd by 1/8.
cwnd = cwnd + 1
cwnd = cwnd *cwnd
cwnd = cwnd * (1-1/2)
cwnd = cwnd * (1-1/8)
Packet loss
Packet loss
Packet loss
Packet loss
TCP
cwnd
Slow start
Congestion avoidance
Time (RTT)

17. cwnd = cwnd * (1-dec(cwnd))
HSTCP (High Speed TCP)
HSTCP adaptively increases cwnd, and adaptively decreases cwnd.
The larger the cwnd, the larger the increment, and the smaller the decrement.
cwnd = cwnd * (1-1/2)
cwnd = cwnd * (1-dec(cwnd))
cwnd = cwnd + 1
cwnd = cwnd + inc(cwnd)
Packet loss
Packet loss
Packet loss
Packet loss
TCP
cwnd
Slow start
Congestion avoidance
Time (RTT)

18. Response Functions of HSTCP and STCP
Bandwidth Scalable
HSTCP:
STCP:
HSTCP and STCP are both bandwidth scalable and TCP friendly
Now let’s look at the response functions of STCP and HSTCP. The blue line is the response function of HSTCP, and the red line is the response function of STCP. We can see that both of them are scalable protocols, because HSTCP can achieve 10Gbps throughput with a packet loss probability 10-7, that is same as AIMD. STCP can achieve 10Gbps throughput with a packet loss probability of 10-6.
We can also see that both HSTCP and STCP are TCP-Friendly protocols. Because in low-speed networks, they have similar or less throughput than TCP.
Now, it seems that HSTCP and STCP are perfect protocols for high-speed networks, since they are both scalable and tcp-friendly protocols. However, we have found a problem of both of them.
TCP Friendly

19. Outline Motivation Related Work BIC-TCP Future Work Why another one?
Protocol Design
Parameter Setting
Simulations & Experiments
Future Work
Our objective is to design a new protocol, which is scalable, TCP-friendly, and has a good RTT fairness in high-speed networks. This new protocol is our BIC-TCP.

20. RTT Fairness
Different connections may have quite different round-trip times, and a good protocol should allocate bandwidth fairly among those connections
RTT fairness index = throughout ratio of two flows with different RTTs

21. RTT Fairness on Low-Speed Networks
For a protocol with the following response function, where c and d are protocol-related constants.
The RTT Fairness Index (or the throughput ratio of two flows) on low-speed networks is
We analyzed the RTT fairness using the response function of a protocol. We consider a protocol with the following response function, where c and d are protocol-related parameters. We consider two extreme cases. First we assume that the packet losses of those two flows are independent of each other, that is, two flows are completely asynchronized. As a low-speed network has less synchronization, we use this case to gain insight into the RTT fairness in low-speed networks. Second we assume that two flows always have packet losses at the same time, that is, two flows are completely synchronized. As a high-speed network has more synchronization, we use the case to gain insight into the RTT fairness in high-speed networks.
First, if two flows are completely asynchronized, then the RTT fairness index is just their inverse RTT ratio., which is independent of the protocol. Second, if two flows are completely synchronized, then here is the RTT fairness, which depends on the parameter d of a protocol.
We can see from two expressions, that RTT fairness becomes worse as the network capacity increases. In low-speed networks, RTT fairness is fixed, and independent of the protocol, while in high-speed networks, RTT fairness depends on the d paramter of the protocol. The larger d, the worse the RTT fairness.
On low speed networks, different protocols have the same RTT fairness
Lisong Xu, Khaled Harfoush, and Injong Rhee, "Binary Increase Congestion Control for Fast Long-Distance Networks", in Proceedings of IEEE INFOCOM 2004, March, 2004, HongKong

22. RTT Fairness on High-Speed Networks
For a protocol with the following response function, where c and d are protocol-related constants.
The RTT Fairness Index (or the throughput ratio of two flows) on high-speed networks is
We analyzed the RTT fairness using the response function of a protocol. We consider a protocol with the following response function, where c and d are protocol-related parameters. We consider two extreme cases. First we assume that the packet losses of those two flows are independent of each other, that is, two flows are completely asynchronized. As a low-speed network has less synchronization, we use this case to gain insight into the RTT fairness in low-speed networks. Second we assume that two flows always have packet losses at the same time, that is, two flows are completely synchronized. As a high-speed network has more synchronization, we use the case to gain insight into the RTT fairness in high-speed networks.
First, if two flows are completely asynchronized, then the RTT fairness index is just their inverse RTT ratio., which is independent of the protocol. Second, if two flows are completely synchronized, then here is the RTT fairness, which depends on the parameter d of a protocol.
We can see from two expressions, that RTT fairness becomes worse as the network capacity increases. In low-speed networks, RTT fairness is fixed, and independent of the protocol, while in high-speed networks, RTT fairness depends on the d paramter of the protocol. The larger d, the worse the RTT fairness.
On high speed networks, the RTT fairness of a protocol depends on the exponent d in the response function
Lisong Xu, Khaled Harfoush, and Injong Rhee, "Binary Increase Congestion Control for Fast Long-Distance Networks", in Proceedings of IEEE INFOCOM 2004, March, 2004, HongKong

23. Slope Determines the RTT Fairness
If the response function is: then the RTT Fairness is:
The figure is shown in log-log scale, so exponent d in the response function is the slope of the line in the figure
The slope of the line determines the RTT fairness of a protocol on high-speed networks
Since RTT Fairness becomes worse in high-speed networks, let’s look at the RTT fairness of those protocols in high-speed networks. As we described before, for a protocol with this form of response function, we found that its RTT fairness index can be calculated as a function of the RTT ratio, and the exponent is 1 over 1-d. Now let’s look at the value of d for different protocols. The d of TCP is 0.5, the d of AIMD is also 0.5, the d of HSTCP is 0.835, and the d of STCP is 1. Based on our analysis, TCP and AIMD have the same RTT fairness, and their RTT fairness is best among all the protocols. While STCP has the worst RTT fairness, since the d of STCP is 1, so 1 over 1-d is infinite. That means, the flow with short RTT will grab all the bandwidth, and the flow with long RTT gets nothing. That is just we saw in the previous simulation.
Now let’s look at the figure of the response functions. Since this figure is shown in log-log scale, so the exponent d in a response function is just the slope of the response function in the figure. That is, the slope of a response function determines the RTT fairness of a protocol. For example, AIMD has the same slope as TCP, so AIMD has the same RTT fairness as TCP. While STCP has the largest slope, so STCP has the worst RTT fairness.
Overall , AIMD has a good scalability, and a RTT fairness same as TCP, but AMD is not TCP-friendly. HSTCP and STCP are scalable and TCP-friendly, but they have a RTT fairness, which is much worse than TCP.

24. Simulation Results of RTT Fairness
Throughout ratio of two flows on a 2.5Gbps Link
Inverse RTT Ratio 1 3 6
AIMD 22
HSTCP 29
107
STCP
127
389
So we define RTT fairness index as the throughput ratio of two flows with different RTT ratio. In this table, we show the RTT fairness index of two flows in a network with 2.5Gbps link capacity. W increase the RTT ratio from 1 to 6. For instance, if two flows have the same RTT, then their throughput ratio is 1, for all protocols. If the RTT ratio is 6, that means, the RTT of a flow is 6 times larger than the RTT of another flow, then their throughout ratio is 22 for AIMD, 107 for HSTCP, and 389 for STCP.
We can see that the RTT fairness, or throughput ratio increases as the RTT ratio increases. AIMD has the best RTT fairness, and STCP has the worst RTT fairness.
Now let’s look at another group of simulation results. This time, we measure the RTT fairness of two flows in a network with 100Mbps link capacity. We can see that the RTT fairness is much better in the low-speed network than in the high-speed network. For example, with 2.5Gbps, the RTT fairness index of STCP is about 400, while with 100Mbps, the RTT fairness index of STCP is about 60. That means, as the network capacity increases, the RTT fairness become worse.
Simulation setup: BDP Buffer, Drop Tail, Reverse Traffic, Forward Background Traffic (short-lived TCP, Web Traffic)

25. Simulation Result of 2 STCP Flows
Throughput of two STCP flows with different RTTs on a 2.5Gbps link
In this slide, I will show you a simulation result of ten STCP flows with different RTTs. Two of them have a longer RTT, and the rest of them have a shorter RTT. We observe that the throughput of the two long-RTT flows becomes smaller and smaller. Finally, the 8 short-RTT flows grab almost all the bandwidth, while the other two long-RTT flows has almost zero bandwidth. So there is a very serious fairness problem between flows with different RTTs.
Simulation setup: 2.5Gbps, BDP Buffer, Drop Tail, Reverse Traffic, Forward Background Traffic (short-lived TCP, Web Traffic)

26. Scalability, RTT Fairness
Design Goal
Scalability, RTT Fairness
TCP Fairness

27. Outline Motivation Related Work BIC-TCP Future Work Why another one?
Protocol Design
Parameter Setting
Simulations & Experiments
Future Work
Our objective is to design a new protocol, which is scalable, TCP-friendly, and has a good RTT fairness in high-speed networks. This new protocol is our BIC-TCP.

28. BIC (Binary Increase Congestion control)
BIC adaptively increase cwnd, and decrease cwnd by 1/8
cwnd = cwnd + 1
cwnd = cwnd + f(cwnd, history)
cwnd = cwnd * (1-1/2)
cwnd = cwnd * (1-1/8)
Packet loss
Packet loss
Packet loss
Packet loss
TCP
cwnd
Similar to other high-speed protocols, BIC also modifies the increase and decrease part of congestion avoidance of TCP. In Bic, the increase depends not only the current window size cwnd, but also depends on some history information. The decrease part of BIC is the same as AIMD and STCP. After a packet loss, cwnd is decreased by 1/8.
Now let’s look at the increase part of BIC.
Slow start
Congestion avoidance
Time (RTT)

29. A Search Problem A Search Problem
We consider the increase part of congestion avoidance as a search problem, in which a connection looks for the available bandwidth by comparing its current throughput with the available bandwidth, and adjusting cwnd accordingly.
Q: How to compare R with A?
R = current throughput
= cwnd/RTT
A = available bandwidth
A: Check for packet losses
No packet loss: R <= A
Packet losses : R > A
How does TCP find the available bandwidth?
Linear search
while (no packet loss){
cwnd++; }

30. Linear Search
Available Bandwidth

31. BIC: Binary Search with Smax and Smin
while (Wmin <= Wmax){
inc = (Wmin+Wmax)/2 - cwnd;
if (inc > Smax)
inc = Smax;
else if (inc < Smin)
inc = Smin;
cwnd = cwnd + inc;
if (no packet losses)
Wmin = cwnd;
else
break; }
Wmax: Max Window
Wmin: Min Window
Smax: Max Increment
Smin: Min Increment

32. Binary Search with Smax and Smin
Available Bandwidth
Wmax
Smin
Smax
Wmin

33. Binary Increase Congestion Control (BIC)
Binary search increase
Additive increase

34. Outline Motivation Related Work BIC-TCP Future Work Why another one?
Protocol Design
Parameter Setting
Simulations & Experiments
Future Work

35. Setting Smax Response Function of BIC on high-speed networks
Bandwidth scalability
Bandwidth scalability of BIC depends only on Smax
RTT Fairness of BIC on high-speed networks is the same as that of AIMD

36. Setting Smin Response Function of BIC on low-speed networks
TCP-friendliness of BIC depends only on Smin
TCP friendliness

37. Bandwidth scalability
Response Functions
Bandwidth scalability
RTT Fairness
TCP-Friendliness
Finally, here is the response function of BIC. Which has a good scalability, a good TCP-friendliness, and a good RTT Fairness in high-speed networks.

38. Outline Motivation Related Work BIC-TCP Future Work Why another one?
Protocol Design
Parameter Setup
Simulations & Experiments
Future Work

39. Bandwidth Scalability
First, we measure the total throughput of 10 connections as we increase the network capacity from 155Mbps to 10Gbps. We already saw the result for TCP, which has a really poor performance on 10Gbps networks. Here are results of those high-speed protocols. All of them have good performance on all networks. That is consistent with our previous analysis based on response function.
NS-2 Simulation (100 sec)
Link Capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps,
Drop-Tail Routers, 0.1BDP Buffer
5 Connections, 100ms RTT, 1000-Byte Packet Size
Ns-2: capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps
100 sources, 100 ms round trip propagation delay, 1500-Byte Packet Size

40. TCP - Friendliness
This figure compares the TCP-friendliness of different protocols. In the simulation, there are 2 long-lived TCP connections, 2 long-lived high-speed connections, and some background web traffic. For example, the blue is the throughput of the long-lived TCP connections, and the purple is the throughput of the long-lived high-speed connection.
We can see that, regular TCP has much smaller throughput with AIMD than with other three protocols. With AIMD, the throughput of TCP is about 2% of total link capacity, while the throughput of TCP is about 10% with other three protocols.
Simulation setup: 20Mbps, BDP Buffer, Drop Tail, Reverse Traffic
2 TCP flows, 2 high-speed flows, and some background traffic

41. RTT Fairness
Throughput ratio of two flows with different RTTs on a 2.5Gbps link
Inverse RTT Ratio 1 3 6
BIC 12 38
AIMD 22
HSTCP 29
107
STCP
127
389
We already saw the RTT fairness of the other three protocols on 100Mbps networks and 2.5Gbps networks. Here is the RTT Fairness of BIC. We can see that the RTT fairness of BIC is similar to the RTT fairness of AIMD, and much better than that of HSTCP and STCP.
Simulation setup: BDP Buffer, Drop Tail, Reverse Traffic, Forward Background Traffic (short-lived TCP, Web Traffic)

42. Buffer Size and Throughput

43. Queue Size Vs. Buffer Size

44. Packet loss vs. Buffer Size

45.   Simulation Summary AIMD HSTCP STCP BIC Scalability
TCP-Friendliness 
RTT Fairness
Finally, here is the summary of the simulation results. BIC performs well in all simulations. AIMD has the worst TCP-frienliness, STCP has the worst RTT fairness, and worst convergence speed. The RT fairness and convergence speed of HSTCP are OK, but not good.

46. SLAC Experiments
Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks by SLAC (Stanford Linear Accelerator Center)
Linux TCP
Parallel TCP
HighSpeed TCP
Scalable TCP
HighSpeed TCP Low Priority
Hamilton TCP
BIC TCP
“BIC TCP overall performs very well in our tests”
Those high-speed protocols have been implemented as a patch for Linux Kernel, and have been tested by SLAC on real networks. They tested the scalability and tcp-friendliness, and convergence speed of different protocols. Their conclusion is

47. Outline
Motivation
Related Work
BIC-TCP
Future Work

48. Conclusion and Future Work
Designed a congestion control protocol, which is scalable, TCP-friendly, and has a good RTT fairness.
"Binary Increase Congestion Control for Fast Long-Distance Networks", in Proceedings of IEEE INFOCOM 2004, March, 2004, HongKong
Our work has been reported by some newspapers.
NBC (US)
The Washington Times (US)
The Times of India (India)
Sina (China)
Future Work
Further improve the performance of BIC
Solve other problems in high-speed networks

49. CUBIC

50. More information: Thank you! Any questions?
Thank you!
Any questions?