Restless bandits and congestion control Mark Handley, Costin Raiciu, Damon Wischik UCL

* * * “Resource pricing and the evolution of congestion control”, Gibbens and Kelly, 1999.

TCP has changed recently, and changes are ongoing. What sort of mathematical modeling has influenced the changes, and what sort of modeling is still needed? Compound TCP in Windows Vista (2007) CUBIC TCP in Linux (2006) Ongoing IETF working group “ledbat”, standardizing BitTorrent’s congestion control Ongoing IETF working group “mptcp”, standardizing multipath TCP

At UCL we have been developing a multipath version of TCP. I will describe some issues that have arisen. TCP decides when to send out data packets. It infers the level of network congestion by the acknowledgements it receives, and it adjusts its sending rate accordingly. What should multipath TCP do? On a congested path, packets will be dropped and acknowledgements will be missing. TCP sends data packets over several paths simultaneously, and the destination sends back acknowledgements.

Multipath congestion control theory has been developed by Kelly and Voice (2005), and by Han, Shakkottai, Hollot, Srikant, Towsley (2006). Interpretation – Increase x r by a constant, every time you get an acknowledgement on path r – Decrease x r by an amount proportional to y s(r) if you detect a drop on path r There is a large body of work on fluid models of congestion control:write down a network utility maximization problem,write down a system of differential equations,show that the (unique) fixed point solves the utility maximization,and interpret it as a discrete congestion control algorithm.

How we expect the fluid model to behave

We simulated it. It didn’t work how we expected. When there are many flows, then each flow will flip independently, and the aggregate will behave how the fluid models predict.

The naïve multipath congestion controller is bistable when the paths are equally congested. Is this a problem? Load-balancing traffic will cause there to be resource pooling in the Internet, hence equal congestion will be the common case. If congestion levels were static, it wouldn’t matter if you flapped from one path to another. But congestion is not static. (That is why TCP has to continuously test for more capacity.) Hence users should be in equipoise, i.e. balance traffic across all their good paths, so that they learn quickly about any changes in congestion.

“Indexability of restless bandit problems and optimality of Whittle’s index for dynamic multichannel access”, Liu and Zhao, arXiv 2008

This is the Bellman equation for a long- term average-cost dynamic programming problem. Note: this equation is for single-path congestion control, not multipath. ! Control: at what rate the user should send packets State: the user’s current belief about the network Plant: Bayesian update of user’s beliefs, based on acknowledgements and drops, and incorporating a preconceived notion of how quickly congestion levels might fluctuate

We consider a model in which, each round trip time (RTT), the user chooses how many packets to send in that RTT. We assume u ϵ {0,1,…,u max }. D is the number of dropped packets. The reward is u-D, the number of delivered packets. The cost is γ D, for some constant γ >0.

The distribution of D depends on the packet drop probability, Q. The user’s current Bayesian belief about Q is specified by a Beta distribution, parameterized by n and p. (Here, p is the expected drop probability and n is the “amount of evidence” for p.) The user’s belief about q is updated every RTT, in two ways: the user gains information about the distribution of Q, from observing D congestion levels may change over an RTT, which adds uncertainty to the distribution of Q. That is, the network is a restless bandit.

I solved the Bellman equation numerically, and derived an optimal congestion control algorithm. I then ran this algorithm on a link with packet drop probability 0.02.

Conclusion Internet congestion control can be analyzed using a mixture of control theory and stochastic analysis – Control theory = how do aggregates react to each other? It is fairly well understood. – Stochastic analysis = how do individual parts act, and how does the behaviour of an aggregate arise out of its individual parts? It is poorly understood. Dynamic programming is a mechanical way to derive a congestion control algorithm, given a “belief structure” about congestion. It is appropriate for situations where there is little aggregation. Arguably, recent practical developments in congestion control can be better understood by dynamic programming than by control theory.

Some questions Is there a “belief structure” about congestion that yields classic TCP as an optimal policy? What about Compound TCP or CUBIC? Why has classic TCP been good enough for so long? Does it arise from a belief structure that has been fairly accurate for most of the Internet’s history? How does a dynamic-programming congestion control algorithm fit in with the Network Utility Maximization approach?