Understanding Buffer Size Requirements in a Router Thanks to Nick McKeown and John Lockwood for numerous slides

Buffer Requirements in a Router Buffer size matters: Small queues reduce delay Large buffers are expensive Theoretical tools predict requirements Queuing theory Large deviation theory Mean field theory Yet, there is no direct answer Flows have a closed-loop nature Question arises on whether focus should be on equilibrium state or transient state Having said that, one might think the buffer sizing problem must be very well understood. After all, we are equipped with tools like queueing theory, large deviations theory, and mean field theory which are focused on solving exactly this type of problem. You would think this is simply a matter of understanding the random process that describes the queue occupancy over time. Unfortunately, this is not the case. The closed-loop nature of the flows, and the fact that flows react to the state of the system makes it necessary to use control theoretic tools, but those tools emphasize on the equilibrium state of the system, and fail to describe transient delays.

Rule-of-thumb Universally applied rule-of-thumb: Context Source Router Destination C 2T Universally applied rule-of-thumb: A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge routers Appears in RFPs and IETF architectural guidelines Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design So if the problem is not easy, what do people do in practice? Buffer sizes in today’s Internet routers are set based on a rule-of-thumb which says If we want the core routers to have 100% utilization, The buffer size should be greater than or equal to 2TxC Here 2T is the two way propagation delay of packets going through the router And C is the capacity of the target link. This rule is mandated in backbone and edge routers, and Appears in RFPs and IETF architectural guidelines. It has been known almost since the time TCP was invented. Note that if the capacity of the network is increased, based on this rule, we need to increase the buffer size linearly with capacity. We don’t expect the propagation delay changed that much over time, but we expect the capacity to grow very rapidly, Therefore, this rule can have major consequences in router design, and that’s exactly why today’s routers have so much buffering as I showed you a few moments ago.

The Story So Far 10,000 20 1,000,000 # packets at 10Gb/s After this relatively long introduction, let me give an overview of the rest of my presentation. I'll talk about three different rules for sizing router buffers. The first rule is the rule-of-thumb which I just described. As I mentioned, this rule is based on the assumption that we want to have 100% link utilization at the core links. The second rule is a more recent result proposed by Appenzeller, Keslassy, and McKeown which basically challenges the original rule-of-thumb. Based on this rule if we have N flows going through the router, we can reduce the buffer size by a factor of sqrt(N) The underlying assumption is that we have a large number of flows, and the flows are desynchronized. Finally, the third rule which I’ll talk about today, says that If we are willing to sacrifice a very small amount of throughput, i.e. if having a throughput less than 100% is acceptable, We might be able to reduce the buffer sizes significantly to just O(log(W)) packets. Here W is the maximum congestion window size. If we apply each of these rules to a 10Gb/s link We will need to buffer 1,000,000 packets based on the first rule, About 10,000 packets based on the 2nd one, And only 20 packets based on the 3rd rule. For the rest of this presentation I’ll show you the intuition behind each of these rules; and Will provide some evidence that validates the rule. Let’s start with the rule-of-thumb. Assume: Large number of desynchronized flows; 100% utilization Assume: Large number of desynchronized flows; <100% utilization

Using NetFPGA to explore buffer size Need to reduce buffer size and measure occupancy Alas, not possible in commercial routers So, we will use the NetFPGA instead Objective: Use the NetFPGA to understand how large a buffer we need for a single TCP flow.

Why 2TxC for a single TCP Flow? Rule for adjusting W If an ACK is received: W ← W+1/W If a packet is lost: W ← W/2 Only W packets may be outstanding

Time Evolution of a Single TCP Flow Time evolution of a single TCP flow through a router. Buffer is 2T*C Time evolution of a single TCP flow through a router. Buffer is < 2T*C Here is a simulation of a single TCP flow with a buffer size equal to the bandwidth delay product. As you can see, the congestion window changes according to a sawtooth shape, and varies between 140, and 280. On the bottom graph we can see the queue occupancy. As you can see, when the congestion window is halved the buffer occupancy becomes zero, and the two curves change similarly. Note that since the pipe is full at all times, and the link utilization remains at 100%. Now, on the other hand, when the buffer size is less than the bandwidth delay product, we can see that When the congestion window is halved, the queue occupancy goes to zero and Remains at zero for a while before the congestion window is increased again, and can fill up the pipe. During this time, we see a reduction in link utilization.

Instructions for a real evaluation using NetFPGA Observing and Controlling the Queue Size Instructions for a real evaluation using NetFPGA

Setup for the Buffer Evaluation Adjacent Web & Video Server nf2c1 eth2 NetFPGA Local Host nf2c2 eth1 Router 9

Enhanced Router Objectives Execution Observe router with new modules New modules: rate limiting, event capture Execution Run event capture router Look at routing tables Explore details pane Start tcp transfer, look at queue occupancy Change rate, look at queue occupancy

Step 1 - Run Pre-made Enhanced Router Start terminal and cd to “NF2/projects/tutorial_router/sw/” Type “./tut_adv_router_gui.pl” A familiar GUI should start

Step 2 - Explore Enhanced Router Click on the Details tab A similar pipeline to the one seen previously shown with some additions

Enhanced Router Pipeline MAC RxQ CPU Input Arbiter Output Port Lookup TxQ Output Queues Rate Limiter Event Capture Two modules added Event Capture to capture output queue events (writes, reads, drops) Rate Limiter to create a bottleneck

Step 3 - Decrease the Link Rate To create bottleneck and show the TCP “sawtooth,” link-rate is decreased. In the Details tab, click the “Rate Limit” module Check Enabled Set link rate to 1.953Mbps 14

Step 4 – Decrease Queue Size Go back to the Details panel and click on “Output Queues” Select the “Output Queue 2” tab Change the output queue size in packets slider to 16

Step 5 - Start Event Capture Click on the Event Capture module under the Details tab This should start the configuration page

Step 6 - Configure Event Capture Check Send to local host to receive events on the local host Check Monitor Queue 2 to monitor output queue of MAC port1 Check Enable Capture to start event capture

Step 7 - Start TCP Transfer We will use iperf to run a large TCP transfer and look at queue evolution Start a terminal and cd to “NF2/projects/tutorial_router/sw” Type “./iperf.sh” 18

Step 8 - Look at Event Capture Results Click on the Event Capture module under the Details tab. The sawtooth pattern should now be visible.

Queue Occupancy Charts Observe the TCP/IP sawtooth Leave the control windows open