1. Understanding Buffer Size Requirements in a Router
Thanks to
Nick McKeown and John Lockwood for numerous slides

2. Buffer Requirements in a Router
Buffer size matters:
Small queues reduce delay
Large buffers are expensive
Theoretical tools predict requirements
Queuing theory
Large deviation theory
Mean field theory
Yet, there is no direct answer
Flows have a closed-loop nature
Question arises on whether focus should be on equilibrium state or transient state
Having said that, one might think the buffer sizing problem must be very well understood.
After all, we are equipped with tools like queueing theory, large deviations theory, and mean field theory which are focused on solving exactly this type of problem.
You would think this is simply a matter of understanding the random process that describes the queue occupancy over time.
Unfortunately, this is not the case. The closed-loop nature of the flows, and the fact that flows react to the state of the system makes it necessary to use control theoretic tools, but those tools emphasize on the equilibrium state of the system, and fail to describe transient delays.

3. Rule-of-thumb Universally applied rule-of-thumb: Context
Source
Router
Destination C 2T
Universally applied rule-of-thumb:
A router needs a buffer size:
2T is the two-way propagation delay (or just 250ms)
C is capacity of bottleneck link
Context
Mandated in backbone and edge routers
Appears in RFPs and IETF architectural guidelines
Already known by inventors of TCP
[Van Jacobson, 1988]
Has major consequences for router design
So if the problem is not easy, what do people do in practice?
Buffer sizes in today’s Internet routers are set based on a rule-of-thumb which says
If we want the core routers to have 100% utilization,
The buffer size should be greater than or equal to 2TxC
Here 2T is the two way propagation delay of packets going through the router
And C is the capacity of the target link.
This rule is mandated in backbone and edge routers, and
Appears in RFPs and IETF architectural guidelines.
It has been known almost since the time TCP was invented.
Note that if the capacity of the network is increased, based on this rule, we need to increase the buffer size linearly with capacity.
We don’t expect the propagation delay changed that much over time, but we expect the capacity to grow very rapidly,
Therefore, this rule can have major consequences in router design, and that’s exactly why today’s routers have so much buffering as I showed you a few moments ago.

4. The Story So Far 10,000 20 1,000,000 # packets at 10Gb/s
After this relatively long introduction, let me give an overview of the rest of my presentation.
I'll talk about three different rules for sizing router buffers.
The first rule is the rule-of-thumb which I just described.
As I mentioned, this rule is based on the assumption that we want to have 100% link utilization at the core links.
The second rule is a more recent result proposed by Appenzeller, Keslassy, and McKeown which basically challenges the original rule-of-thumb.
Based on this rule if we have N flows going through the router,
we can reduce the buffer size by a factor of sqrt(N)
The underlying assumption is that we have
a large number of flows,
and the flows are desynchronized.
Finally, the third rule which I’ll talk about today, says that
If we are willing to sacrifice a very small amount of throughput, i.e. if having a throughput less than 100% is acceptable,
We might be able to reduce the buffer sizes significantly to just O(log(W)) packets.
Here W is the maximum congestion window size.
If we apply each of these rules to a 10Gb/s link
We will need to buffer 1,000,000 packets based on the first rule,
About 10,000 packets based on the 2nd one,
And only 20 packets based on the 3rd rule.
For the rest of this presentation
I’ll show you the intuition behind each of these rules; and
Will provide some evidence that validates the rule.
Let’s start with the rule-of-thumb.
Assume: Large number of desynchronized flows; 100% utilization
Assume: Large number of desynchronized flows; <100% utilization

5. Using NetFPGA to explore buffer size
Need to reduce buffer size and measure occupancy
Alas, not possible in commercial routers
So, we will use the NetFPGA instead
Objective:
Use the NetFPGA to understand how large a buffer we need for a single TCP flow.

6. Why 2TxC for a single TCP Flow?
Rule for adjusting W
If an ACK is received: W ← W+1/W
If a packet is lost: W ← W/2
Only W packets may be outstanding

7. Time Evolution of a Single TCP Flow
Time evolution of a single TCP flow through a router. Buffer is 2T*C
Time evolution of a single TCP flow through a router. Buffer is < 2T*C
Here is a simulation of a single TCP flow with a buffer size equal to the bandwidth delay product.
As you can see, the congestion window changes according to a sawtooth shape, and varies between 140, and 280.
On the bottom graph we can see the queue occupancy. As you can see, when the congestion window is halved the buffer occupancy becomes zero, and the two curves change similarly.
Note that since the pipe is full at all times, and the link utilization remains at 100%.
Now, on the other hand, when the buffer size is less than the bandwidth delay product, we can see that
When the congestion window is halved, the queue occupancy goes to zero and
Remains at zero for a while before the congestion window is increased again, and can fill up the pipe.
During this time, we see a reduction in link utilization.

8. Instructions for a real evaluation using NetFPGA
Observing and Controlling the Queue Size
Instructions for a real evaluation using NetFPGA

9. Setup for the Buffer Evaluation
Adjacent
Web & Video Server
nf2c1
eth2
NetFPGA
Local Host
nf2c2
eth1
Router 9

10. Enhanced Router Objectives Execution Observe router with new modules
New modules: rate limiting, event capture
Execution
Run event capture router
Look at routing tables
Explore details pane
Start tcp transfer, look at queue occupancy
Change rate, look at queue occupancy

11. Step 1 - Run Pre-made Enhanced Router
Start terminal and cd to “NF2/projects/tutorial_router/sw/”
Type “./tut_adv_router_gui.pl”
A familiar GUI should start

12. Step 2 - Explore Enhanced Router
Click on the Details tab
A similar pipeline to the one seen previously shown with some additions

13. Enhanced Router Pipeline
MAC
RxQ
CPU
Input Arbiter
Output Port Lookup
TxQ
Output Queues
Rate
Limiter
Event Capture
Two modules added
Event Capture to capture output queue events (writes, reads, drops)
Rate Limiter to create a bottleneck

14. Step 3 - Decrease the Link Rate
To create bottleneck and show the TCP “sawtooth,” link-rate is decreased.
In the Details tab, click the “Rate Limit” module
Check Enabled
Set link rate to 1.953Mbps 14

15. Step 4 – Decrease Queue Size
Go back to the Details panel and click on “Output Queues” Select the “Output Queue 2” tab Change the output queue size in packets slider to 16

16. Step 5 - Start Event Capture
Click on the Event Capture module under the Details tab
This should start the configuration page

17. Step 6 - Configure Event Capture
Check Send to local host to receive events on the local host
Check Monitor Queue 2 to monitor output queue of MAC port1
Check Enable Capture to start event capture

18. Step 7 - Start TCP Transfer
We will use iperf to run a
large TCP transfer and
look at queue evolution
Start a terminal and cd to
“NF2/projects/tutorial_router/sw”
Type “./iperf.sh” 18

19. Step 8 - Look at Event Capture Results
Click on the Event Capture module under the Details tab.
The sawtooth pattern should now be visible.

20. Queue Occupancy Charts
Observe the TCP/IP sawtooth
Leave the control windows open