1. Sizing Router Buffers How much packet buffers does a router need? C Router Source Destination 2T The current “Rule of Thumb” A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context –Mandated in backbone and edge routers. –Appears in RFPs and IETF architectural guidelines –Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR, 1994. –Already known by inventors of TCP [Van Jacobson, 1988] Buffer Size Probability Distribution B 0 Simulation Our new rule for sizing buffers A router needs a buffer size: 2T is the two-way propagation delay C is capacity of bottleneck link n is the number of long-lived flows through the router Why? What has changed? –High-end routers serve many unsynchronized flows –Statistical multiplexing reduces amount of buffering needed TCP Flows Router BufferLink Utilization PktsRAMModelSimExp 1000.5 x 1 x 2 x 3 x 64 129 258 387 1Mb 2Mb 4Mb 8Mb 96.9% 99.9% 100% 94.7% 99.3% 99.9% 99.8% 94.9% 98.1% 99.8% 99.7% 4000.5 x 1 x 2 x 3 x 32 64 128 192 512kb 1Mb 2Mb 4Mb 99.7% 100% 99.2% 99.8% 100% 99.5% 100% 99.9% What determines router buffer size? TCP can’t fully utilize a link without any buffers –Only with maximum congestion window the link is full –Window scales down just after it reaches maximum What is the effect of too much or too little buffer? –Too much buffer causes the buffer to never empty and increases latency –Too little buffering causes the buffer to be empty over extended periods of time and the link to be underutilized Right amount of buffering Overbuffered LinkUnderbuffered Link t B 0 For many flows, TCP windows are multiplexed –Buffer size is reduced Sum of congestion windows is Gaussian –Reason: The central limit theorem –Width of Gaussian is dependent on square root of number of flows –Buffer has to be wide enough to “fit” Gaussian Buffer requirement decreases with 1/sqrt(n) –Verified experimentally t Congestion Window W Link not fully utilized 4 8 The main role of a router buffer is to compensate for the difference in outstanding packets –TCP Window sizes changes over time (the TCP saw tooth) –Number of outstanding packets changes over time –Buffer has to buffer these packets Smaller buffers have a major effect on router design An Example: 10Gb/s linecard with 200,000 x 56kb/s flows –Rule-of-thumb: Buffer = 2.5Gbits Requires external, slow DRAM –Becomes: Buffer = 6Mbits Can use on-chip, fast SRAM 40Gb/s linecard with 40,000 x 1Mb/s flows –Rule-of-thumb: Buffer = 10Gbits –Becomes: Buffer = 50Mbits It is estimated that reduced buffers on a line cards could –Reduce power consumption by 10%-40% –Save 10%-40% of board space –Reduce cost by 10%-30% Do these results hold for a physical router? Yes. We verified our model by simulation and on a physical router –Measurements on a Cisco GSR with an OC3 line card and 400 flows –Simulations with ns2 over a broad range of parameters Below is a side by side comparison of Model, Simulation and Experiments on a physical router Smaller buffers help short flows In mixes of long and short TCP flows, short flows benefit from smaller buffers –RTT is decreased significantly (almost haved) by smaller buffers –Loss increases, but effect of RTT decrease outweighs loss Example: Completion time of short (14 packet) flows in a mix of long and short flows –We increase the number of long flows and measure average completion time –With smaller buffers completion time is substantially lower