1. Networks with Very Small Buffers Yashar Ganjali, Guido Appenzeller, High Performance Networking Group Prof. Ashish Goel, Prof. Tim Roughgarden, Prof. Nick McKeown A router needs a buffer size: 2T is the two-way propagation delay C is the capacity of bottleneck link Context Mandated in backbone and edge routers Appears in RFPs and IETF architectural guidelines Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR, 1994 Already known by inventors of TCP [Van Jacobson, 1988] C Router Source Destination 2T A router needs a buffer size: 2T is the two-way propagation delay C is the capacity of bottleneck link n is the number of long-lived flows through the router Why? What has changed? High-end routers serve many unsynchronized flows Statistical multiplexing reduces the amount of buffering needed TCP needs buffers to fully utilize a link Only with maximum congestion window the links is full Window scales down just after it reaches maximum The main role of a router buffer is to compensate for the difference in outstanding packets TCP window size changes over time (the TCP saw tooth) Number of outstanding packets changes over time Buffer has to buffer these packets router source server source 10Gb/s 5Mb/s router source server source 10Gb/s >10Gb/s The current “Rule of Thumb” Our new rule for sizing buffers What determines router buffer size? Slow access links, lots of flows Congested core router; RTT = 100ms Each flow limited to 2.5Mb/s Congested core router RTT = 100ms; Each flow limited to 2.5Mb/s Fast access links, TCP Reno vs. Paced TCP t B 0 Sum of congestion windows is Gaussian For many flows, TCP windows are multiplexed Buffer size is reduced Sum of congestion windows is Gaussian Reason: the central limit theorem Width of Gaussian is dependent on square root of number of flows Buffer has to be wide enough to “fit” Gaussian Probability Distribution B 0 Buffer Size Validation Theoretical results validated by Thousands of ns-2 simulations Network lab (Cisco routers) at University of Wisconsin Stanford University dorm traffic Internet2 experiments Buffer size is reduced Ongoing work with network operators and router vendors …. Simulation How small buffers can be? Example: 40 Gb/s linecard with 40,000 x 1Mb/s flows Rule-of-thumb: Buffer = 10Gbits New rule: Buffer = 50Mbits What we know so far about very small buffers? Arbitrary Injection Process If Poisson Process with load < 1 Complete Centralized Control Any rate > 0 need unbounded buffers TheoryExperiment Need buffer size of approx: O(logD + logW) D=#of hops W=window size [Goel 2004] TCP Pacing: Results as good or better than for Poisson Constant fraction throughput with constant buffers [Leighton] 1.Date-rate of most flows small fraction of line rate Access links throttle flows to low rate (1-2Mb/s) Core:Access > 1000:1 Today’s TCP’s window size is limited 2.If packet arrivals were Poisson We get 80-90% utilization with 5-10 packet buffers 3.In an all-optical network capacity is plentiful (presumably)