1. Jon Turner jst@cs.wustl.edu http://www.arl.wustl.edu/arl Extreme Networking Achieving Nonstop Network Operation Under Extreme Operating Conditions

2. 2 - Jonathan Turner - January 15, 2002 Project Overview Motivation »data networks have become mission-critical resource »networks often subject to extreme traffic conditions »need to design networks for worst-case conditions »technology advances making extreme defenses practical Extreme network services »Lightweight Flow Setup (LFS) »Network Access Service (NAS) »Distributed Tree Service (DTS) Key router technology components »Super-Scalable Packet Scheduling (SPS) »Dynamic Queues with Auto-aggregation (DQA) »Scalable Distributed Queueing (SDQ)

3. 3 - Jonathan Turner - January 15, 2002 Extreme Router Architecture Control Processor Switch Fabric... Flow/Route Lookup Dist. Q. Ctl. Output Port Proc. Flow Lookup Input Port Proc. Flow/Route Lookup Dist. Q. Ctl. Flow Lookup Lookup route or state for reserved flows Scalable switch fabric system mgmt. route table cfg. signalling Distrib. queueing traffic isolation protect res. flows

4. 4 - Jonathan Turner - January 15, 2002 Switch Fabric IPPOPP FPX SPC TI IPPOPP FPX SPC TI IPPOPP FPX SPC TI IPPOPP FPX SPC TI IPPOPP FPX SPC TI IPPOPP FPX SPC TI Control Processor Prototype Extreme Router Field Programmable Port Ext. Network Interface Device Reprogrammable Application Device SDRAM 128 MB SRAM 4 MB Field Programmable Port Extenders Smart Port Card Sys. FPGA 64 MB Pentium Cache North Bridge APIC ATM Switch Core Transmisson InterfacesEmbedded Processors

5. 5 - Jonathan Turner - January 15, 2002 Distributed Queueing Switch Fabric TI IOIOIO IO IO IO Control Processor Routing Sched. Routing Sched. Routing Sched. Routing Sched. Routing Sched. Routing Sched. queue per output periodic queue length reports Scheduler paces each queue according to backlog share

6. 6 - Jonathan Turner - January 15, 2002 Is Distributed Queueing Necessary? ATM switches generally do not do it. »switch is engineered with small speedup (typically 2:1) »with well-regulated traffic, do not expect >2:1 overload Overloads more likely in IP networks. »limited route diversity makes congested links common »route selection not guided by session bandwidth needs »routing changes cause rapid shifts in traffic »crude, slow congestion control mechanism »no protection from malicious users Challenges »prevent congestion while avoiding “underflow” »scalability - target 1000x10 Gb/s systems »support fair queueing and reserved flow queueing

7. 7 - Jonathan Turner - January 15, 2002 Basic Distributed Queueing Algorithm Goal: avoid switch congestion and output queue underflow. Let hi(i,j) be input i’s share of input-side backlog to output j. »can avoid switch congestion by sending from input i to output j at rate  L  S  hi(i,j) »where L is external link rate and S is switch speedup Let lo(i,j) be input i’s share of total backlog for output j. »can avoid underflow of queue at output j by sending from input i to output j at rate  L  lo(i,j) »this works if L  (lo(i,1)+···+lo(i,n))  L  S for all i Let wt(i,j) be the ratio of lo(i,j) to lo(i,1) + ··· + lo(i,n). Let rate(i,j)=L  S  min{wt(i,j),hi(i,j)}. Note: algorithm avoids congestion and for large enough S, avoids underflow. »what is the smallest value of S for which underflow cannot occur?

8. 8 - Jonathan Turner - January 15, 2002 Stress Test can vary number of inputs and outputs used, and length of “phases”

9. 9 - Jonathan Turner - January 15, 2002 Stress Test Simulation - Min Rates first phase second critical rate

10. 10 - Jonathan Turner - January 15, 2002 Stress Test - Actual Rates critical rate first phase second Under-use of input bandwidth

11. 11 - Jonathan Turner - January 15, 2002 Stress Test - Input Queue Lengths input side backlog for final output implies underflow

12. 12 - Jonathan Turner - January 15, 2002 Stress Test - Output Queue Lengths persistent output side backlog caused by earlier dip in forwarding rate

13. 13 - Jonathan Turner - January 15, 2002 Improving Basic Algorithm Basic algorithm does not always make full use of available input bandwidth. »does not reallocate bandwidth that is “sacrificed” by queues that are “output limited” »extend algorithm to reallocate Revised rate allocation at input i: R = S  L repeat n times Let j be unassigned queue with smallest ratio hi(i,j)/lo(i,j) Let wt(i,j) = lo(i,j)/(sum of lo(i,q) for unassigned queues q) rate(i,j) = min{R  wt(i,j),S  L  hi(i,j)} R = R - rate(i,j) Plus other refinements.

14. 14 - Jonathan Turner - January 15, 2002 Performance Gain - Allocated Rates full use of input bandwidth preallocate bandwidth to idle outputs

15. 15 - Jonathan Turner - January 15, 2002 Performance Gain - Min Rates critical rate

16. 16 - Jonathan Turner - January 15, 2002 Worst-Case Min Rate Sums

17. 17 - Jonathan Turner - January 15, 2002 Results for Random Bursty Traffic Lost link capacity is negligible for speedups greater than 1.2

18. 18 - Jonathan Turner - January 15, 2002 Extending for Fair Queueing Fair queueing gives each flow equal share of congested link. »limits impact of “greedy” users on others »improves performance of congestion control mechanisms, reducing queueing delays and packet loss Partial solution »per flow queues with packet scheduler at each output »provides fairness when no significant input-side queueing Better solution »per flow input and output queues »distributed queueing controls rates of per-output schedulers at the inputs »bandwidth allocated by number of backlogged queues

19. 19 - Jonathan Turner - January 15, 2002 Fair Distributed Queueing Periodic update messages contain information on both backlog and number of backlogged queues.... Switch Fabric... dq... to output 1 to output 2 to output n separate queue set for each output dist. queueing controls rate of each queue set

20. 20 - Jonathan Turner - January 15, 2002 Fair Distributed Queueing Algorithm Same objectives as before plus fairness. »each backlogged queue gets equal share of congested output »so, allocate bandwidth according to number of backlogged queues Let Q(i,j) be number of backlogged queues at input i for j. Let hi(i,j) = Q(i,j)/(Q(1,j) +    + Q(n,j)). »can avoid switch congestion by ensuring rate(i,j)  L  S  hi(i,j) Let need(j) be total input-side share of backlog to output j. Let lo(i,j) = need(j)  Q(i,j)/(Q(1,j) +    + Q(n,j)). »can avoid underflow by ensuring rate(i,j)  L  lo(i,j) »this works if L  (lo(i,1)+···+lo(i,n))  L  S for all i Use same rate allocation as before with modified lo and hi. For weighted fair queueing, re-define Q(i,j) to be total weight of backlogged queues at input i for output j.

21. 21 - Jonathan Turner - January 15, 2002 Summary Growing reliance on data networks creates higher expectations - reliability, consistent performance. »design for worst-case - constructive paranoia »extreme defenses can be practical Distributed queueing is key component of scalable extreme routers. »with small speedup, prevents congestion (always) and underflow (almost always) while ensuring fairness (mostly) »increases latency and complexity Current reconfigurable hardware capabilities. »67K elementary logic cells (LUT+FF) plus 2.5 Mb of SRAM »over 1K IO pads, high speed IOs (>500 MHz) »enables experimental implementation of complex features