1. May 28th, 2002Nick McKeown 1 Scaling routers: Where do we go from here? HPSR, Kobe, Japan May 28 th, 2002 Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford University nickm@stanford.edu www.stanford.edu/~nickm

2. May 28th, 2002Nick McKeown 2 Router capacity x2.2/18 months Moore’s law x2/18 m

3. May 28th, 2002Nick McKeown 3 Router capacity x2.2/18 months DRAM access rate x1.1/18 m Moore’s law x2/18 m

4. May 28th, 2002Nick McKeown 4 Router vital statistics Cisco GSR 12416Juniper M160 6ft 19” 2ft Capacity: 160Gb/s Power: 4.2kW 3ft 2.5ft 19” Capacity: 80Gb/s Power: 2.6kW

5. May 28th, 2002Nick McKeown 5 Internet traffic x2/yr Router capacity x2.2/18 months 5x

6. May 28th, 2002Nick McKeown 6 POP with smaller routersPOP with large routers  Interfaces: Price >$200k, Power > 400W  About 50-60% of interfaces are used for interconnection within the POP.  Industry trend is towards large, single router per POP. Fast (large) routers  Big POPs need big routers

7. May 28th, 2002Nick McKeown 7 Job of router architect  For a given set of features:

8. May 28th, 2002Nick McKeown 8 Mind the gap  Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption.  Our options: 1. Make routers simple 2. Use more parallelism 3. Use more optics

9. May 28th, 2002Nick McKeown 9 Mind the gap  Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption.  Our options: 1. Make routers simple 2. Use more parallelism 3. Use more optics

10. May 28th, 2002Nick McKeown 10 Make routers simple  We tell our students that Internet routers are simple. All routers do is make a forwarding decision, update a header, then forward packets to the correct outgoing interface.  But I don’t understand them anymore.  List of required features is huge and still growing,  Software is complex and unreliable,  Hardware is complex and power-hungry.

11. May 28th, 2002Nick McKeown 11 Router linecard Physical Layer Framing & Maintenance Packet Processing Buffer Mgmt & Scheduling Buffer Mgmt & Scheduling Buffer & State Memory Buffer & State Memory OC192c linecard  30M gates  2.5Gbits of memory  1m 2  $25k cost, $200k price. Lookup Tables Optics Scheduler

12. May 28th, 2002Nick McKeown 12 Things that slow routers down  250ms of buffering  Requires off-chip memory, more board space, pins and power.  Multicast  Affects everything!  Complicates design, slows deployment.  Latency bounds  Limits pipelining.  Packet sequence  Limits parallelism.  Small internal cell size  Complicates arbitration.  DiffServ, IntServ, priorities, WFQ etc.  Others: IPv6, Drop policies, VPNs, ACLs, DOS traceback, measurement, statistics, …

13. May 28th, 2002Nick McKeown 13 An example: Packet processing CPU Instructions per minimum length packet since 1996

14. May 28th, 2002Nick McKeown 14 Reducing complexity Conclusion  Need aggressive reduction in complexity of routers.  Get rid of irrelevant requirements and irrational tests.  It is not clear who has the right incentive to make this happen.  Else, be prepared for core routers to be replaced by optical circuit switches.

15. May 28th, 2002Nick McKeown 15 Mind the gap  Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption.  Our options: 1. Make routers simpler 2. Use more parallelism 3. Use more optics

16. May 28th, 2002Nick McKeown 16 Use more parallelism  Parallel packet buffers  Parallel lookups  Parallel packet switches  Things that make parallelism hard:  Maintaining packet order,  Making throughput guarantees,  Making delay guarantees,  Latency requirements,  Multicast.

17. May 28th, 2002Nick McKeown 17 Parallel Packet Switches 1 2 k 1 N rate, R 1 N Router Bufferless

18. May 28th, 2002Nick McKeown 18 Characteristics  Advantages  k  memory bandwidth   k  lookup/classification rate   k  routing/classification table size   With appropriate algorithms  Packets remain in order,  100% throughput,  Delay guarantees (at least in theory).

19. May 28th, 2002Nick McKeown 19 Mind the gap  Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption.  Our options: 1. Make routers simpler 2. Use more parallelism 3. Use more optics

20. May 28th, 2002Nick McKeown 20  A router is a packet-switch, and so requires  A switch fabric,  Per-packet address lookup,  Large buffers for times of congestion.  Packet processing/buffering infeasible with optics  A typical 10 Gb/s router linecard has 30 Mgates and 2.5 Gbits of memory.  Research Problem  How to optimize the architecture of a router that uses an optical switch fabric? All-optical routers don’t make sense

21. May 28th, 2002Nick McKeown 21 100Tb/s optical router Stanford University Research Project  Collaboration  4 Professors at Stanford (Mark Horowitz, Nick McKeown, David Miller and Olav Solgaard), and our groups.  Objective  To determine the best way to incorporate optics into routers.  Push technology hard to expose new issues. Photonics, Electronics, System design  Motivating example: The design of a 100 Tb/s Internet router Challenging but not impossible (~100x current commercial systems) It identifies some interesting research problems

22. May 28th, 2002Nick McKeown 22 Arbitration 160Gb/s 40Gb/s Optical Switch Line termination IP packet processing Packet buffering Line termination IP packet processing Packet buffering 160- 320Gb/s 160- 320Gb/s Electronic Linecard #1 Electronic Linecard #625 Request Grant (100Tb/s = 625 * 160Gb/s) 100Tb/s optical router

23. May 28th, 2002Nick McKeown 23 Research Problems  Linecard  Memory bottleneck: Address lookup and packet buffering.  Architecture  Arbitration: Computation complexity.  Switch Fabric  Optics: Fabric scalability and speed,  Electronics: Switch control and link electronics,  Packaging: Three surface problem.

24. May 28th, 2002Nick McKeown 24 160Gb/s Linecard: Packet Buffering  Problem  Packet buffer needs density of DRAM (40 Gbits) and speed of SRAM (2ns per packet)  Solution  Hybrid solution uses on-chip SRAM and off-chip DRAM.  Identified optimal algorithms that minimize size of SRAM (12 Mbits).  Precisely emulates behavior of 40 Gbit, 2ns SRAM. DRAM 160 Gb/s Queue Manager klamath.stanford.edu/~nickm/papers/ieeehpsr2001.pdf SRAM

25. May 28th, 2002Nick McKeown 25 The Arbitration Problem  A packet switch fabric is reconfigured for every packet transfer.  At 160Gb/s, a new IP packet can arrive every 2ns.  The configuration is picked to maximize throughput and not waste capacity.  Known algorithms are too slow.

26. May 28th, 2002Nick McKeown 26 Approach  We know that a crossbar with VOQs, and uniform Bernoulli i.i.d. arrivals, gives 100% throughput for the following scheduling algorithms:  Pick a permutation uar from all permutations.  Pick a permutation uar from the set of size N in which each input-output pair (i,j) are connected exactly once in the set.  From the same set as above, repeatedly cycle through a fixed sequence of N different permutations.  Can we make non-uniform, bursty traffic uniform “enough” for the above to hold?

27. May 28th, 2002Nick McKeown 27 2-Stage Switch External Outputs Internal Inputs 1 N External Inputs Spanning Set of Permutations 1 N 1 N  Recently shown to have 100% throughput  Mild conditions: weakly mixing arrival processes C.S.Chang et al.: http://www.ee.nthu.edu.tw/~cschang/PartI.pdf

28. May 28th, 2002Nick McKeown 28 2-Stage Switch External Outputs Internal Inputs 1 N External Inputs Spanning Set of Permutations 1 N 1 N

29. May 28th, 2002Nick McKeown 29 Problem: Unbounded Mis-sequencing External Outputs Internal Inputs 1 N External Inputs Spanning Set of Permutations 1 N 1 N 1 1 2 2  Side-note: Mis-sequencing is maximized when arrivals are uniform.

30. May 28th, 2002Nick McKeown 30 Preventing Mis-sequencing 1 N Spanning Set of Permutations 1 N 1 N  The Full Frames First algorithm:  Keep packets ordered and  Guarantees a delay bound within the optimum Infocom’02: klamath.stanford.edu/~nickm/papers/infocom02_two_stage.pdf Small Coordination Buffers & ‘FFF’ Algorithm Large Congestion Buffers

31. May 28th, 2002Nick McKeown 31 1 2 3 Phase 2 Phase 1 Idea: Use a single-stage twice Example Optical 2-stage Switch Lookup Buffer Lookup Buffer Lookup Buffer Linecards

32. May 28th, 2002Nick McKeown 32 Example Passive Optical 2-Stage “Switch” R/N Ingress Linecard 1 Ingress Linecard 2 Ingress Linecard n Midstage Linecard 1 Midstage Linecard 2 Midstage Linecard n Egress Linecard 1 Egress Linecard 2 Egress Linecard n It is helpful to think of it as spreading rather than switching.

33. May 28th, 2002Nick McKeown 33 2-Stage spreading 111 N Buffer stage NN

34. May 28th, 2002Nick McKeown 34 Passive Optical Switching 1 1 2 2 n n 1 2 n Midstage Linecard 1 Midstage Linecard 2 Midstage Linecard n Ingress Linecard 1 Ingress Linecard 2 Ingress Linecard n 1 2 n Egress Linecard 1 Egress Linecard 2 Egress Linecard n 11 22 nn Integrated AWGR or diffraction grating based wavelength router

35. May 28th, 2002Nick McKeown 35 100Tb/s Router Optical Switch Fabric Racks of 160Gb/s Linecards Optical links

36. May 28th, 2002Nick McKeown 36 Racks with 160Gb/s linecards DRAM Queue Manager SRAM Lookup DRAM Queue Manager SRAM Lookup

37. May 28th, 2002Nick McKeown 37 Additional Technologies  Demonstrated or in development  Chip to chip optical interconnects with total power dissipations of several mW.  Demonstration of wavelength division multiplexed chip interconnect.  Integrated laser modulators.  8Gsample/s serial links.  Low-power variable power supply serial links.  Integrated array waveguide routers. 40 μm

38. May 28th, 2002Nick McKeown 38 Mind the gap  Operators are unlikely to deploy 5 times as many POPs, or make them 5 times bigger, with 5 times the power consumption.  Our options: 1. Make routers simpler 2. Use more parallelism 3. Use more optics

39. May 28th, 2002Nick McKeown 39 Some predictions about core Internet routers  The need for more capacity for a given power and volume budget will mean:  Fewer functions in routers:  Little or no optimization for multicast,  Continued overprovisioning will lead to little or no support for QoS, DiffServ, …,  Fewer unnecessary requirements:  Mis-sequencing will be tolerated,  Latency requirements will be relaxed.  Less programmability in routers, and hence no network processors.  Greater use of optics to reduce power in switch.

40. May 28th, 2002Nick McKeown 40 What I believe is most likely The need for capacity and reliability will mean:  Widespread replacement of core routers with transport switching based on circuits:  Circuit switches have proved simpler, more reliable, lower power, higher capacity and lower cost per Gb/s. Eventually, this is going to matter.  Internet will evolve to become edge routers interconnected by rich mesh of WDM circuit switches.