1. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 High Speed Router Design Shivkumar Kalyanaraman Rensselaer Polytechnic Institute shivkuma@ecse.rpi.edu http://www.ecse.rpi.edu/Homepages/shivkuma Also based on slides of S. Keshav (Ensim), Douglas Comer (Purdue), Minkenberg (IBM Zurich), Sonia Fahmy (Purdue) Raj Yavatkar (Intel), Cyriel Minkenberg (IBM Zurich), Sonia Fahmy (Purdue) Many slides thanks to Nick McKeown (Stanford),

2. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 2 q Introduction q Evolution of High-Speed Routers q High Speed Router Components: q Lookup Algorithm q Switching q Classification, Scheduling q Multi-Tbps Routers: Challenges & Trends Overview

3. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 3 What do switches/routers look like? Access routers e.g. ISDN, ADSL Core router e.g. OC48c POS Core ATM switch

4. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 4 Dimensions, Power Consumption Cisco GSR 12416Juniper M160 6ft 19 ” 2ft Capacity: 160Gb/s Power: 4.2kW 3ft 2.5ft 19 ” Capacity: 80Gb/s Power: 2.6kW

5. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 5 Where high performance packet switches are used Enterprise WAN access & Enterprise Campus Switch - Carrier Class Core Router - ATM Switch - Frame Relay Switch The Internet Core Edge Router

6. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 6 Where are routers? Ans: Points of Presence (POPs) A B C POP1 POP3 POP2 POP4 D E F POP5 POP6 POP7 POP8

7. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 7 POP with smaller routersPOP with large routers q Interfaces: Price >$200k, Power > 400W q Space, power, interface cost economics! q About 50-60% of i/fs are used for interconnection within the POP. q Industry trend is towards large, single router per POP. Why the Need for Big/Fast/Large Routers?

8. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 8 Job of router architect q For a given set of features:

9. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 9 Performance metrics 1. Capacity  “maximize C, s.t. volume < 2m 3 and power < 5kW” 2. Throughput q Maximize usage of expensive long-haul links. q Trivial with work-conserving output-queued routers 3. Controllable Delay q Some users would like predictable delay. q This is feasible with output-queueing plus weighted fair queuing (WFQ). WFQ

10. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 10 DWDM Link speed x2/8 months Router capacity x2.2/18 months Moore’s law x2/18 m DRAM access rate x1.1/18 m Internet x2/yr

11. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 11 Alt: Memory Bandwidth Commercial DRAM q Memory speed is not keeping up with Moore’s Law. DRAM 1.1x / 18months Moore’s Law 2x / 18 months Router Capacity 2.2x / 18months Line Capacity 2x / 7 months

12. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 12 An Example: Packet buffers 40Gb/s router linecard Buffer Memory Write Rate, R One 40B packet every 8ns Read Rate, R One 40B packet every 8ns 10Gbits Buffer Manager  Use SRAM? + Fast enough random access time, but - Too low density to store 10Gbits of data.  Use DRAM? + High density means we can store data, but - Can’t meet random access time.

13. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 13 Eg: Problems w/ Output Queuing q Output queued switches are impractical R R R R DRAM NR data R R R R output 1 N Can’t I just use N separate memory devices per output?

14. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 14 Packet processing is getting harder CPU Instructions per minimum length packet since 1996

15. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 15 Basic Ideas: Part I

16. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 16 First-Generation IP Routers  Most Ethernet switches and cheap packet routers  Bottleneck can be CPU, host-adaptor or I/O bus  What is costly? Bus ? Memory? Interface? CPU? Shared Backplane Line Interface CPU Memory CPU Buffer Memory Line Interface DMA MAC Line Interface DMA MAC Line Interface DMA MAC

17. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 17 First Generation Routers Shared Backplane Line Interface CPU Memory Route Table CPU Buffer Memory Line Interface MAC Line Interface MAC Line Interface MAC Fixed length “DMA” blocks or cells. Reassembled on egress linecard Fixed length cells or variable length packets Typically <0.5Gb/s aggregate capacity

18. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 18 Output 2 Output N First Generation Routers Queueing Structure: Shared Memory Large, single dynamically allocated memory buffer: N writes per “cell” time N reads per “cell” time. Limited by memory bandwidth. Input 1 Output 1 Input N Input 2 Numerous work has proven and made possible: q Fairness q Delay Guarantees q Delay Variation Control q Loss Guarantees q Statistical Guarantees

19. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 19 Second-Generation IP Routers CPU Buffer Memory Line Card DMA MAC LocalBufferMemory Line Card DMA MAC LocalBufferMemory Line Card DMA MAC LocalBufferMemory  Port mapping intelligence in line cards  Higher hit rate in local lookup cache  What is costly? Bus ? Memory? Interface? CPU?

20. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 20 Second Generation Routers Route Table CPU Line Card Buffer Memory Line Card MAC Buffer Memory Line Card MAC Buffer Memory Fwding Cache Fwding Cache Fwding Cache MAC Slow Path Drop Policy Drop Policy Or Backpressure Output Link Scheduling Buffer Memory Typically <5Gb/s aggregate capacity

21. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 21 Route Table CPU Second Generation Routers As caching became ineffective Line Card Buffer Memory Line Card MAC Buffer Memory Line Card MAC Buffer Memory Fwding Table Fwding Table Fwding Table MAC Exception Processor

22. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 22 Second Generation Routers Queuing Structure: Combined Input and Output Queuing (CIOQ) Bus 1 write per “cell” time1 read per “cell” time Rate of writes/reads determined by bus speed

23. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 23 Third-Generation Switches/Routers Line Card MAC Local Buffer Memory CPU Card Line Card MAC Local Buffer Memory Switched Backplane Line Interface CPU Memory  Third generation switch provides parallel paths (fabric)  What’s costly? Bus? Memory, CPU?

24. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 24 Third Generation Routers Line Card MAC Local Buffer Memory CPU Card Line Card MAC Local Buffer Memory Switched Backplane Line Interface CPU Memory Fwding Table Routing Table Fwding Table Typically <50Gb/s aggregate capacity

25. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 25 Arbiter Third Generation Routers Queueing Structure Switch 1 write per “cell” time1 read per “cell” time Rate of writes/reads determined by switch fabric speedup

26. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 26 Arbiter Third Generation Routers Queueing Structure: VOQs Switch 1 write per “cell” time1 read per “cell” time Rate of writes/reads determined by switch fabric speedup Per-flow/class or per- output queues (VOQs) Per-flow/class or per- input queues Flow-control backpressure

27. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 27 Third Generation Routers: limits 19” or 23” 7’ Size-constrained: 19” or 23” wide. Power-constrained: ~<8kW. Supply: 100A/200A maximum at 48V

28. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 28 Fourth Generation: Clustering/Multi-stage Switch Core Linecards Optical links 100’s of feet

29. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 29 Key: Physically Separating Switch Core and Linecards q Distributes power over multiple racks. q Allows all buffering to be placed on the linecard: q Reduces power. q Places complex scheduling, buffer mgmt, drop policy etc. on linecard.

30. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 30 Fourth Generation Routers/Switches Switch Core Linecards Optical links 100’s of feet The LCS Protocol

31. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 31 Linecard LCSLCS LCSLCS 1: Req Physical Separation 3: Data Switch Scheduler Switch Scheduler 2: Grant/credit Seq num Switch Fabric Switch Fabric Switch Port Req Grant 1 RTT Per Queue Counters

32. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 32 Physical Separation Aligning Cells Switch Scheduler Switch Scheduler Switch Fabric Switch Fabric LCSLCS LCSLCS LCSLCS Switch Core Linecard

33. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 33 Fourth Generation Routers/Switches Queueing Structure 1 write per “cell” time 1 read per “cell” time Rate of writes/reads determined by switch fabric speedup Lookup & Drop Policy Output Scheduling Virtual Output Queues Output Scheduling Output Scheduling Switch Fabric Switch Arbitration Linecard Switch Core (Bufferless) Lookup & Drop Policy Lookup & Drop Policy Typically <5Tb/s aggregate capacity