1. 048866: Packet Switch Architectures Dr. Isaac Keslassy Electrical Engineering, Technion isaac@ee.technion.ac.il http://comnet.technion.ac.il/~isaac/ Introduction

2. Spring 2006048866 – Packet Switch Architectures2 Course Content  Most of the results in the course are less than 10 years old!  Material is new and still changing…  No reference book available yet…  Uses slides by Profs. Nick McKeown and Balaji Prabahakar (Stanford)

3. Spring 2006048866 – Packet Switch Architectures3 Outline  Background  What is a router?  Why do we need faster routers?  Why are they hard to build?  Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

4. Spring 2006048866 – Packet Switch Architectures4 What is Routing? R3 A B C R1 R2 R4D E F R5 F R3E D Next HopDestination D

5. Spring 2006048866 – Packet Switch Architectures5 What is Routing? R3 A B C R1 R2 R4D E F R5 F R3E D Next HopDestination D 16 32 41 Data Options (if any) Destination Address Source Address Header ChecksumProtocolTTL Fragment Offset Flags Fragment ID Total Packet LengthT.ServiceHLenVer 20 bytes

6. Spring 2006048866 – Packet Switch Architectures6 What is Routing? A B C R1 R2 R3 R4D E F R5

7. Spring 2006048866 – Packet Switch Architectures7 Points of Presence (POPs) A B C POP1 POP3 POP2 POP4 D E F POP5 POP6 POP7 POP8

8. Spring 2006048866 – Packet Switch Architectures8 Where High Performance Routers are Used R10 R11 R4 R13 R9 R5 R2 R1 R6 R3 R7 R12 R16 R15 R14 R8 (10 Gb/s)

9. Spring 2006048866 – Packet Switch Architectures9 What a Router Looks Like Cisco GSR 12416Juniper M160 6ft 19 ” 2ft Capacity: 160Gb/s Power: 4.2kW Full rack 3ft 2.5ft 19 ” Capacity: 80Gb/s Power: 2.6kW Half-a-rack

10. Spring 2006048866 – Packet Switch Architectures10 Core Router Market  Estimates cover different parameters  Dell’Oro (Feb. 17, 2005):  $1.2 billion in 2004 (up 66%)  Includes high-end (10Gbps) router market  Dell’Oro (Feb. 21, 2006):  $4.4 billion in 2005  Includes “worldwide service provider router market”  Infonetics (Feb. 21, 2006):  $5.2 billion in 2005 (up 31%)  Includes core/edge routers  Core router market  Relatively small (in comparison to enterprise market), but brings expertise to company  Vendors: Cisco, Juniper, Avici, Nortel, Lucent, Alcatel, Huawei, etc.

11. Spring 2006048866 – Packet Switch Architectures11 Router Market Source: Infonetics (Nov. 2004)

12. Spring 2006048866 – Packet Switch Architectures12 Basic Architectural Components of an IP Router Control Plane Datapath per-packet processing Switching Forwarding Table Routing Table Routing Protocols

13. Spring 2006048866 – Packet Switch Architectures13 Per-packet processing in an IP Router 1. Accept packet arriving on an incoming link. 2. Lookup: lookup packet destination address in the forwarding table, to identify outgoing port(s). 3. Header Processing: Manipulate packet header: e.g., decrement TTL, update header checksum. 4. Switching: Send packet to the outgoing port(s). 5. Buffering: Buffer packet in the queue. 6. Transmit packet onto outgoing link.

14. Spring 2006048866 – Packet Switch Architectures14 Generic Router Architecture Lookup IP Address Update Header Header Processing DataHdrDataHdr ~1M prefixes Off-chip DRAM Address Table Address Table IP AddressNext Hop Queue Packet Buffer Memory Buffer Memory ~1M packets Off-chip DRAM

15. Spring 2006048866 – Packet Switch Architectures15 Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table DataHdrDataHdrDataHdr Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory DataHdrDataHdrDataHdr

16. Spring 2006048866 – Packet Switch Architectures16 Outline  Background  What is a router?  Why do we need faster routers?  Why are they hard to build?  Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

17. Spring 2006048866 – Packet Switch Architectures17 Why Do We Need Faster Routers? 1. To prevent routers from becoming the bottleneck in the Internet. 2. To increase POP capacity, and to reduce cost, size and power.

18. Spring 2006048866 – Packet Switch Architectures18 Why We Need Faster Routers 1: To prevent routers from being the bottleneck Packet Processing Power 2x / 18 months Source: SPEC95Int & Coffman and Odlyzko. Single Fiber Capacity (commercial) ≥ 2x / year

19. Spring 2006048866 – Packet Switch Architectures19 POP with smaller routers Why we Need Faster Routers 2: To reduce cost, power & complexity of POPs POP with large routers  Ports: Price >$50k, Power > 400W.  It is common for 50-60% of ports to be for interconnection.

20. Spring 2006048866 – Packet Switch Architectures20 Why are Fast Routers Difficult to Make? 1. It’s hard to keep up with Moore’s Law:  The bottleneck is memory speed.  Memory speed is not keeping up with Moore’s Law.

21. Spring 2006048866 – Packet Switch Architectures21 Why are Fast Routers Difficult to Make? Speed of Commercial DRAM 1. It’s hard to keep up with Moore’s Law:  The bottleneck is memory speed.  Memory speed is not keeping up with Moore’s Law. Moore’s Law 2x / 18 months 1.1x / 18 months

22. Spring 2006048866 – Packet Switch Architectures22 Why are Fast Routers Difficult to Make? 1. It’s hard to keep up with Moore’s Law:  The bottleneck is memory speed.  Memory speed is not keeping up with Moore’s Law. 2. Moore’s Law is too slow:  Routers need to improve faster than Moore’s Law.

23. Spring 2006048866 – Packet Switch Architectures23 Router Performance Exceeds Moore’s Law Growth in capacity of commercial routers:  Capacity 1992 ~ 2Gb/s  Capacity 1995 ~ 10Gb/s  Capacity 1998 ~ 40Gb/s  Capacity 2001 ~ 160Gb/s  Capacity 2003 ~ 640Gb/s Average growth rate: 2.2x / 18 months.

24. Spring 2006048866 – Packet Switch Architectures24 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

25. Spring 2006048866 – Packet Switch Architectures25 Route Table CPU Buffer Memory Line Interface MAC Line Interface MAC Line Interface MAC Typically <0.5Gb/s aggregate capacity Shared Backplane Line Interface CPU Memory First-Generation Routers

26. Spring 2006048866 – Packet Switch Architectures26 Route Table CPU Line Card Buffer Memory Line Card MAC Buffer Memory Line Card MAC Buffer Memory Fwding Cache Fwding Cache Fwding Cache MAC Buffer Memory Typically <5Gb/s aggregate capacity Second-Generation Routers

27. Spring 2006048866 – Packet Switch Architectures27 Line Card MAC Local Buffer Memory CPU Card Line Card MAC Local Buffer Memory Switched Backplane Line Interface CPU Memory Fwding Table Routing Table Fwding Table Typically <50Gb/s aggregate capacity Third-Generation Routers

28. Spring 2006048866 – Packet Switch Architectures28 Switch Core Linecards Optical links 100s of metres 0.3 - 10Tb/s routers Fourth-Generation Routers Multi-Racks, Optical Links

29. Spring 2006048866 – Packet Switch Architectures29 Optical Switch Core Linecards Optical links 100s of metres 10-100Tb/s routers, in project (Future) Fifth-Generation Routers Optical Switch Core

30. Spring 2006048866 – Packet Switch Architectures30 Optical Switch Core Optical Linecards Optical links 100s of metres 100-1000Tb/s routers, in the far future (Future) Sixth-Generation Routers All-Optical Routers

31. Spring 2006048866 – Packet Switch Architectures31 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

32. Spring 2006048866 – Packet Switch Architectures32 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Lookup IP Address Address Table Address Table Lookup IP Address Address Table Address Table Lookup IP Address Address Table Address Table

33. Spring 2006048866 – Packet Switch Architectures33 IP Address Lookup  Why it’s thought to be hard:  It’s not an exact match: it’s a longest prefix match.  The table is large: about 150,000 entries today, and growing.  The lookup must be fast: about 30ns for a 10Gb/s line.

34. Spring 2006048866 – Packet Switch Architectures34 IP Lookups find Longest Prefixes 128.9.16.0/21128.9.172.0/21 128.9.176.0/24 0 2 32 -1 128.9.0.0/16 142.12.0.0/19 65.0.0.0/8 128.9.16.14 Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.

35. Spring 2006048866 – Packet Switch Architectures35 IP Addressing Classless Interdomain Routing (CIDR)  The IP address space is broken into line segments.  Each line segment is described by a prefix.  A prefix is of the form x/y where x indicates the prefix of all addresses in the line segment, and y indicates the length of the segment.  e.g. The prefix 128.9/16 represents the line segment containing addresses in the range: 128.9.0.0 … 128.9.255.255. 0 2 32 -1 128.9/16 128.9.0.0 2 16 65/8 128.9.16.14

36. Spring 2006048866 – Packet Switch Architectures36 IP Addressing Classless Interdomain Routing (CIDR) 0 2 32 -1 128.9/16 128.9.0.0 2 16 65/8 128.9.16.14 128=2^7 9=2^3+2^0 16=2^4 14=2^3+2^2+2^1 128.9/16 =1000000.00001001.xxxxxxxx.xxxxxxxx  128.9.16.14 = 10000000.00001001.00010000.00001110 76543210

37. Spring 2006048866 – Packet Switch Architectures37 IP Address Lookup  Why it’s thought to be hard:  It’s not an exact match: it’s a longest prefix match.  The table is large: about 150,000 entries today, and growing.  The lookup must be fast: about 30ns for a 10Gb/s line.

38. Spring 2006048866 – Packet Switch Architectures38 Address Tables are Large Source: http://www.cidr-report.org/

39. Spring 2006048866 – Packet Switch Architectures39 IP Address Lookup  Why it’s thought to be hard:  It’s not an exact match: it’s a longest prefix match.  The table is large: about 150,000 entries today, and growing.  The lookup must be fast: about 30ns for a 10Gb/s line.

40. Spring 2006048866 – Packet Switch Architectures40 Lookups Must be Fast 12540Gb/s2003 31.2510Gb/s2001 7.812.5Gb/s1999 1.94622Mb/s1997 40B packets (Mpkt/s) LineYear 500160Gb/s2005?

41. Spring 2006048866 – Packet Switch Architectures41 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

42. Spring 2006048866 – Packet Switch Architectures42 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory Buffer Manager Buffer Memory Buffer Memory

43. Spring 2006048866 – Packet Switch Architectures43 Fast Packet Buffers Example: 40Gb/s packet buffer Size = RTT*BW = 10Gb; 40 byte packets Write Rate, R 1 packet every 8 ns Read Rate, R 1 packet every 8 ns Buffer Manager Buffer Memory Use SRAM? + fast enough random access time, but - too low density to store 10Gb of data. Use SRAM? + fast enough random access time, but - too low density to store 10Gb of data. Use DRAM? + high density means we can store data, but - too slow (50ns random access time). Use DRAM? + high density means we can store data, but - too slow (50ns random access time).

44. Spring 2006048866 – Packet Switch Architectures44 Outline Background  What is a router?  Why do we need faster routers?  Why are they hard to build? Architectures and techniques  The evolution of router architecture.  IP address lookup.  Packet buffering.  Switching.

45. Spring 2006048866 – Packet Switch Architectures45 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory DataHdr DataHdr DataHdr 1 2 N 1 2 N N times line rate

46. Spring 2006048866 – Packet Switch Architectures46 Generic Router Architecture Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Lookup IP Address Update Header Header Processing Address Table Address Table Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory Queue Packet Buffer Memory Buffer Memory DataHdr DataHdr DataHdr 1 2 N 1 2 N DataHdr DataHdr DataHdr Scheduler

47. Spring 2006048866 – Packet Switch Architectures47 Current Internet Router Technology Summary  There are three potential bottlenecks:  Address lookup,  Packet buffering, and  Switching. Most difficult – and focus of this course!