1. 1 EE384Y: Packet Switch Architectures Part II Load-balanced Switches Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford University nickm@stanford.edu http://www.stanford.edu/~nickm

2. 2 The Arbitration Problem  A packet switch fabric is reconfigured for every packet transfer.  For example, at 160Gb/s, a new IP packet can arrive every 2ns.  The configuration is picked to maximize throughput and not waste capacity.  Known algorithms are probably too slow.

3. 3 Approach  We know that a crossbar with VOQs, and uniform Bernoulli i.i.d. arrivals, gives 100% throughput for the following scheduling algorithms:  Pick a permutation uar from all permutations.  Pick a permutation uar from the set of size N in which each input- output pair (i,j) are connected exactly once in the set.  From the same set as above, repeatedly cycle through a fixed sequence of N different permutations.  Can we make non-uniform, bursty traffic uniform “enough” for the above to hold?

4. 4 Design Example Goals Scale to High Linecard Speeds (160Gb/s)  No Centralized Scheduler  Optical Switch Fabric  Low Packet-Processing Complexity Scale to High Number of Linecards (640) Provide Performance Guarantees  100% Throughput Guarantee  No Packet Reordering Stanford “Optics in Routers” project http://yuba.stanford.edu/or/  Some challenging numbers:  100Tb/s  160Gb/s linecards  640 linecards

5. 5 Outline  Basic idea of load-balancing  Packet mis-sequencing  An optical switch fabric  Scaling number of linecards

6. 6 In Out R R R R R R Router capacity = NR Switch capacity = N 2 R 100% Throughput in a Mesh Fabric ? ? ? ? ? ? ? ? ? R R R R R R R R R R R R R

7. 7 R In Out R R R R R R/N If Traffic Is Uniform R R

8. 8 Real Traffic is Not Uniform R In Out R R R R R R/N R R R R R R R R R ?

9. 9 Out R R R R/N Load-Balanced Switch Load-balancing stageForwarding stage In Out R R R R/N R R R 100% throughput for weakly mixing traffic (Valiant, C.-S. Chang)

10. 10 Out R R R R/N In R R R R/N 1 1 2 2 3 3 Load-Balanced Switch

11. 11 Out R R R R/N In R R R R/N 3 3 2 2 1 1 Load-Balanced Switch

12. 12 Out R R R R/N In R R R R/N Intuition: 100% Throughput  Arrivals to second mesh:  Capacity of second mesh:  Second mesh: arrival rate < service rate [C.-S. Chang]

13. 13 Another way of thinking about it 1 N 1 N 1 NExternal Outputs Internal Inputs External Inputs Load-balancing cyclic shift Switching cyclic shift Load Balancing  First stage load-balances incoming packets  Second stage is a cyclic shift

14. 14 Load-Balanced Switch External Outputs Internal Inputs 1 N External Inputs Load-balancing cyclic shift Switching cyclic shift 1 N 1 N 1 1 2 2

15. 15

16. 16 Outline of Chang’s Proof

17. 17 Outline  Basic idea of load-balancing  Packet mis-sequencing  An optical switch fabric  Scaling number of linecards

18. 18 Out R R R R/N In R R R R/N Packet Reordering 1 2

19. 19 Out R R R R/N In R R R R/N Bounding Delay Difference Between Middle Ports 1 2

20. 20 Out R R R R/N In R R R R/N 1 2 3 UFS (Uniform Frame Spreading) 1 2

21. 21 Out R R R R/N In R R R R/N FOFF (Full Ordered Frames First) 1 2

22. 22 FOFF (Full Ordered Frames First)  Input Algorithm  N FIFO queues corresponding to the N output flows  Spread each flow uniformly: if last packet was sent to middle port k, send next to k+1.  Every N time-slots, pick a flow: - If full frame exists, pick it and spread like UFS - Else if all frames are partial, pick one in round-robin order and send it 12 3 1 2 4 N

23. 23 Out R R R R/N In R R R R/N Bounding Reordering 1 2 3

24. 24 FOFF  Output properties  N FIFO queues corresponding to the N middle ports  Buffer size less than N 2 packets  If there are N 2 packets, one of the head-of-line packets is in order 11 1 2 2 3 3 3 Output 4 N

25. 25 FOFF Properties  Property 1: FOFF maintains packet order.  Property 2: FOFF has O(1) complexity.  Property 3: Congestion buffers operate independently.  Property 4: FOFF maintains an average packet delay within constant from ideal output-queued router.  Corollary: FOFF has 100% throughput for any adversarial traffic.

26. 26 In Out R R R R R R Output-Queued Router ? ? ? ? ? ? ? ? ? R R R R R R R R R R R R R

27. 27 Outline  Basic idea of load-balancing  Packet mis-sequencing  An optical switch fabric  Scaling number of linecards

28. 28 Out R R R R/N In R R R R/N From Two Meshes to One Mesh One linecard In Out

29. 29 From Two Meshes to One Mesh First mesh In Out In Out In Out In Out One linecard Second mesh R R R R R

30. 30 From Two Meshes to One Mesh Combined mesh In Out In Out In Out In Out 2R R

31. 31 Many Fabric Options Options Space: Full uniform mesh Time: Round-robin crossbar Wavelength: Static WDM Any spreading device C 1, C 2, …, C N C1C1 C2C2 C3C3 CNCN In Out In Out In Out In Out N channels each at rate 2R/N One linecard

32. 32 AWGR (Arrayed Waveguide Grating Router) A Passive Optical Component  Wavelength i on input port j goes to output port (i+j-1) mod N  Can shuffle information from different inputs  1,  2 …  N NxN AWGR Linecard 1 Linecard 2 Linecard N  1  2  N Linecard 1 Linecard 2 Linecard N

33. 33 In Out In Out In Out In Out Static WDM Switching: Packaging AWGR Passive and Almost Zero Power A B C D A, B, C, D A, A, A, A B, B, B, B C, C, C, C D, D, D, D N WDM channels, each at rate 2R/N

34. 34 Outline  Basic idea of load-balancing  Packet mis-sequencing  An optical switch fabric  Scaling number of linecards

35. 35 Scaling Problem  For N < 64, an AWGR is a good solution.  We want N = 640.  Need to decompose.

36. 36 A Different Representation of the Mesh In Out In Out In Out In Out R 2R Mesh 2R In Out In Out In Out In Out R 2R R

37. 37 A Different Representation of the Mesh In Out In Out In Out In Out R In Out In Out In Out In Out R 2R/N

38. 38 1 2 3 4 Example: N=8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 2R/8

39. 39 When N is Too Large Decompose into groups (or racks) 4R/4 2R2R2R2R 1 2 3 4 5 6 7 8 2R2R 2R2R 1 2 3 4 5 6 7 8 4R

40. 40 When N is Too Large Decompose into groups (or racks) 12L 2R 12L Group/Rack 1 Group/Rack G 12L 2R Group/Rack 1 12L 2R Group/Rack G 2RL 2RL/G

41. 41 Outline  Basic idea of load-balancing  Packet mis-sequencing  An optical switch fabric  Scaling number of linecards