1. Shanghai Jiao Tong University 2012 Indirect Networks or Dynamic Networks Guihai Chen …with major presentation contribution from José Flich, UPV (and Cell BE EIB slides by Tom Ainsworth, USC) Interconnection Networks

2. Shanghai Jiao Tong University 2012 2 Questions in mind Difference between Static/direct and Dynamic/indirect Networks Why Multi-Stage Interconnection Networks Large Switch and Small Switch How to design non-blocking MINs

3. Shanghai Jiao Tong University 2012 3 Outline Network Topology Preliminaries and Evolution Centralized Switched (Indirect) Networks From non-blocking crossbar to blocking MINs From blocking MINs to non-blocking MINs

4. Shanghai Jiao Tong University 2012 4 Network Topology Preliminaries and Evolution One switch suffices to connect a small number of devices –Number of switch ports limited by VLSI technology, power consumption, packaging, and other such cost constraints A fabric of interconnected switches (i.e., switch fabric or network fabric) is needed when the number of devices is much larger –The topology must make a path(s) available for every pair of devices—property of connectedness or full access (What paths?) Topology defines the connection structure across all components –Bisection bandwidth: the minimum bandwidth of all links crossing a network split into two roughly equal halves –Full bisection bandwidth: ›Network BW Bisection = Injection (or Reception) BW Bisection = N/2 –Bisection bandwidth mainly affects performance Topology is constrained primarily by local chip/board pin-outs; secondarily, (if at all) by global bisection bandwidth

5. Shanghai Jiao Tong University 2012 5 Network Topology Preliminaries and Evolution Several tens of topologies proposed, but less than a dozen used 1970s and 1980s –Topologies were proposed to reduce hop count 1990s –Pipelined transmission and switching techniques –Packet latency became decoupled from hop count 2000s –Topology still important (especially OCNs, SANs, P2P Overlays, DCNs) when N is high –Topology impacts performance and has a major impact on cost

6. Shanghai Jiao Tong University 2012 6 Network Topology Centralized Switched (Indirect) Networks Crossbar network –Crosspoint switch complexity increases quadratically with the number of crossbar input/output ports, N, i.e., grows as O(N 2 ) –Has the property of being non-blocking 7 6 5 4 3 2 1 0 7 6 54 3210 7 6 5 4 3 2 1 0 7 6 54 3210

7. Shanghai Jiao Tong University 2012 7 Network Topology From Crossbar to MINs

8. Shanghai Jiao Tong University 2012 8 Network Topology Centralized Switched (Indirect) Networks Multistage interconnection networks (MINs) –Crossbar split into several stages consisting of smaller crossbars –Complexity grows as O(N × log N), where N is # of end nodes –Inter-stage connections represented by a set of permutation functions Omega topology, perfect-shuffle exchange 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

9. Shanghai Jiao Tong University 2012 9 Network Topology Appendix Shuffle function –N= 0…N-1 –f(i)= 2i, when I <N/2 – f(i)=(2i+1), mod N when i ≥ N/2 –Often used as a connection pattern –unshuffle function, also often used perfect-shuffle 7 6 5 4 3 2 1 0 unshuffle 7 6 5 4 3 2 1 0

10. Shanghai Jiao Tong University 2012 10 Network Topology Centralized Switched (Indirect) Networks 16 port, 4 stage Omega network 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111

11. Shanghai Jiao Tong University 2012 11 Network Topology Centralized Switched (Indirect) Networks 16 port, 4 stage Baseline network 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111

12. Shanghai Jiao Tong University 2012 12 Network Topology Centralized Switched (Indirect) Networks 16 port, 4 stage Butterfly network 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111

13. Shanghai Jiao Tong University 2012 13 Network Topology- Correction to Butterfly Centralized Switched (Indirect) Networks 16 port, 4 stage Butterfly network 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111

14. Shanghai Jiao Tong University 2012 14 Network Topology Centralized Switched (Indirect) Networks 16 port, 4 stage Cube network 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111

15. Shanghai Jiao Tong University 2012 15 Network Topology Centralized Switched (Indirect) Networks Multistage interconnection networks (MINs) –MINs interconnect N input/output ports using k x k switches ›log k N switch stages, each with N/k switches ›N/k(log k N) total number of switches –Example: Compute the switch and link costs of interconnecting 4096 nodes using a crossbar relative to a MIN, assuming that switch cost grows quadratically with the number of input/output ports (k). Consider the following values of k: ›MIN with 2 x 2 switches ›MIN with 4 x 4 switches ›MIN with 16 x 16 switches

16. Shanghai Jiao Tong University 2012 16 Network Topology Centralized Switched (Indirect) Networks Example: compute the relative switch and link costs, N = 4096 relative_cost(2 × 2) switches = 4096 2 / (2 2 × 4096/2 × log 2 4096) = 170 relative_cost(4 × 4) switches = 4096 2 / (4 2 × 4096/4 × log 4 4096) = 170 relative_cost(16 × 16) switches = 4096 2 / (16 2 × 4096/16 × log 16 4096) = 85 relative_cost(2 × 2) links = 8192 / (4096 × (log 2 4096 + 1)) = 2/13 = 0.1538 relative_cost(4 × 4) links = 8192 / (4096 × (log 4 4096 + 1)) = 2/7 = 0.2857 relative_cost(16 × 16) links = 8192 / (4096 × (log 16 4096 + 1)) = 2/4 = 0.5 cost(crossbar) switches = 4096 2 cost(crossbar) links = 8192

17. Shanghai Jiao Tong University 2012 17 Network Topology Relative link costRelative switch cost Centralized Switched (Indirect) Networks Relative switch and link costs for various values of k and N (crossbar relative to a MIN)

18. Shanghai Jiao Tong University 2012 18 Network Topology From blocking to non-blocking again

19. Shanghai Jiao Tong University 2012 19 Network Topology Centralized Switched (Indirect) Networks Reduction in MIN switch cost comes at the price of performance –Network has the property of being blocking –Contention is more likely to occur on network links ›Paths from different sources to different destinations share one or more links blocking topology X non-blocking topology 7 6 5 4 3 2 1 0 7 6 54 3210 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0

20. Shanghai Jiao Tong University 2012 20 Network Topology Centralized Switched (Indirect) Networks How to reduce blocking in MINs? Provide alternative paths! –Use larger switches (can equate to using more switches) ›Clos network: minimally three stages (non-blocking) »A larger switch in the middle of two other switch stages provides enough alternative paths to avoid all conflicts –Use more switches ›Add log k N - 1 stages, mirroring the original topology »Rearrangeably non-blocking »Allows for non-conflicting paths »Doubles network hop count (distance), d »Centralized control can rearrange established paths ›Benes topology: 2(log 2 N) - 1 stages (rearrangeably non-blocking) »Recursively applies the three-stage Clos network concept to the middle-stage set of switches to reduce all switches to 2 x 2

21. Shanghai Jiao Tong University 2012 21 Network Topology Centralized Switched (Indirect) Networks 16 port Crossbar network 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8

22. Shanghai Jiao Tong University 2012 22 Network Topology Centralized Switched (Indirect) Networks 16 port, 3-stage Clos network 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8

23. Shanghai Jiao Tong University 2012 23 Network Topology Centralized Switched (Indirect) Networks 16 port, 5-stage Clos network 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8

24. Shanghai Jiao Tong University 2012 24 Network Topology Centralized Switched (Indirect) Networks 16 port, 7 stage Clos network = Benes topology 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8

25. Shanghai Jiao Tong University 2012 25 Network Topology Centralized Switched (Indirect) Networks Alternative paths from 0 to 1. 16 port, 7 stage Clos network = Benes topology 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8

26. Shanghai Jiao Tong University 2012 26 Network Topology Centralized Switched (Indirect) Networks 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 Alternative paths from 4 to 0. 16 port, 7 stage Clos network = Benes topology

27. Shanghai Jiao Tong University 2012 27 Network Topology Centralized Switched (Indirect) Networks 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 Contention free, paths 0 to 1 and 4 to 1. 16 port, 7 stage Clos network = Benes topology

28. Shanghai Jiao Tong University 2012 28 Network Topology Bidirectional MINs Increase modularity Reduce hop count, d Fat tree network –Nodes at tree leaves –Switches at tree vertices –Total link bandwidth is constant across all tree levels, with full bisection bandwidth –Equivalent to folded Benes topology –Preferred topology in many SANs Folded Clos = Folded Benes = Fat tree network Centralized Switched (Indirect) Networks Network Bisection

29. Shanghai Jiao Tong University 2012 29 Network Topology Myrinet-2000 Clos Network for 128 Hosts Backplane of the M3- E128 Switch M3-SW16-8F fiber line card (8 ports) http://myri.com

30. Shanghai Jiao Tong University 2012 30 Network Topology Myrinet-2000 Clos Network for 128 Hosts “Network in a Box” 16 fiber line cards connected to the M3-E128 Switch backplane http://myri.com

31. Shanghai Jiao Tong University 2012 31 Network Topology Myrinet-2000 Clos Network Extended to 512 Hosts http://myri.com

32. Shanghai Jiao Tong University 2012 Assignment 2-2 Chose one of the following exercises: Calculate how many permutations nxn Omega network could support. Prove that folded Clos network, folded Bens network, and Fat tree network are isomorphic to each other. SJTU@Fall 2012Parallel Processing, Low-Diameter Architectures Slide 32