1. NC 論２ (No.2) 1 Indirect (dynamic) Networks Communication between any two nodes has to be carried through some switches. Classified into: –Crossbar network –Multistage interconnection networks (MIN) Nonblocking networks Rearrangeable nonblocking networks Blocking networks –Single-stage interconnection networks

2. NC 論２ (No.2) 2 Crossbar (nonblocking, 1-hop) An N x M crossbar The cost increases as N x M. Switch point (crosspoint) M outputs N inputs N:1 multiplexer

3. NC 論２ (No.2) 3 A large crossbar 0 1 2 3

4. NC 論２ (No.2) 4 MINs Three-stage MIN first stagesecond stage third stage N inputs M outputs

5. NC 論２ (No.2) 5 Nonblocking MIN Any input port can be connected to any free output port without affecting the existing connections. Same functionality as a crossbar. The Clos network is proposed to reduce the cost of crossbar by decreasing the number of switches (1953).

6. NC 論２ (No.2) 6 History (telephone net.) Early telephone networks were built from elector-mechanical crossbars. Key developments in telephone switching include the Clos network in 1953 and the Benes network in 1962. Many large telephone switches today are still built from Clos or Clos-like networks.

7. NC 論２ (No.2) 7 3-stage Clos network (m,n,r) Nonblocking: m >= 2n-1 Rearrangeable: 2n-1 > m >= n N inputs N outputs 1 2 r 1 n 1 n 1 n 1 2 m 1 2 r 1 n 1 n 1 n n×mr×rm×n

8. NC 論２ (No.2) 8 Rearrangeable nonblocking MIN Any input port can be connected to any free output port with rearrangement of paths for the existing connections. Less stages or smaller switches than a nonblocking networks. The best known example is the Benes network.

9. NC 論２ (No.2) 9 Benes network An 8x8 Benes network (Eσ ３ －１ E σ ２ － 1 Eσ ２ Eσ ３ E) N(2log 2 N - 1) / 2 switches, 2log 2 N - 1 hops, E: stage 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 σ ３ －１ σ ２ －１ σ２σ２ σ３σ３ 010 001 (2,2,2) clos (2,4,2) Clos

10. NC 論２ (No.2) 10 2x2 switch (radix: 2) straightexchangelower broadcast upper broadcast

11. NC 論２ (No.2) 11 Permutation Permutations are traffic patterns in which all traffic from each source is directed to one destination. It can be compactly represented by a permutation function that maps source to destination. Bit (digit) permutation are those in which each bit (digit) of the destination address is a function of one bit (digit) of the source address. –d i = s j or d x = f(s x )

12. NC 論２ (No.2) 12 Permutation functions (1/2) Communication from node address x(a n,a n-1,a n-2,…, a 1 ) to Perfect shuffle: σ(x)=(a n-1,a n-2,…, a 1,a n ) k sub-shuffule: σ k (x)=(a n,a n-1,…, a k+1,a k-1, …, a 1,a k ) k supershuffule: σ k (x)=(a n-1,a n-2,…, a n-k+1, a n, a n-k, …, a 1 ) Inverse perfect shuffle: σ -1 (x)=(a 1, a n, a n-1,a n-2,…, a 2 ) k sub-inverse shuffule: σ k -1 (x)=(a n, …, a k+1, a 1,a k,…, a 2 ) Butterfly: β(x)=(a 1, a n-1,a n-2,…, a 2, a n ) k sub-butterfly: β k (x)=(a n,…, a k+1, a 1,a k-1, …, a 2,a k ) k superbutterfly: β k (x)=(a n-k+1, a n-1,…, a n-k+2, a n,a n-k, …, a 1 )

13. NC 論２ (No.2) 13 Permutation functions (2/2) Communication from node address x(a n,a n-1,a n-2,…, a 1 ) to Bit reversal: ρ(x)=(a 1,a 2,…, a n-1, a n ) k sub-bit reversal : ρ k (x)=(a n,a n-1,…, a k+1,a 1, a 2,…, a k-1,a k ) k super-bit reversal: ρ k (x)=(a n-k+1,a n-k+2,…, a n-1, a n, a n-k, …, a 1 ) shift: α(x)= | x+1| (2 n ), where modulo by 2 n k sub-shift: α k (x)= | x+1| (2 n ) +floor(x/2 k )2 k k super-shift: α k (x)= | x+ 2 n-k | (2 n ) Exchange: E i (x)=(a n,a n-1, …, ~a i,…, a 1 )

14. NC 論２ (No.2) 14 Perfect shuffle (1/2) 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 Perfect shuffle σ(x)=(a n-1,a n-2,…, a 1,a n ) Inverse perfect shuffle σ -1 (x)=(a 1,a n a n-1,a n-2,…, a 2 )

15. NC 論２ (No.2) 15 Perfect shuffle (2/2) 000 001 010 011 100 101 110 111 000 001 010 011 100 101101 110 111 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 3 sub-inverse shuffule σ 3 -1 (x)=(a 1, a 3, a 2 ) 2 sub-inverse shuffle σ 2 -1 (x)=(a 3, a 1, a 2 )

16. NC 論２ (No.2) 16 Butterfly 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 000 001 010 011 100100 101 110 111 000 001001 010 011 100 101 110 111 butterfly (bit reversal) β 3 (x)=(a 1, a 2, a 3 ) ＝ ρ ３ (x) 2 sub-butterfly β 2 (x)=(a 3, a 1, a 2 )

17. NC 論２ (No.2) 17 shift 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 shift α(x)= | x+1 | (2 3 ) 2 sub-shift α 2 (x)=| x+1 | (2 2 )+floor(x/ 2 2 ) 2 2

18. NC 論２ (No.2) 18 exchange 000 001001 010 011 100 101 110 111 000 001 010 011011 100 101 110 111 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 E 3 (x) = (~a 3, a 2, a 1 ) E 2 (x) = (a 3, ~a 2, a 1 )

19. NC 論２ (No.2) 19 Blocking MIN A free input/output pair is not always possible because of conflicts with the existing connections. Minimizing the number of switches and stages. Unidirectional MIN (Cenju-3, IBM PR3) Bidirectional MIN (IBM SP, TMC CM5)

20. NC 論２ (No.2) 20 Unidirectional MINs Channels and switches are unidirectional. At least ceiling(log k N) stages are required for a MIN of N input and output ports with k×k switches. Every path through the MIN crosses all the stages (same length). Examples: baseline, butterfly, indirect cube logN stages for N inputs/outputs with 2x2 switches ( binary logN-fly in the form of k-ary n-fly) the number of communication patterns are (√N) N

21. NC 論２ (No.2) 21 Baseline networks (1/2) (B = Eσ n －１ Eσ n-1 －１ E ・・・ Eσ ２ － 1 E) 000 001 010 011 100 101 110 111 000 001 (UUL) 010 (ULU) 011 100 101 110 111 abccabcbacba σ２－1σ２－1 σ３－1σ３－1 001001

22. NC 論２ (No.2) 22 Baseline networks (2/2) To route (a n,a n-1,a n-2,…, a 1 ) to (b n,b n-1,b n-2,…, b 1 ), select upper link when bi=0, otherwise select lower link on (n-i+1) stage (MSB to LSB). (path is determined only by the destination address) switch upper link lower link inputs

23. NC 論２ (No.2) 23 Omega networks (1/2) Ω= (σE) n 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 σσσ abc bcabca cabcab

24. NC 論２ (No.2) 24 Omega networks (2/2) To route (a n,a n-1,a n-2,…, a 1 ) to (b n,b n-1,b n-2,…, b 1 ), set the switch as straight when ai^bi=0, otherwise set it as exchange on (n-i+1) stage (MSB to LSB). straightexchange

25. NC 論２ (No.2) 25 Indirect binary n-cube (1/2) (C = Eβ ２ Eβ ３ ・・・ E β ｎ Eσ － 1 ) 000000 001001 010010 011011 100100 101101 110110 111111 000 001 010 011 100 101 110 111 σ -1 β３β３ β２β２ 000 001 100 101 010 110 011 111 abcacbabc bcabca

26. NC 論２ (No.2) 26 Indirect binary n-cube (2/2) To route (a n,a n-1,a n-2,…, a 1 ) to (b n,b n-1,b n-2,…, b 1 ), set the switch as straight when ai^bi=0, otherwise set it as exchange on i-th stage (LSB to MSB). A k-ary n-fly network consists of k n source/destination terminals, n stages of k n-1 k×k crossbar switches. –Degree of the switches: 2k –Diameter: n+1 or log k N +1 where N = k n

27. NC 論２ (No.2) 27 Relation among blocking networks For omega, indirect n-cube and Banyan(Y): Ω -1 =C=Yσ So they are almost functionally equal (isomorphic). The communication patterns are different, but same number of combinations ( (√N) N ). Smaller number of communication patterns than nonblocking networks (N!).

28. NC 論２ (No.2) 28 Bitonic sort 8 inputs sorter4 inputs sorter2 inputs sorter A L B H

29. NC 論２ (No.2) 29 Batcher-banyan network Bitonic Sorter (Batcher stage) Ω (banyan stage) Non- blocking 5∞31∞･･∞5∞31∞･･∞ 135∞∞･･∞135∞∞･･∞ 135135 receiverOrdered addresses

30. NC 論２ (No.2) 30 Tree saturation (e.g. on Omega) 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 Combining: combine two messages hotspot

31. NC 論２ (No.2) 31 Fault-tolerant MINs Blocking MINs provide a single path between any input/output pair. If a link becomes congested or fails, the unique path property can easily disrupt the communication (no path diversity). A solution is to provide multiple routing paths by adding extra stages or channels.

32. NC 論２ (No.2) 32 Fault-tolerant MINs (Omega’s case) 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 σσσ Additional stage

33. NC 論２ (No.2) 33 Fault-tolerant MINs (Baseline’s case) 000 001 010 011 100 101 110 111 000 001 010 011 100 101 110 111 σ２－1σ２－1 σ３－1σ３－1

34. NC 論２ (No.2) 34 Bidirectional MINs (BMINs) Channels and switches are bidirectional. Information can be transmitted simultaneously in opposite directions between neighboring switches. Paths are established in BMINs by crossing stages in forward, then turnaround, and finally backward. Several paths are possible in forward direction, but a single path is available in backward direction.

35. NC 論２ (No.2) 35 Bidirectional Banyan networks 000 001 010 011 100 101 110 111 β３β３ β２β２ 000 001 100 101 010 110 011 111 000000 001001 100100 101101 010010 110110 011011 111111 forward backward turnaround

36. NC 論２ (No.2) 36 A fat tree BMIN forward backward

37. NC 論２ (No.2) 37 Hybrid networks Combine mechanisms from shared-medium networks and direct or indirect networks. Increase bandwidth and reduce the distance between nodes. Examples: –Multiple backplane buses –Hierarchical buses –Cluster-based networks

38. NC 論２ (No.2) 38 Multiple backplane buses device bus

39. NC 論２ (No.2) 39 Hierarchical buses Global bus Cluster bus bridge

40. NC 論２ (No.2) 40 Cluster-based networks Cluster bus Cluster router

41. NC 論２ (No.2) 41 History (Clos networks, etc.) Clos introduced non-blocking networks in1953. Benes discovered rearrangeable non-blocking networks in 1965. Bachter introduced the bitonic sort and the Batcher sorting network in 1968. Knuth’s book in 1998 treats sorting networks in detail. Multicast in Clos networks was first considered by Masson and Jordan(1972), and has since been studied by many researchers.

42. NC 論２ (No.2) 42 History (indirect networks) Processor-memory networks emerged in the late 1960s in the form of indirect netwoks. The smallest machine employed crossbar, whereas larger sytems used butterfly, etc.. Since the early 1990s, a variant of the Clos and Benes networks of telephony has emerged in SMP (MPP) networks in the form of the fat tree. Today, the high pin bandwidth of router chips relative to message length motivates the use of networks of higher node degree, such as Clos and fat free.