1. Parallel Architectures: Topologies Heiko Schröder, 2003

2. Parallel Architectures 2 Types of sequential processors (SISD) processor memory processor memory cache memory processor Von Neumann bottleneck

3. Heiko Schröder, 2003 Parallel Architectures 3 SIMD MIMD PE Global control unit Interconnection network PE + control unit PE + control unit PE + control unit PE + control unit Interconnection network SPMD SIMD

4. Heiko Schröder, 2003 Parallel Architectures 4 Message passing / shared address space PE + M control unit PE + M control unit PE + M control unit PE + M control unit Interconnection network P P P P P M M M M P/M

5. Heiko Schröder, 2003 Parallel Architectures 5 Various communication networks State of the art technology Important aspects of routing schemes Known results (theory) The internet

6. Heiko Schröder, 2003 Parallel Architectures 6 Desirable feature of a network 1. Algorithmic Low diameter (1, complete graph) High bisection width (complete graph) n(n-1)/2 edges Degree n-1 2. Technical Low degree (pin limitations – constant – modular – mesh) Short wires (mesh) Small area (mesh) Regular structure (mesh)

7. Heiko Schröder, 2003 Parallel Architectures 7 Diameter n-1 Bisection width 1 Connection networks I 1-D mesh (linear array)

8. Heiko Schröder, 2003 Parallel Architectures 8 Tree Diameter 2(log n) Bisection width 1

9. Heiko Schröder, 2003 Parallel Architectures 9 H-tree Area: O(n) Longest wire :O(  n) Clock distribution

10. Heiko Schröder, 2003 Parallel Architectures 10 2-D Mesh Diameter: Bisection width :

11. Heiko Schröder, 2003 Parallel Architectures 11 Torus 12345678182736451 2 3 4 5 6 7 8 1 8 2 7 3 6 4 5 12345678 Reduced diameter Increased bisection width All nodes equivalent Long wires?

12. Heiko Schröder, 2003 Parallel Architectures 12 3-D Mesh Diameter: Bisection:

13. Heiko Schröder, 2003 Parallel Architectures 13 Hypercube 0-D 0 1 1-D 0 0101 10101 2-D 000000010010 001001011011 100100110110 101101111111 3-D 01 4-D diameter log n bisection width n/2

14. Heiko Schröder, 2003 Parallel Architectures 14 Cube Connected Cycles nodes # nodes nodes Diameter> bisection

15. Heiko Schröder, 2003 Parallel Architectures 15 Exchange (lsb) Shuffle (rotate -- left or right) 000 001 100 010011 101 110111 8-node shuffle-exchange graph Degree: 3 Diameter: 2 log n –1 : at most (log n –1) shuffles + (log n ) exchanges Bisection width:  (n / log n)

16. Heiko Schröder, 2003 Parallel Architectures 16 0000 0001 11101111 0100 01011010 1000 0010 1001 00110110 1100 1011 0111 1101 Exchange (lsb) Shuffle (rotate -- left or right) 16-node shuffle-exchange graph u 1 u 2 …u k-1 u k ex u 1 u 2 …u k-1 v 1 u k v 1 v 2 …v k-1 … u 2 …u k v 1 v 2 ls+ex v 1 v 2 …v k ls+ex Diameter: 2 log n –1 : at most (log n –1) shuffles + (log n ) exchanges Bisection width:  (n / log n) Degree: 3

17. Heiko Schröder, 2003 Parallel Architectures 17 u 1 u 2 …u k-1 u k u 2 u 3 …u k-1 u k 0 0 u 1 u 2 …u k-1 u k u 2 u 3 …u k-1 u k 1 1 3-dimensional de Bruijn graph In-degree = out-degree = 2 Diameter: log n Bisection width:  (n / log n) Each Eulerian tour = De Bruijn sequence = contains each possible sub-string of length 4 exactly once 1111001011010000 De Bruijn sequence 000 100 001 111 110 101 010 011 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1

18. Heiko Schröder, 2003 Parallel Architectures 18 Butterfly network Unique path FFT routing sorting

19. Heiko Schröder, 2003 Parallel Architectures 19 Benes network

20. Heiko Schröder, 2003 Parallel Architectures 20 Mesh of trees Diameter  (log n) Bisection width  ( )

21. Heiko Schröder, 2003 Parallel Architectures 21 The Power of Hypercubes 4-D Hamiltonian cycle Gray codes k-D meshes (tori), N-nodes simulates mesh of trees simulates hypercubic networks contains complete binary tree, almost normal algorithms

22. Heiko Schröder, 2003 Parallel Architectures 22 Hamiltonian Cycle A hypercube contains a Hamiltonian cycle -- proof by induction. Each Hamiltonian cycle corresponds to a Gray code (only one bit is changed per link).

23. Heiko Schröder, 2003 Parallel Architectures 23 Gray code 0101 00 01 11 10 000 001 011 010 110 111 101 100 reflection

24. Heiko Schröder, 2003 Parallel Architectures 24 Hypercube contains meshes/tori 20 30 21 31 23 33 22 32 10 00 11 01 13 03 12 02 wrap around Theorem: Any n 1 x n 2 x … x n k mesh (with or without wrap arounds) is a sub-graph of an n-D hypercube if  n i = 2 n. Proof: (see Leighton: Each sub-cube has Hamiltonian cycle)

25. Heiko Schröder, 2003 Parallel Architectures 25 Hypercube contains double-rooted trees HC can implement all tree algorithms and also all mesh-of-tree-algorithms (possibly with minor delay). double-roots (different dimension)

26. Heiko Schröder, 2003 Parallel Architectures 26 Normal algorithms A hypercube algorithm is said to be normal if only one dimension of hypercube edges is used at any step and if consecutive dimensions are used in consecutive steps. Most hypercube algorithms are normal. Normal algorithms can be embedded efficiently on hypercubic networks

27. Heiko Schröder, 2003 Parallel Architectures 27 Josephus graph: Every even node k is connected to k+2 i -3 Diameter: about (log n) / 2 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

28. Heiko Schröder, 2003 Parallel Architectures 28 1234 3214 2314 1324 3124 2134 4132 1432 3412 4312 1342 3142 2143 4123 1423 2413 4213 1243 3241 2341 4321 3421 2431 4231 Star graph: Set of nodes: k! nodes of degree k-1. Permutations of k elements. Set of edges: Exchange of first element with one other. Small degree, diameter about 2 log n. Open problems: E.g. are there (k-1)/2 edge disjoint Hamiltonian cycles? Number of nodes versus degree (Star/HC): 24, 120, 720, 4340, 34720, 312480 16, 32, 64, 128, 256, 512

29. Heiko Schröder, 2003 Parallel Architectures 29 pin - limitations 1 4-D 12 192 16 256 16

30. Heiko Schröder, 2003 Parallel Architectures 30 wiring - limitations 4-D 12 1 2 16 nodes bisection width: 256 32 K 25cm 32 m

31. Heiko Schröder, 2003 Parallel Architectures 31 Improve the topology? The internet

32. Heiko Schröder, 2003 Parallel Architectures 32 against parallelism cost(large) < cost (2 small) all the FORTRAN / C software let’s stick to pipelining let’s wait for faster machines Amdahl’s Law