1. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn

2. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Chapter 2 Parallel Architectures

3. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Outline Interconnection networks Interconnection networks Processor arrays Processor arrays Multiprocessors Multiprocessors Multicomputers Multicomputers Flynn’s taxonomy Flynn’s taxonomy

4. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Interconnection Networks Uses of interconnection networks Uses of interconnection networks  Connect processors to shared memory  Connect processors to each other Interconnection media types Interconnection media types  Shared medium  Switched medium

5. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shared versus Switched Media

6. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shared Medium Allows only message at a time Allows only message at a time Messages are broadcast Messages are broadcast Each processor “listens” to every message Each processor “listens” to every message Arbitration is decentralized Arbitration is decentralized Collisions require resending of messages Collisions require resending of messages Ethernet is an example Ethernet is an example

7. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Switched Medium Supports point-to-point messages between pairs of processors Supports point-to-point messages between pairs of processors Each processor has its own path to switch Each processor has its own path to switch Advantages over shared media Advantages over shared media  Allows multiple messages to be sent simultaneously  Allows scaling of network to accommodate increase in processors

8. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Switch Network Topologies View switched network as a graph View switched network as a graph  Vertices = processors or switches  Edges = communication paths Two kinds of topologies Two kinds of topologies  Direct  Indirect

9. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Direct Topology Ratio of switch nodes to processor nodes is 1:1 Ratio of switch nodes to processor nodes is 1:1 Every switch node is connected to Every switch node is connected to  1 processor node  At least 1 other switch node

10. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Indirect Topology Ratio of switch nodes to processor nodes is greater than 1:1 Ratio of switch nodes to processor nodes is greater than 1:1 Some switches simply connect other switches Some switches simply connect other switches

11. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Switch Topologies Diameter Diameter Bisection width Bisection width Number of edges / node Number of edges / node Constant edge length? (yes/no) Constant edge length? (yes/no)

12. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 2-D Mesh Network Direct topology Direct topology Switches arranged into a 2-D lattice Switches arranged into a 2-D lattice Communication allowed only between neighboring switches Communication allowed only between neighboring switches Variants allow wraparound connections between switches on edge of mesh Variants allow wraparound connections between switches on edge of mesh

13. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 2-D Meshes

14. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating 2-D Meshes Diameter:  (n 1/2 ) Diameter:  (n 1/2 ) Bisection width:  (n 1/2 ) Bisection width:  (n 1/2 ) Number of edges per switch: 4 Number of edges per switch: 4 Constant edge length? Yes Constant edge length? Yes

15. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Binary Tree Network Indirect topology Indirect topology n = 2 d processor nodes, n-1 switches n = 2 d processor nodes, n-1 switches

16. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Binary Tree Network Diameter: 2 log n Diameter: 2 log n Bisection width: 1 Bisection width: 1 Edges / node: 3 Edges / node: 3 Constant edge length? No Constant edge length? No

17. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypertree Network Indirect topology Indirect topology Shares low diameter of binary tree Shares low diameter of binary tree Greatly improves bisection width Greatly improves bisection width From “front” looks like k-ary tree of height d From “front” looks like k-ary tree of height d From “side” looks like upside down binary tree of height d From “side” looks like upside down binary tree of height d

18. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypertree Network

19. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating 4-ary Hypertree Diameter: log n Diameter: log n Bisection width: n / 2 Bisection width: n / 2 Edges / node: 6 Edges / node: 6 Constant edge length? No Constant edge length? No

20. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Butterfly Network Indirect topology Indirect topology n = 2 d processor nodes connected by n(log n + 1) switching nodes n = 2 d processor nodes connected by n(log n + 1) switching nodes

21. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Butterfly Network Routing

22. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Butterfly Network Diameter: log n Diameter: log n Bisection width: n / 2 Bisection width: n / 2 Edges per node: 4 Edges per node: 4 Constant edge length? No Constant edge length? No

23. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypercube Directory topology Directory topology 2 x 2 x … x 2 mesh 2 x 2 x … x 2 mesh Number of nodes a power of 2 Number of nodes a power of 2 Node addresses 0, 1, …, 2 k -1 Node addresses 0, 1, …, 2 k -1 Node i connected to k nodes whose addresses differ from i in exactly one bit position Node i connected to k nodes whose addresses differ from i in exactly one bit position

24. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypercube Addressing

25. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Hypercubes Illustrated

26. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Hypercube Network Diameter: log n Diameter: log n Bisection width: n / 2 Bisection width: n / 2 Edges per node: log n Edges per node: log n Constant edge length? No Constant edge length? No

27. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shuffle-exchange Direct topology Direct topology Number of nodes a power of 2 Number of nodes a power of 2 Nodes have addresses 0, 1, …, 2 k -1 Nodes have addresses 0, 1, …, 2 k -1 Two outgoing links from node i Two outgoing links from node i  Shuffle link to node LeftCycle(i)  Exchange link to node [xor (i, 1)]

28. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shuffle-exchange Illustrated 01234567

29. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shuffle-exchange Addressing

30. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Evaluating Shuffle-exchange Diameter: 2log n - 1 Diameter: 2log n - 1 Bisection width:  n / log n Bisection width:  n / log n Edges per node: 2 Edges per node: 2 Constant edge length? No Constant edge length? No

31. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Comparing Networks All have logarithmic diameter except 2-D mesh All have logarithmic diameter except 2-D mesh Hypertree, butterfly, and hypercube have bisection width n / 2 Hypertree, butterfly, and hypercube have bisection width n / 2 All have constant edges per node except hypercube All have constant edges per node except hypercube Only 2-D mesh keeps edge lengths constant as network size increases Only 2-D mesh keeps edge lengths constant as network size increases

32. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Multiprocessors Multiprocessor: multiple-CPU computer with a shared memory Multiprocessor: multiple-CPU computer with a shared memory Same address on two different CPUs refers to the same memory location Same address on two different CPUs refers to the same memory location Avoid three problems of processor arrays Avoid three problems of processor arrays  Can be built from commodity CPUs  Naturally support multiple users  Maintain efficiency in conditional code

33. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Centralized Multiprocessor Straightforward extension of uniprocessor Straightforward extension of uniprocessor Add CPUs to bus Add CPUs to bus All processors share same primary memory All processors share same primary memory Memory access time same for all CPUs Memory access time same for all CPUs  Uniform memory access (UMA) multiprocessor  Symmetrical multiprocessor (SMP)

34. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Centralized Multiprocessor

35. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Private and Shared Data Private data: items used only by a single processor Private data: items used only by a single processor Shared data: values used by multiple processors Shared data: values used by multiple processors In a multiprocessor, processors communicate via shared data values In a multiprocessor, processors communicate via shared data values

36. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Problems Associated with Shared Data Cache coherence Cache coherence  Replicating data across multiple caches reduces contention  How to ensure different processors have same value for same address? Synchronization Synchronization  Mutual exclusion  Barrier

37. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Cache-coherence Problem Cache CPU A Cache CPU B Memory 7 X

38. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Cache-coherence Problem CPU ACPU B Memory 7 X 7

39. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Cache-coherence Problem CPU ACPU B Memory 7 X 7 7

40. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Cache-coherence Problem CPU ACPU B Memory 2 X 7 2

41. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Write Invalidate Protocol CPU ACPU B 7 X 7 7 Cache control monitor

42. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Write Invalidate Protocol CPU ACPU B 7 X 7 7 Intent to write X

43. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Write Invalidate Protocol CPU ACPU B 7 X 7 Intent to write X

44. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Write Invalidate Protocol CPU ACPU B X 2 2

45. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Distributed Multiprocessor Distribute primary memory among processors Distribute primary memory among processors Increase aggregate memory bandwidth and lower average memory access time Increase aggregate memory bandwidth and lower average memory access time Allow greater number of processors Allow greater number of processors Also called non-uniform memory access (NUMA) multiprocessor Also called non-uniform memory access (NUMA) multiprocessor

46. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Distributed Multiprocessor

47. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Cache Coherence Some NUMA multiprocessors do not support it in hardware Some NUMA multiprocessors do not support it in hardware  Only instructions, private data in cache  Large memory access time variance Implementation more difficult Implementation more difficult  No shared memory bus to “snoop”  Directory-based protocol needed

48. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Directory-based Protocol Distributed directory contains information about cacheable memory blocks Distributed directory contains information about cacheable memory blocks One directory entry for each cache block One directory entry for each cache block Each entry has Each entry has  Sharing status  Which processors have copies

49. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Sharing Status Uncached Uncached  Block not in any processor’s cache Shared Shared  Cached by one or more processors  Read only Exclusive Exclusive  Cached by exactly one processor  Processor has written block  Copy in memory is obsolete

50. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Directory-based Protocol Interconnection Network Directory Local Memory Cache CPU 0 Directory Local Memory Cache CPU 1 Directory Local Memory Cache CPU 2

51. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Directory-based Protocol Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X U 0 0 0 Bit Vector

52. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X U 0 0 0 Read Miss

53. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X S 1 0 0

54. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X S 1 0 0 7 X

55. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 2 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X S 1 0 0 7 X Read Miss

56. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 2 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X S 1 0 1 7 X

57. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 2 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X S 1 0 1 7 X 7 X

58. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 6 to X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X S 1 0 1 7 X 7 X Write Miss

59. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 6 to X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X S 1 0 1 7 X 7 X Invalidate

60. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 6 to X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X E 1 0 0 6 X

61. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 1 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X E 1 0 0 6 X Read Miss

62. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 1 Reads X Interconnection Network CPU 0CPU 1CPU 2 7 X Caches Memories Directories X E 1 0 0 6 X Switch to Shared

63. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 1 Reads X Interconnection Network CPU 0CPU 1CPU 2 6 X Caches Memories Directories X E 1 0 0 6 X

64. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 1 Reads X Interconnection Network CPU 0CPU 1CPU 2 6 X Caches Memories Directories X S 1 1 0 6 X 6 X

65. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 2 Writes 5 to X Interconnection Network CPU 0CPU 1CPU 2 6 X Caches Memories Directories X S 1 1 0 6 X 6 X Write Miss

66. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 2 Writes 5 to X Interconnection Network CPU 0CPU 1CPU 2 6 X Caches Memories Directories X S 1 1 0 6 X 6 X Invalidate

67. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 2 Writes 5 to X Interconnection Network CPU 0CPU 1CPU 2 6 X Caches Memories Directories X E 0 0 1 5 X

68. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 4 to X Interconnection Network CPU 0CPU 1CPU 2 6 X Caches Memories Directories X E 0 0 1 5 X Write Miss

69. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 4 to X Interconnection Network CPU 0CPU 1CPU 2 6 X Caches Memories Directories X E 1 0 0 Take Away 5 X

70. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 4 to X Interconnection Network CPU 0CPU 1CPU 2 5 X Caches Memories Directories X E 0 1 0 5 X

71. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 4 to X Interconnection Network CPU 0CPU 1CPU 2 5 X Caches Memories Directories X E 1 0 0

72. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 4 to X Interconnection Network CPU 0CPU 1CPU 2 5 X Caches Memories Directories X E 1 0 0 5 X

73. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes 4 to X Interconnection Network CPU 0CPU 1CPU 2 5 X Caches Memories Directories X E 1 0 0 4 X

74. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes Back X Block Interconnection Network CPU 0CPU 1CPU 2 5 X Caches Memories Directories X E 1 0 0 4 X 4 X Data Write Back

75. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. CPU 0 Writes Back X Block Interconnection Network CPU 0CPU 1CPU 2 4 X Caches Memories Directories X U 0 0 0

76. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Multicomputer Distributed memory multiple-CPU computer Distributed memory multiple-CPU computer Same address on different processors refers to different physical memory locations Same address on different processors refers to different physical memory locations Processors interact through message passing Processors interact through message passing Commercial multicomputers Commercial multicomputers Commodity clusters Commodity clusters

77. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Asymmetrical Multicomputer

78. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Asymmetrical MC Advantages Back-end processors dedicated to parallel computations  Easier to understand, model, tune performance Back-end processors dedicated to parallel computations  Easier to understand, model, tune performance Only a simple back-end operating system needed  Easy for a vendor to create Only a simple back-end operating system needed  Easy for a vendor to create

79. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Asymmetrical MC Disadvantages Front-end computer is a single point of failure Front-end computer is a single point of failure Single front-end computer limits scalability of system Single front-end computer limits scalability of system Primitive operating system in back-end processors makes debugging difficult Primitive operating system in back-end processors makes debugging difficult Every application requires development of both front-end and back-end program Every application requires development of both front-end and back-end program

80. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Symmetrical Multicomputer

81. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Symmetrical MC Advantages Alleviate performance bottleneck caused by single front-end computer Alleviate performance bottleneck caused by single front-end computer Better support for debugging Better support for debugging Every processor executes same program Every processor executes same program

82. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Symmetrical MC Disadvantages More difficult to maintain illusion of single “parallel computer” More difficult to maintain illusion of single “parallel computer” No simple way to balance program development workload among processors No simple way to balance program development workload among processors More difficult to achieve high performance when multiple processes on each processor More difficult to achieve high performance when multiple processes on each processor

83. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. ParPar Cluster, A Mixed Model

84. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Commodity Cluster Co-located computers Co-located computers Dedicated to running parallel jobs Dedicated to running parallel jobs No keyboards or displays No keyboards or displays Identical operating system Identical operating system Identical local disk images Identical local disk images Administered as an entity Administered as an entity

85. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Network of Workstations Dispersed computers Dispersed computers First priority: person at keyboard First priority: person at keyboard Parallel jobs run in background Parallel jobs run in background Different operating systems Different operating systems Different local images Different local images Checkpointing and restarting important Checkpointing and restarting important

86. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Flynn’s Taxonomy Instruction stream Instruction stream Data stream Data stream Single vs. multiple Single vs. multiple Four combinations Four combinations  SISD  SIMD  MISD  MIMD

87. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. SISD Single Instruction, Single Data Single Instruction, Single Data Single-CPU systems Single-CPU systems Note: co-processors don’t count Note: co-processors don’t count  Functional  I/O Example: PCs Example: PCs

88. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. SIMD Single Instruction, Multiple Data Single Instruction, Multiple Data Two architectures fit this category Two architectures fit this category  Pipelined vector processor (e.g., Cray-1)  Processor array (e.g., Connection Machine)

89. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. MISD Multiple Instruction, Single Data Multiple Instruction, Single Data Example: systolic array Example: systolic array

90. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. MIMD Multiple Instruction, Multiple Data Multiple Instruction, Multiple Data Multiple-CPU computers Multiple-CPU computers  Multiprocessors  Multicomputers

91. Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Summary Commercial parallel computers appeared in 1980s Commercial parallel computers appeared in 1980s Multiple-CPU computers now dominate Multiple-CPU computers now dominate Small-scale: Centralized multiprocessors Small-scale: Centralized multiprocessors Large-scale: Distributed memory architectures (multiprocessors or multicomputers) Large-scale: Distributed memory architectures (multiprocessors or multicomputers)