Lecture 3 TTH 03:30AM-04:45PM Dr. Jianjun Hu CSCE569 Parallel Computing University of South Carolina Department of.

Slides:



Advertisements
Similar presentations
Multiple Processor Systems
Advertisements

Prepared 7/28/2011 by T. O’Neil for 3460:677, Fall 2011, The University of Akron.
SE-292 High Performance Computing
Super computers Parallel Processing By: Lecturer \ Aisha Dawood.
Today’s topics Single processors and the Memory Hierarchy
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn.
Multiple Processor Systems
Jie Liu, Ph.D. Professor Department of Computer Science
Cache Coherent Distributed Shared Memory. Motivations Small processor count –SMP machines –Single shared memory with multiple processors interconnected.
CSCI 8150 Advanced Computer Architecture Hwang, Chapter 1 Parallel Computer Models 1.2 Multiprocessors and Multicomputers.
Multiprocessors CSE 4711 Multiprocessors - Flynn’s Taxonomy (1966) Single Instruction stream, Single Data stream (SISD) –Conventional uniprocessor –Although.
Parallel Architectures: Topologies Heiko Schröder, 2003.
Parallel Architectures: Topologies Heiko Schröder, 2003.
Prepared 7/28/2011 by T. O’Neil for 3460:677, Fall 2011, The University of Akron.
Advanced Topics in Algorithms and Data Structures An overview of the lecture 2 Models of parallel computation Characteristics of SIMD models Design issue.

Chapter 17 Parallel Processing.
Multiprocessors CSE 471 Aut 011 Multiprocessors - Flynn’s Taxonomy (1966) Single Instruction stream, Single Data stream (SISD) –Conventional uniprocessor.
Lecture 10 Outline Material from Chapter 2 Interconnection networks Processor arrays Multiprocessors Multicomputers Flynn’s taxonomy.
1 Lecture 23: Multiprocessors Today’s topics:  RAID  Multiprocessor taxonomy  Snooping-based cache coherence protocol.
Models of Parallel Computation Advanced Algorithms & Data Structures Lecture Theme 12 Prof. Dr. Th. Ottmann Summer Semester 2006.
Course Outline Introduction in algorithms and applications Parallel machines and architectures Overview of parallel machines, trends in top-500 Cluster.
1 CSE SUNY New Paltz Chapter Nine Multiprocessors.
1 Computer Science, University of Warwick Architecture Classifications A taxonomy of parallel architectures: in 1972, Flynn categorised HPC architectures.
Introduction to Parallel Processing Ch. 12, Pg
Lecture 37: Chapter 7: Multiprocessors Today’s topic –Introduction to multiprocessors –Parallelism in software –Memory organization –Cache coherence 1.
Course Outline Introduction in software and applications. Parallel machines and architectures –Overview of parallel machines –Cluster computers (Myrinet)
CS668- Lecture 2 - Sept. 30 Today’s topics Parallel Architectures (Chapter 2) Memory Hierarchy Busses and Switched Networks Interconnection Network Topologies.
August 15, 2001Systems Architecture II1 Systems Architecture II (CS ) Lecture 12: Multiprocessors: Non-Uniform Memory Access * Jeremy R. Johnson.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn.
Parallel Computer Architecture and Interconnect 1b.1.
Chapter 2 Parallel Architectures. Outline Interconnection networks Interconnection networks Processor arrays Processor arrays Multiprocessors Multiprocessors.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn.
CHAPTER 12 INTRODUCTION TO PARALLEL PROCESSING CS 147 Guy Wong page
Chapter 2 Parallel Architecture. Moore’s Law The number of transistors on a chip doubles every years. – Has been valid for over 40 years – Can’t.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn.
An Overview of Parallel Computing. Hardware There are many varieties of parallel computing hardware and many different architectures The original classification.
Course Outline Introduction in algorithms and applications Parallel machines and architectures Overview of parallel machines, trends in top-500, clusters,
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Shared versus Switched Media.
Lecture 4 TTH 03:30AM-04:45PM Dr. Jianjun Hu CSCE569 Parallel Computing University of South Carolina Department of.
PARALLEL PROCESSOR- TAXONOMY. CH18 Parallel Processing {Multi-processor, Multi-computer} Multiple Processor Organizations Symmetric Multiprocessors Cache.
Outline Why this subject? What is High Performance Computing?
Super computers Parallel Processing
Day 2. Agenda Parallelism basics Parallel machines Parallelism again High Throughput Computing –Finding the right grain size.
Multiprocessor So far, we have spoken at length microprocessors. We will now study the multiprocessor, how they work, what are the specific problems that.
Spring EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Advanced Computer Architecture Lecture 22 Distributed computer Interconnection.
Multiprocessor  Use large number of processor design for workstation or PC market  Has an efficient medium for communication among the processor memory.
1 Lecture 17: Multiprocessors Topics: multiprocessor intro and taxonomy, symmetric shared-memory multiprocessors (Sections )
Background Computer System Architectures Computer System Software.
CMSC 611: Advanced Computer Architecture Shared Memory Most slides adapted from David Patterson. Some from Mohomed Younis.
COMP8330/7330/7336 Advanced Parallel and Distributed Computing Tree-Based Networks Cache Coherence Dr. Xiao Qin Auburn University
Lecture 13 Parallel Processing. 2 What is Parallel Computing? Traditionally software has been written for serial computation. Parallel computing is the.
1 Computer Architecture & Assembly Language Spring 2001 Dr. Richard Spillman Lecture 26 – Alternative Architectures.
Auburn University COMP8330/7330/7336 Advanced Parallel and Distributed Computing Parallel Hardware Dr. Xiao Qin Auburn.
Overview Parallel Processing Pipelining
Parallel Architecture
CHAPTER SEVEN PARALLEL PROCESSING © Prepared By: Razif Razali.
Distributed and Parallel Processing
Multiprocessor Systems
Course Outline Introduction in algorithms and applications
CS 147 – Parallel Processing
Parallel Architectures Based on Parallel Computing, M. J. Quinn
Chapter 17 Parallel Processing
Outline Interconnection networks Processor arrays Multiprocessors
Multiprocessors - Flynn’s taxonomy (1966)
AN INTRODUCTION ON PARALLEL PROCESSING
High Performance Computing & Bioinformatics Part 2 Dr. Imad Mahgoub
Advanced Computer and Parallel Processing
Advanced Computer and Parallel Processing
Lecture 23: Virtual Memory, Multiprocessors
Presentation transcript:

Lecture 3 TTH 03:30AM-04:45PM Dr. Jianjun Hu CSCE569 Parallel Computing University of South Carolina Department of Computer Science and Engineering

Outline- Parallel Architecture Interconnection networks Processor arrays Multiprocessors Multicomputers Flynn’s taxonomy

Interconnection Networks Uses of interconnection networks Connect processors to shared memory Connect processors to each other Interconnection media types Shared medium Switched medium

Shared versus Switched Media

Shared Medium Allows only message at a time Messages are broadcast Each processor “listens” to every message Arbitration is decentralized Collisions require resending of messages Ethernet is an example

Switched Medium Supports point-to-point messages between pairs of processors Each processor has its own path to switch Advantages over shared media Allows multiple messages to be sent simultaneously Allows scaling of network to accommodate increase in processors

Switch Network Topologies View switched network as a graph Vertices = processors or switches Edges = communication paths Two kinds of topologies Direct Indirect

Direct Topology Ratio of switch nodes to processor nodes is 1:1 Every switch node is connected to 1 processor node At least 1 other switch node

Indirect Topology Ratio of switch nodes to processor nodes is greater than 1:1 Some switches simply connect other switches

Evaluating Switch Topologies Diameter Bisection width Number of edges / node Constant edge length? (yes/no)

2-D Mesh Network Direct topology Switches arranged into a 2-D lattice Communication allowed only between neighboring switches Variants allow wraparound connections between switches on edge of mesh

2-D Meshes

Evaluating 2-D Meshes Diameter:  (n 1/2 ) Bisection width:  (n 1/2 ) Number of edges per switch: 4 Constant edge length? Yes

Binary Tree Network Indirect topology n = 2 d processor nodes, n-1 switches

Evaluating Binary Tree Network Diameter: 2 log n Bisection width: 1 Edges / node: 3 Constant edge length? No

Hypertree Network Indirect topology Shares low diameter of binary tree Greatly improves bisection width From “front” looks like k-ary tree of height d From “side” looks like upside down binary tree of height d

Hypertree Network

Evaluating 4-ary Hypertree Diameter: log n Bisection width: n / 2 Edges / node: 6 Constant edge length? No

Butterfly Network Indirect topology n = 2 d processor nodes connected by n(log n + 1) switching nodes

Butterfly Network Routing

Evaluating Butterfly Network Diameter: log n Bisection width: n / 2 Edges per node: 4 Constant edge length? No

Hypercube Directory topology 2 x 2 x … x 2 mesh Number of nodes a power of 2 Node addresses 0, 1, …, 2 k -1 Node i connected to k nodes whose addresses differ from i in exactly one bit position

Hypercube Addressing

Hypercubes Illustrated

Evaluating Hypercube Network Diameter: log n Bisection width: n / 2 Edges per node: log n Constant edge length? No

Shuffle-exchange Direct topology Number of nodes a power of 2 Nodes have addresses 0, 1, …, 2 k -1 Two outgoing links from node i Shuffle link to node LeftCycle(i) Exchange link to node [xor (i, 1)]

Shuffle-exchange Illustrated

Shuffle-exchange Addressing

Evaluating Shuffle-exchange Diameter: 2log n - 1 Bisection width:  n / log n Edges per node: 2 Constant edge length? No

Comparing Networks All have logarithmic diameter except 2-D mesh Hypertree, butterfly, and hypercube have bisection width n / 2 All have constant edges per node except hypercube Only 2-D mesh keeps edge lengths constant as network size increases

Vector Computers Vector computer: instruction set includes operations on vectors as well as scalars Two ways to implement vector computers Pipelined vector processor: streams data through pipelined arithmetic units Processor array: many identical, synchronized arithmetic processing elements

Why Processor Arrays? Historically, high cost of a control unit Scientific applications have data parallelism

Processor Array

Data/instruction Storage Front end computer Program Data manipulated sequentially Processor array Data manipulated in parallel

Processor Array Performance Performance: work done per time unit Performance of processor array Speed of processing elements Utilization of processing elements

Performance Example processors Each adds a pair of integers in 1  sec What is performance when adding two 1024-element vectors (one per processor)?

Performance Example processors Each adds two integers in 1  sec Performance adding two vectors of length 600?

2-D Processor Interconnection Network Each VLSI chip has 16 processing elements

Processor Array Shortcomings Not all problems are data-parallel Speed drops for conditionally executed code Don’t adapt to multiple users well Do not scale down well to “starter” systems Rely on custom VLSI for processors Expense of control units has dropped

Multiprocessors Multiprocessor: multiple-CPU computer with a shared memory Same address on two different CPUs refers to the same memory location Avoid three problems of processor arrays Can be built from commodity CPUs Naturally support multiple users Maintain efficiency in conditional code

Centralized Multiprocessor Straightforward extension of uniprocessor Add CPUs to bus All processors share same primary memory Memory access time same for all CPUs Uniform memory access (UMA) multiprocessor Symmetrical multiprocessor (SMP)

Centralized Multiprocessor

Private and Shared Data Private data: items used only by a single processor Shared data: values used by multiple processors In a multiprocessor, processors communicate via shared data values

Problems Associated with Shared Data Cache coherence Replicating data across multiple caches reduces contention How to ensure different processors have same value for same address? Synchronization Mutual exclusion Barrier

Cache-coherence Problem Cache CPU A Cache CPU B Memory 7 X

Cache-coherence Problem CPU ACPU B Memory 7 X 7

Cache-coherence Problem CPU ACPU B Memory 7 X 7 7

Cache-coherence Problem CPU ACPU B Memory 2 X 7 2

Write Invalidate Protocol CPU ACPU B 7 X 7 7 Cache control monitor

Write Invalidate Protocol CPU ACPU B 7 X 7 7 Intent to write X

Write Invalidate Protocol CPU ACPU B 7 X 7 Intent to write X

Write Invalidate Protocol CPU ACPU B X 2 2

Memory consistency Use memory consistency model to achieve memory consistency Define allowable behavior of the memory system used by programmer of parallel programs. Consistency model guarantee that for any read/write input to the memory system, only allowable outputs are produced..

Sequential Consistency Model Require all write operations appear to all processors in the same order Can be guaranteed by the following sufficient conditions: 1) Every processor issues memory operations in program order 2) After a processor issues write op, wait until write is done bf. Issue next op 3) After a processor issued Read op, it waits until this read and write op whose value should be returned to completed. 4) R  R R  W W  W W  R 1) X  Y: X must complete bf. Y

Network embedding/Mapping E.g.: How to embed a network into hypercube Reflected Gray Code (RGC): Mapping nodes of network A to nodes of network B. Example: Map ring to hypercube

Embed ring/2d mesh to hypercube

Embed ring to hyperCube

Embed 2d mesh to hyperCube

Routing Routing algorithms depend on network topology, network contention, and network congestion. Deterministic or adaptive routing algo. XY routing for 2D meshes:

E-Cube routing for Hypercube

Routing for Omega network Stage 0Stage 1Stage 2

Omega network allows max 8 concurrent messaging. Blocking network since only 1 path exists for (A  B)

Non-blocking network: Benes For any permutation of There exists a switching of the Benes network to realize connection without collision… HOW do u find this configuration?

Distributed Multiprocessor Distribute primary memory among processors Increase aggregate memory bandwidth and lower average memory access time Allow greater number of processors Also called non-uniform memory access (NUMA) multiprocessor

Distributed Multiprocessor

Multicomputer Distributed memory multiple-CPU computer Same address on different processors refers to different physical memory locations Processors interact through message passing Commercial multicomputers Commodity clusters

Asymmetrical Multicomputer

Asymmetrical MC Advantages Back-end processors dedicated to parallel computations  Easier to understand, model, tune performance Only a simple back-end operating system needed  Easy for a vendor to create

Asymmetrical MC Disadvantages Front-end computer is a single point of failure Single front-end computer limits scalability of system Primitive operating system in back-end processors makes debugging difficult Every application requires development of both front-end and back-end program

Symmetrical Multicomputer

Symmetrical MC Advantages Alleviate performance bottleneck caused by single front-end computer Better support for debugging Every processor executes same program

Symmetrical MC Disadvantages More difficult to maintain illusion of single “parallel computer” No simple way to balance program development workload among processors More difficult to achieve high performance when multiple processes on each processor

ParPar Cluster, A Mixed Model

Commodity Cluster Co-located computers Dedicated to running parallel jobs No keyboards or displays Identical operating system Identical local disk images Administered as an entity

Network of Workstations Dispersed computers First priority: person at keyboard Parallel jobs run in background Different operating systems Different local images Checkpointing and restarting important

Flynn’s Taxonomy Instruction stream Data stream Single vs. multiple Four combinations SISD SIMD MISD MIMD

SISD Single Instruction, Single Data Single-CPU systems Note: co-processors don’t count Functional I/O Example: PCs

SIMD Single Instruction, Multiple Data Two architectures fit this category Pipelined vector processor (e.g., Cray-1) Processor array (e.g., Connection Machine)

MISD Multiple Instruction, Single Data Example: systolic array

MIMD Multiple Instruction, Multiple Data Multiple-CPU computers Multiprocessors Multicomputers

Summary Commercial parallel computers appeared in 1980s Multiple-CPU computers now dominate Small-scale: Centralized multiprocessors Large-scale: Distributed memory architectures (multiprocessors or multicomputers)

Summary (1/2) High performance computing U.S. government Capital-intensive industries Many companies and research labs Parallel computers Commercial systems Commodity-based systems