Lecture 3: Computer Architectures

Basic Computer Architecture Von Neumann Architecture Memory instruction data Input unit Output unit ALU Processor CU Reg.

Levels of Parallelism Bit level parallelism Within arithmetic logic circuits Instruction level parallelism Multiple instructions execute per clock cycle Memory system parallelism Overlap of memory operations with computation Operating system parallelism More than one processor Multiple jobs run in parallel on SMP Loop level Procedure level

Within arithmetic logic circuits Levels of Parallelism Bit Level Parallelism Within arithmetic logic circuits

Instruction Level Parallelism (ILP) Levels of Parallelism Instruction Level Parallelism (ILP) Multiple instructions execute per clock cycle Pipelining (instruction - data) Multiple Issue (VLIW)

Memory System Parallelism Levels of Parallelism Memory System Parallelism Overlap of memory operations with computation

Operating System Parallelism Levels of Parallelism Operating System Parallelism There are more than one processor Multiple jobs run in parallel on SMP Loop level Procedure level

Flynn’s Taxonomy Single Instruction stream - Single Data stream (SISD) Single Instruction stream - Multiple Data stream (SIMD) Multiple Instruction stream - Single Data stream (MISD) Multiple Instruction stream - Multiple Data stream (MIMD)

Single Instruction stream - Single Data stream (SISD) Von Neumann Architecture Memory instruction data ALU CU Processor

Flynn’s Taxonomy Single Instruction stream - Single Data stream (SISD) Single Instruction stream - Multiple Data stream (SIMD) Multiple Instruction stream - Single Data stream (MISD) Multiple Instruction stream - Multiple Data stream (MIMD)

Single Instruction stream - Multiple Data stream (SIMD) Instructions of the program are broadcast to more than one processor Each processor executes the same instruction synchronously, but using different data Used for applications that operate upon arrays of data data PE data PE instruction CU Memory data PE data PE instruction

Flynn’s Taxonomy Single Instruction stream - Single Data stream (SISD) Single Instruction stream - Multiple Data stream (SIMD) Multiple Instruction stream - Single Data stream (MISD) Multiple Instruction stream - Multiple Data stream (MIMD)

Multiple Instruction stream - Multiple Data stream (MIMD) Each processor has a separate program An instruction stream is generated for each program on each processor Each instruction operates upon different data

Multiple Instruction stream - Multiple Data stream (MIMD) Shared memory Distributed memory

Shared vs Distributed Memory Each processor has its own local memory Message-passing is used to exchange data between processors Shared memory Single address space All processes have access to the pool of shared memory P M Network Memory Bus P

Distributed Memory Processors cannot directly access another processor’s memory Each node has a network interface (NI) for communication and synchronization M M M M P P P P NI NI NI NI Network

Distributed Memory Each processor executes different instructions asynchronously, using different data instr data M CU PE data data instr M CU PE data Network data instr M CU PE data data instr M CU PE data

Shared Memory Each processor executes different instructions asynchronously, using different data data CU PE data CU PE Memory data CU PE data CU PE instruction

Shared Memory Uniform memory access (UMA) Each processor has uniform access to memory (symmetric multiprocessor - SMP) Non-uniform memory access (NUMA) Time for memory access depends on the location of data Local access is faster than non-local access Easier to scale than SMPs P P P P Bus Memory Memory Bus P Memory Bus P Network

Distributed Shared Memory Making the main memory of a cluster of computers look as if it is a single memory with a single address space Shared memory programming techniques can be used

Multicore Systems Many general purpose processors GPU (Graphics Processor Unit) GPGPU (General Purpose GPU) Hybrid Memory The trend is: Board composed of multiple manycore chips sharing memory Rack composed of multiple boards A room full of these racks

Distributed Systems Clusters Grid Individual computers, that are tightly coupled by software, in a local environment, to work together on single problems or on related problems Grid Many individual systems, that are geographically distributed, are tightly coupled by software, to work together on single problems or on related problems