Class CS 775/875, Spring 2011 Amit H. Kumar, OCCS Old Dominion University

Take Away  Motivation  Basic difference between: MPI vs. RPC  Parallel Computer Memory Architectures  Parallel Programming Models  MPI Definition  MPI Examples

Motivation Work Queues  Work queues allow threads from one task to send processing work to another task in a decoupled fashion P P C C Shared Queue ProducerConsumer

Motivation …  Make this work in a distributed setting … P P C C Shared Network Queue ProducerConsumer

MPI vs. RPC  In simple terms they both are methods of Inter-Process Communication (IPC)  And both fall between Transport Layer and Application layer in the OSI model. FeaturesMPIRPC Portability Massively Parallel Computing Systems & Cluster of workstations Integral part of all OS* (Not completely sure about every OS in the market) Process Creation Static & Dynamic Topology Support SupportedNot Supported Load balancing ExcellentNot Supported

Parallel Computing one liner  Ultimately, parallel computing is an attempt to maximize the infinite but seemingly scarce commodity called time.

Parallel Computer Memory Architectures  Shared Memory: Shared memory parallel computers vary widely, but generally have in common the ability for all processors to access all memory as global address space.  Uniform Memory Access (UMA). Ex. SMP  Non-Uniform Memory Access (NUMA). Ex two or more SMPs linked.

Parallel Computer Memory Architectures…  Distributed Memory:  Like shared memory systems, distributed memory systems vary widely but share a common characteristic. Distributed memory systems require a communication network to connect inter-processor memory.  Memory addresses in one processor do not map to another processor, so there is no concept of global address space across all processors.

Parallel Computer Memory Architectures…  Hybrid Distributed-Shared Memory:  The largest and fastest computers in the world today employ both shared and distributed memory architectures.

Parallel Programming Models Parallel programming models exist as an abstraction above hardware and memory architectures. There are several parallel programming models in common use:  Shared Memory  tasks share a common address space, which they read and write asynchronously.  Threads  a single process can have multiple, concurrent execution paths. Ex implementations: POSIX threads & OpenMP  Message Passing  tasks exchange data through communications by sending and receiving messages. Ex: MPI & MPI-2 specification.  Data Parallel  tasks perform the same operation on their partition of work. Ex: High Performance Fortran – support data parallel constructs.  Hybrid

Define MPI  M P I = Message Passing Interface  In short:  MPI is a specification for the developers and users of message passing libraries. By itself, it is NOT a library - but rather the specification of what such a library should be.  Few implementations of MPI: MPICH,MPICH2, MVAPICH, MVAPICH2…

MPI Examples… Getting Started 

MPI uses objects called communicators and groups to define which collection of processes may communicate with each other.

Rank  Within a communicator, every process has its own unique, integer identifier assigned by the system when the process initializes. A rank is sometimes also called a "task ID". Ranks are contiguous and begin at zero.  Used by the programmer to specify the source and destination of messages. Often used conditionally by the application to control program execution (if rank=0 do this / if rank=1 do that).

First C Program

References  References of the content in the presentation is available upon request.  MPICH2 download: pich2/downloads/index.php?s=downloads pich2/downloads/index.php?s=downloads  MVAPICH2 dowload: state.edu/download/mvapich2/ state.edu/download/mvapich2/