Download presentation
Presentation is loading. Please wait.
1
1 Parallel Computing—Introduction to Message Passing Interface (MPI)
2
2 Two Important Concepts Two fundamental concepts of parallel programming are: Domain decomposition Functional decomposition
3
3 Domain Decomposition
4
4 Functional Decomposition
5
5 Message Passing Interface (MPI) MPI is a standard (an interface or an API): It defines a set of methods that are used by application developers to write their applications MPI library implement these methods MPI itself is not a library—it is a specification document that is followed! MPI-1.2 is the most popular specification version Reasons for popularity: Software and hardware vendors were involved Significant contribution from academia MPICH served as an early reference implementation MPI compilers are simply wrappers to widely used C and Fortran compilers History: The first draft specification was produced in 1993 MPI-2.0, introduced in 1999, adds many new features to MPI Bindings available to C, C++, and Fortran MPI is a success story: It is the mostly adopted programming paradigm of IBM Blue Gene systems At least two production-quality MPI libraries: MPICH2 (http://www-unix.mcs.anl.gov/mpi/mpich2/)http://www-unix.mcs.anl.gov/mpi/mpich2/ OpenMPI (http://open-mpi.org)http://open-mpi.org There’s even a Java library: MPJ Express (http://mpj-express.org)http://mpj-express.org
6
6 Message Passing Model Message passing model allows processors to communicate by passing messages: Processors do not share memory Data transfer between processors required cooperative operations to be performed by each processor: One processor sends the message while other receives the message
7
7 Proc 6 Proc 0 Proc 1 Proc 3 Proc 2 Proc 4 Proc 5 Proc 7 message CPU Memory LAN Ethernet Myrinet Infiniband etc Distributed Memory Cluster
8
8 Writing “Hello World” MPI Program MPI is very simple: Initialize MPI environment: MPI_Init(&argc,&argv); // C Code MPI.Init(args); // Java Code Send or receive message: MPI_Send(..); // C Code MPI.COMM_WORLD.Send(); // Java Code Finalize MPI environment MPI_Finalize(); // C Code MPI.Finalize(); // Java Code
9
9 Hello World in C #include #include “mpi.h”.. // Initialize MPI MPI_Init(&argsc,&&argsv); // Find out the `id’ or `rank’ of current process MPI_Comm_Rank(MPI_COMM_WORLD,&my_rank); //get the rank // Get total number of processes MPI_Comm_Size(MPI_COMM_WORLD,&p); //get total processor // Print the rank of the process printf(“Hello World from process no %d”,my_rank); MPI_Finalize();..
10
10 Hello World in Java import java.util.*; import mpi.*;.. // Initialize MPI MPI.Init(args); // start up MPI // Get total number of processes and rank size = MPI.COMM_WORLD.Size(); rank = MPI.COMM_WORLD.Rank(); System.out.println(“Hello World ”); MPI_Finalize();..
11
11 After Initialization import java.util.*; import mpi.*;.. // Initialize MPI MPI.Init(args); // start up MPI // Get total number of processes and rank size = MPI.COMM_WORLD.Size(); rank = MPI.COMM_WORLD.Rank();..
12
12 What is size? Total number of processes in a communicator: The size of MPI.COMM_WORLD is 6 import java.util.*; import mpi.*;.. // Get total number of processes size = MPI.COMM_WORLD.Size();..
13
13 What is rank? The “unique” identify (id) of a process in a communicator: Each of the six processes in MPI.COMM_WORLD has a distinct rank or id import java.util.*; import mpi.*;.. // Get total number of processes rank = MPI.COMM_WORLD.Rank();..
14
14 Running “HelloWorld” in C Write parallel code Start MPICH2 daemon Write machines file Start the parallel job
15
15
16
16
17
17 Running “Hello World” in Java The code is executed on a cluster called “Starbug”: One head-node “holly” and eight compute-nodes Steps: Write machines files Bootstrap MPJ Express (or any MPI library) runtime Write parallel application Compile and execute
18
18
19
19 Write machines files
20
20 Bootstrap MPJ Express runtime
21
21 Write Parallel Program
22
22 Compile and Execute
23
23 Single Program Multiple Data (SPMD) Model import java.util.*; import mpi.*; public class HelloWorld { MPI.Init(args); // start up MPI size = MPI.COMM_WORLD.Size(); rank = MPI.COMM_WORLD.Rank(); if (rank == 0) { System.out.println(“I am Process 0”); } else if (rank == 1) { System.out.println(“I am Process 1”); } MPI.Finalize(); }
24
24 Single Program Multiple Data (SPMD) Model import java.util.*; import mpi.*; public class HelloWorld { MPI.Init(args); // start up MPI size = MPI.COMM_WORLD.Size(); rank = MPI.COMM_WORLD.Rank(); if (rank%2 == 0) { System.out.println(“I am an even process”); } else if (rank%2 == 1) { System.out.println(“I am an odd process”); } MPI.Finalize(); }
25
25 Point to Point Communication The most fundamental facility provided by MPI Basically “exchange messages between two processes”: One process (source) sends message The other process (destination) receives message
26
26 Point to Point Communication It is possible to send message for each basic datatype: Floats, Integers, Doubles … Each message contains a “tag”—an identifier Tag1 Tag2
27
27 Process 6 Process 0 Process 1 Process 3 Process 2 Process 4 Process 5 Process 7 message Integers Process 4 Tag COMM_WORLD Point to Point Communication
28
28 Blocking and Non-blocking There are blocking and non-blocking version of send and receive methods Blocking versions: A process calls send() or recv(), these methods return when the message has been physically sent or received Non-blocking versions: A process calls isend() or irecv(), these methods return immediately The user can check the status of message by calling test() or wait() Note the “ i ” in isend() and irecv() Non-blocking versions provide overlapping of computation and communication: It also depends on the “quality” of the implementation
29
29 CPU waits “Blocking” send() recv() Sender Receiver time CPU waits “Non Blocking” isend() irecv() Sender Receiver time CPU perform task iwait() CPU waits iwait() CPU waits CPU perform task
30
30 Modes of Send The MPI standard defines four modes of send: Standard Synchronous Buffered Ready
31
31 Standard Mode (Eager send protocol used for small messages)
32
32 Synchronous Mode (Rendezvous Protocol used for large messages)
33
33 Performance Evaluation of Point to Point Communication Normally ping pong benchmarks are used to calculate: Latency: How long it takes to send N bytes from sender to receiver? Throughput: How much bandwidth is achieved? Latency is a useful measure for studying the performance of “small” messages Throughput is a useful measure for studying the performance of “large” messages
34
34 Latency on Fast Ethernet
35
35 Throughput on Fast Ethernet
36
36 Latency on Gigabit Ethernet
37
37 Throughput on GigE
38
38 Latency on Myrinet
39
39 Throughput on Myrinet
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.