PVM and MPI

Message Passing Paradigm Each processor in a message passing program runs a separate process (sub-program, task) written in a conventional sequential language all variables are private communicate via special subroutine calls

Messages Messages are packets of data moving between processes The message passing system has to be told the following information: Sending process Source location Data type Data length Receiving process(es) Destination location Destination size

What is Master/Slave principle? The master has the control over the running application, it controls all data and it calls the slaves to do their work PROGRAM IF (process = master) THEN master-code ELSE slave-code ENDIF END

PVM and MPI Goals PVM MPI A distributed operating system Portability Heterogeneity Handling communication failures MPI A library for writing application program, not a distributed operating system portability High Performance Heterogeneity Well-defined behavior MPI implementations: LAM, MPICH,…

MPI - Message Passing Interface What is MPI ? MPI - Message Passing Interface MPI is a specification for the developers and users of message passing libraries. By itself, it is NOT a library - but rather the specification of what such a library should be. A fixed set of processes is created at program initialization, one process is created per processor Each process knows its personal number (rank) Each process knows number of all processes Each process can communicate with other processes Process can’t create new processes

General MPI Program Structure

Groups and Communicators

Environment Management Routines MPI_Init: Initializes the MPI execution environment. MPI_Comm_size: Determines the number of processes in the group associated with a communicator. MPI_Comm_rank: Determines the rank of the calling process within the communicator. MPI_Send / MPI_ISend MPI_Recv / MPI_IRecv MPI_Abort: Terminates all MPI processes associated with the communicator. MPI_Finalize: Terminates the MPI execution environment.

Format in C Language #include "mpi.h" #include <stdio.h> int main(argc,argv) int argc; char *argv[]; { int numtasks, rank, rc; rc = MPI_Init(&argc,&argv); if (rc != MPI_SUCCESS) printf ("Error starting MPI program. Terminating.\n"); MPI_Abort(MPI_COMM_WORLD, rc); } MPI_Comm_size(MPI_COMM_WORLD,&numtasks); MPI_Comm_rank(MPI_COMM_WORLD,&rank); printf ("Number of tasks= %d My rank= %d\n", numtasks,rank); /******* do some work *******/ MPI_Finalize();

Example #include "mpi.h" #include <stdio.h> int main(argc,argv) int argc; char *argv[]; { int numtasks, rank, dest, source, rc, count, tag=1; char inmsg, outmsg='x'; MPI_Status Stat; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD, &numtasks); MPI_Comm_rank(MPI_COMM_WORLD, &rank); if (rank == 0) { dest = 1; source = 1; rc = MPI_Send(&outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD); rc = MPI_Recv(&inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat); } else if (rank == 1) { dest = 0; source = 0; rc = MPI_Recv(&inmsg, 1, MPI_CHAR, source, tag, MPI_COMM_WORLD, &Stat); rc = MPI_Send(&outmsg, 1, MPI_CHAR, dest, tag, MPI_COMM_WORLD); } rc = MPI_Get_count(&Stat, MPI_CHAR, &count); printf("Task %d: Received %d char(s) from task %d with tag %d \n", rank, count, Stat.MPI_SOURCE, Stat.MPI_TAG); MPI_Finalize(); }

MPI Broadcast

MPI Scatter and Gather

MPI All to All

MPI All to Gather

MPI Reduce and All Reduce

PVM - Parallel Virtual Machine What is PVM ? PVM - Parallel Virtual Machine Is a software package that allows a heterogeneous collection of workstations (host pool) to function as a single high performance parallel machine (virtual) PVM, through its virtual machine provides a simple yet useful distributed operating system It has daemon running on all computers making up the virtual machine

PVM Daemon (pvmd) The pvmd serves as a message router and controller One pvmd runs on each host of a virtual machine The first pvmd (started by hand) is designated the master, while the others (started by the master) are called slaves Only the master can start new slaves and add them to configuration or delete slave hosts from the machine

Resource Control PVM is inherently dynamic in nature, and it has a rich set of resource control functions. Hosts can be added or deleted load balancing task migration fault tolerance efficiency MPI is specifically designed to be static in nature to improve performance

Message Passing operations MPI : Rich message support PVM: Simple message passing

Fault Tolerance: MPI MPI standard is based on a static model If a member of a group failed for some reason, the specification mandated that rather than continuing which would lead to unknown results in a doomed application, the group is invalidated and the application halted in a clean manner. In simple if something fails, everything does.

Fault Tolerance: MPI

Fault Tolerance: MPI Failed Node There is a failure and…

Fault Tolerance: MPI Failed Node … the application is shut down

Fault Tolerance: PVM PVM supports a basic fault notification scheme: it doesn’t automatically recover an application after a crash, but it does provide notification primitives to allow fault-tolerant applications to be built The Virtual Machine is dynamically reconfigurable A pvmd can recover from the loss of any foreign pvmd except the master. The master must never crash

Fault Tolerance: PVM Virtual Machine

Fault Tolerance: PVM Virtual Machine Failed Node

Fault Tolerance: PVM Virtual Machine Fast host delete or recovery from fault

Conclusion PVM MPI Each API has it’s unique strengths Virtual machine concept Simple message passing Communication topology unspecified Interoperate across host architecture boundaries Portability over performance Resource and process control Robust fault tolerance MPI No such abstraction Rich message support Support logical communication topologies Some realizations do not interoperate across architectural boundaries Performance over flexibility Primarily concerned with messaging More susceptible to faults

Conclusion PVM is better for: MPI is better for: Each API has it’s unique strengths PVM is better for: Heterogeneous cluster, resource and process control The size of cluster and the time of program’s execution are great MPI is better for: Supercomputers (PVM is not supported) Application for MPP Max performance Application needs rich message support