1. Distributed Systems CS 15-440
Programming Models- Part I
Lecture 14, Oct 20, 2014
Mohammad Hammoud

2. Today… Last Session: Election Algorithms Midterm Overview
Today’s Session:
Programming Models – Part I
Announcements:
P2 is due on Thursday, Oct 23 by 11:59PM (it is worth 10% of the overall grade)
Midterm grades are out
P1 and PS3 grades will be out today as well
Mid-semester grades will be posted today (before 11:00PM) on the system

3. Discussion on Programming Models
Objectives
Discussion on Programming Models
MapReduce, Pregel and GraphLab
Message Passing Interface (MPI)
Types of Parallel Programs
Traditional models of parallel programming
Parallel computer architectures
Why parallelism?
Why parallelism?

4. Amdahl’s Law We parallelize our programs in order to run them faster
How much faster will a parallel program run?
Suppose that the sequential execution of a program takes T1 time units and the parallel execution on p processors takes Tp time units
Suppose that out of the entire execution of the program, s fraction of it is not parallelizable while 1-s fraction is parallelizable
Then the speedup (Amdahl’s formula):

5. Amdahl’s Law: An Example
Suppose that 80% of your program can be parallelized and that you use 4 processors to run your parallel version of the program
The speedup you can get according to Amdahl’s law is:
Although you use 4 processors you cannot get a speedup more than 2.5 times!

6. Real Vs. Actual Cases
Amdahl’s argument is too simplified to be applied to real cases
When we run a parallel program, there are a communication overhead and a workload imbalance among processes (in general) 20 80 20 80
Serial
Serial
Parallel 20 20
Parallel 20 20
Process 1
Process 1
Process 2
Process 2
Cannot be parallelized
Process 3
Process 3
Cannot be parallelized
Can be parallelized
Communication overhead
Process 4
Can be parallelized
Process 4
Load Unbalance
1. Parallel Speed-up: An Ideal Case
2. Parallel Speed-up: An Actual Case

7. Guidelines
In order to efficiently benefit from parallelization, we ought to follow these guidelines:
Maximize the fraction of our program that can be parallelized
Balance the workload of parallel processes
Minimize the time spent for communication

8. Discussion on Programming Models
Objectives
Discussion on Programming Models
MapReduce, Pregel and GraphLab
Message Passing Interface (MPI)
Types of Parallel Programs
Traditional models of parallel programming
Parallel computer architectures
Parallel computer architectures
Why parallelism?

9. Parallel Computer Architectures
We can categorize the architecture of parallel computers in terms of two aspects:
Whether the memory is physically centralized or distributed
Whether or not the address space is shared
Memory
Address Space
Shared
Individual
Centralized
SMP (Symmetric Multiprocessor)
N/A
Distributed
NUMA (Non-Uniform Memory Access)
MPP (Massively Parallel Processors)
Memory
Address Space
Shared
Individual
Centralized
UMA – SMP (Symmetric Multiprocessor)
N/A
Distributed
NUMA (Non-Uniform Memory Access)
MPP (Massively Parallel Processors)
Memory
Address Space
Shared
Individual
Centralized
SMP (Symmetric Multiprocessor)
N/A
Distributed
NUMA (Non-Uniform Memory Access)
MPP (Massively Parallel Processors)
Memory
Address Space
Shared
Individual
Centralized
SMP (Symmetric Multiprocessor)
N/A
Distributed
NUMA (Non-Uniform Memory Access)
MPP (Massively Parallel Processors)
Memory
Address Space
Shared
Individual
Centralized
SMP (Symmetric Multiprocessor)
N/A
Distributed
NUMA (Non-Uniform Memory Access)
MPP (Massively Parallel Processors)

10. Symmetric Multiprocessor
Symmetric Multiprocessor (SMP) architecture uses shared system resources that can be accessed equally from all processors
Usually, a single OS controls the SMP machine and it schedules processes and threads on processors so that the load is balanced
Processor
Processor
Processor
Processor
Cache
Cache
Cache
Cache
Bus or Crossbar Switch
Memory
I/O

11. Massively Parallel Processors
Massively Parallel Processors (MPP) architecture consists of nodes with each having its own processor, memory and I/O subsystem
An independent OS runs at each node
Interconnection Network
Processor
Processor
Processor
Processor
Cache
Cache
Cache
Cache
Bus
Bus
Bus
Bus
Memory
I/O
Memory
I/O
Memory
I/O
Memory
I/O

12. Non-Uniform Memory Access
Non-Uniform Memory Access (NUMA) architecture machines are built on a similar hardware model as MPP
NUMA typically provides a shared address space to applications using a hardware/software directory-based coherence protocol
The memory latency varies according to whether the memory is accessed directly (locally) or through the interconnect (remotely)
As in an SMP machine, usually a single OS controls the whole system

13. Discussion on Programming Models
Objectives
Discussion on Programming Models
MapReduce, Pregel and GraphLab
Message Passing Interface (MPI)
Types of Parallel Programs
Traditional Models of parallel programming
Traditional Models of parallel programming
Parallel computer architectures
Why parallelizing our programs?

14. Models of Parallel Programming
What is a parallel programming model?
A programming model is an abstraction provided by the hardware to programmers
It determines how easily programmers can specify their algorithms into parallel units of computations (i.e., tasks) that the hardware understands
It determines how efficiently parallel tasks can be executed on the hardware
Main Goal: utilize all the processors of the underlying architecture (e.g., SMP, MPP, NUMA, and Distributed Systems) and minimize the elapsed time of your program

15. Traditional Parallel Programming Models
Shared Memory
Message Passing
Message Passing

16. Shared Memory Model
In the shared memory programming model, the abstraction provided implies that parallel tasks can access any location of the memory
Accordingly, parallel tasks can communicate through reading and writing common memory locations
This is similar to threads in a single process (in traditional OSs), which share a single address space
Multi-threaded programs (e.g., OpenMP programs) use the shared memory programming model

17. Shared Memory Model Si = Serial Pj = Parallel Single Thread
Multi-Thread S1 S1
Time
Time
Spawn P1 P1 P2 P3 P3 P2 P3
Join S2
Shared Address Space P4 S2
Process
Process

18. Shared Memory Example Sequential Parallel
begin parallel // spawn a child thread
private int start_iter, end_iter, i;
shared int local_iter=4, sum=0;
shared double sum=0.0, a[], b[], c[];
shared lock_type mylock;
start_iter = getid() * local_iter;
end_iter = start_iter + local_iter;
for (i=start_iter; i<end_iter; i++)
a[i] = b[i] + c[i];
barrier;
if (a[i] > 0) {
lock(mylock);
sum = sum + a[i];
unlock(mylock); }
barrier; // necessary
end parallel // kill the child thread
Print sum;
for (i=0; i<8; i++)
a[i] = b[i] + c[i];
sum = 0;
if (a[i] > 0)
sum = sum + a[i];
Print sum;
Sequential
Parallel

19. Traditional Parallel Programming Models
Shared Memory
Shared Memory
Message Passing

20. Message Passing Model
In message passing, parallel tasks have their own local memories
One task cannot access another task’s memory
Hence, tasks have to rely on explicit messages to communicate
This is similar to the abstraction of processes in traditional OSs, which do not share address spaces
MPI programs use the message passing programming model

21. Message Passing Model Single Thread Message Passing S = Serial
P = Parallel S1 S1 S1 S1 S1
Time
Time P1 P1 P1 P1 P1 P2 S2 S2 S2 S2 P3 P4
Process 0
Process 1
Process 2
Process 3 S2
Node 1
Node 2
Node 3
Node 4
Data transmission over the Network
Process

22. Message Passing Example
id = getpid();
local_iter = 4;
start_iter = id * local_iter;
end_iter = start_iter + local_iter;
if (id == 0)
send_msg (P1, b[4..7], c[4..7]);
else
recv_msg (P0, b[4..7], c[4..7]);
for (i=start_iter; i<end_iter; i++)
a[i] = b[i] + c[i];
local_sum = 0;
if (a[i] > 0)
local_sum = local_sum + a[i];
if (id == 0) {
recv_msg (P1, &local_sum1);
sum = local_sum + local_sum1;
Print sum; }
send_msg (P0, local_sum);
for (i=0; i<8; i++)
a[i] = b[i] + c[i];
sum = 0;
if (a[i] > 0)
sum = sum + a[i];
Print sum;
Sequential
Parallel

23. Shared Memory Vs. Message Passing
Comparison between the shared memory and message passing programming models:
Aspect
Shared Memory
Message Passing
Communication
Implicit (via loads/stores)
Explicit Messages
Synchronization
Explicit
Implicit (Via Messages)
Hardware Support
Typically Required
None
Development Effort
Lower
Higher
Tuning Effort
Aspect
Shared Memory
Message Passing
Communication
Implicit (via loads/stores)
Explicit Messages
Synchronization
Explicit
Implicit (Via Messages)
Hardware Support
Typically Required
None
Development Effort
Lower
Higher
Tuning Effort
Aspect
Shared Memory
Message Passing
Communication
Implicit (via loads/stores)
Explicit Messages
Synchronization
Explicit
Implicit (Via Messages)
Hardware Support
Typically Required
None
Development Effort
Lower
Higher
Tuning Effort
Aspect
Shared Memory
Message Passing
Communication
Implicit (via loads/stores)
Explicit Messages
Synchronization
Explicit
Implicit (Via Messages)
Hardware Support
Typically Required
None
Development Effort
Lower
Higher
Tuning Effort
Aspect
Shared Memory
Message Passing
Communication
Implicit (via loads/stores)
Explicit Messages
Synchronization
Explicit
Implicit (Via Messages)
Hardware Support
Typically Required
None
Development Effort
Lower
Higher
Tuning Effort

24. Discussion on Programming Models
Objectives
Discussion on Programming Models
MapReduce, Pregel and GraphLab
Message Passing Interface (MPI)
Types of Parallel Program
Types of Parallel Program
Traditional Models of parallel programming
Parallel computer architectures
Why parallelizing our programs?

25. SPMD and MPMD
When we run multiple processes with message-passing, there are further categorizations regarding how many different programs are cooperating in parallel execution
We distinguish between two models:
Single Program Multiple Data (SPMD) model
Multiple Programs Multiple Data (MPMD) model

26. SPMD
In the SPMD model, there is only one program and each process uses the same executable working on different sets of data
a.out
Node 1
Node 2
Node 3

27. MPMD
The MPMD model uses different programs for different processes, but the processes collaborate to solve the same problem
MPMD has two styles, the master/worker and the coupled analysis
a.out
b.out
a.out
b.out
c.out
a.out= Structural Analysis,
b.out = fluid analysis and
c.out = thermal analysis
Example
Node 1
Node 2
Node 3
Node 1
Node 2
Node 3
1. MPMD: Master/Slave
2. MPMD: Coupled Analysis

28. An Example A Sequential Program
Read array a() from the input file
Set is=1 and ie=6 //is = index start and ie = index end
Process from a(is) to a(ie)
Write array a() to the output file is ie a
Colored shapes indicate the initial values of the elements
Black shapes indicate the values after they are processed a a

29. Under which category does this program lie?
An Example
Under which category does this program lie?
Process 0
Read array a() from the input file
Get my rank
If rank==0 then is=1, ie=2 If rank==1 then is=3, ie=4 If rank==2 then is=5, ie=6
Process from a(is) to a(ie)
Gather the results to process 0
If rank==0 then write array a() to the output file
Process 1
Read array a() from the input file
Get my rank
If rank==0 then is=1, ie=2 If rank==1 then is=3, ie=4 If rank==2 then is=5, ie=6
Process from a(is) to a(ie)
Gather the results to process 0
If rank==0 then write array a() to the output file
Process 2
Read array a() from the input file
Get my rank
If rank==0 then is=1, ie=2 If rank==1 then is=3, ie=4 If rank==2 then is=5, ie=6
Process from a(is) to a(ie)
Gather the results to process 0
If rank==0 then write array a() to the output file is ie is ie is ie a a a a a a a

30. Concluding Remarks To summarize, keep the following 3 points in mind:
The purpose of parallelization is to reduce the time spent on computation– A caveat, however, is that communication might increase as the cluster size is scaled up
Ideally, the parallel program is p times faster than the sequential program, where p is the number of processes involved in the parallel execution, but this is not always achievable
Message-passing is the tool to consolidate what parallelization has separated. It should not be regarded as the parallelization itself

31. Discussion on Programming Models
Objectives
Discussion on Programming Models
MapReduce, Pregel and GraphLab
Message Passing Interface (MPI)
Message Passing Interface (MPI)
Types of Parallel Programs
Traditional Models of parallel programming
Parallel computer architectures
Why parallelizing our programs?

32. Message Passing Interface
In this part, the following concepts of MPI will be described:
Basics
Point-to-point communication
Collective communication

33. Message Passing Interface
In this part, the following concepts of MPI will be described:
Basics
Point-to-point communication
Collective communication

34. What is MPI?
The Message Passing Interface (MPI) is a message passing library standard  for writing message passing programs
The goal of MPI is to establish a portable, efficient, and flexible standard for message passing
By itself, MPI is NOT a library - but rather the specification of what such a library should be
MPI is not an IEEE or ISO standard, but has in fact, become the industry standard for writing message passing programs on HPC platforms

35. Reasons for using MPI Reason Description Reason Description Reason
Standardization
MPI is the only message passing library which can be considered a standard. It is supported on virtually all HPC platforms
Portability
There is no need to modify your source code when you port your application to a different platform that supports the MPI standard
Performance Opportunities
Vendor implementations should be able to exploit native hardware features to optimize performance
Functionality
Over 115 routines are defined
Availability
A variety of implementations are available, both vendor and public domain
Reason
Description
Standardization
MPI is the only message passing library which can be considered a standard. It is supported on virtually all HPC platforms
Portability
There is no need to modify your source code when you port your application to a different platform that supports the MPI standard
Performance Opportunities
Vendor implementations should be able to exploit native hardware features to optimize performance
Functionality
Over 115 routines are defined
Reason
Description
Standardization
MPI is the only message passing library which can be considered a standard. It is supported on virtually all HPC platforms
Portability
There is no need to modify your source code when you port your application to a different platform that supports the MPI standard
Reason
Description
Standardization
MPI is the only message passing library which can be considered a standard. It is supported on virtually all HPC platforms
Reason
Description
Standardization
MPI is the only message passing library which can be considered a standard. It is supported on virtually all HPC platforms
Portability
There is no need to modify your source code when you port your application to a different platform that supports the MPI standard
Performance Opportunities
Vendor implementations should be able to exploit native hardware features to optimize performance

36. The Programming Model
MPI is an example of a message passing programming model
MPI is now used on just about any common parallel architecture including MPP, SMP clusters, workstation clusters and heterogeneous networks
With MPI the programmer is responsible for correctly identifying parallelism and implementing parallel algorithms using MPI constructs

37. Communicators and Groups
MPI uses objects called communicators/groups to define which collection of processes may communicate with each other to solve a certain problem
Most MPI routines require you to specify a communicator as an argument
The communicator MPI_COMM_WORLD is often used in calling communication subroutines
MPI_COMM_WORLD is the predefined communicator that includes all of your MPI processes

38. Ranks
Within a communicator, every process has its own unique, integer identifier referred to as rank, assigned by the system when the process initializes
A rank is sometimes called a task ID. Ranks are contiguous and begin at zero
Ranks are used by the programmer to specify the source and destination of messages
Ranks are often also used conditionally by the application to control program execution (e.g., if rank=0 do this / if rank=1 do that)

39. Multiple Communicators
It is possible that a problem consists of several sub-problems where each can be solved concurrently
This type of application is typically found in the category of MPMD coupled analysis
We can create a new communicator for each sub-problem as a subset of an existing communicator
MPI allows you to achieve that by using MPI_COMM_SPLIT

40. Example of Multiple Communicators
Consider a problem with a fluid dynamics part and a structural analysis part, where each part can be computed in parallel
MPI_COMM_WORLD
Comm_Fluid
Comm_Struct
Rank=0
Rank=1
Rank=0
Rank=4
Rank=1
Rank=5
- In the first bullet, this happens most of the time. Not all times.
Rank=2
Rank=3
Rank=2
Rank=6
Rank=3
Rank=7
Ranks within MPI_COMM_WORLD are printed in red
Ranks within Comm_Fluid are printed with green
Ranks within Comm_Struct are printed with blue

41. Next Class Discussion on Programming Models
MapReduce, Pregel and GraphLab
Message Passing Interface (MPI)
Types of Parallel Programs
Traditional Models of parallel programming
Parallel computer architectures
Programming Models- Part II
Why parallelizing our programs?