Embarrassingly Parallel (or pleasantly parallel) Characteristics Domain divisible into a large number of independent parts. Little or no communication between processors Processor performs the same calculation independently “Nearly embarrassingly parallel” – Communication is limited to distributing and gathering data – Computation dominates the communication Definition: Problems that scale well to thousands of processors

Embarrassingly Parallel Examples P0P1P2 Embarrassingly Parallel Application P1P2P3 P0 Send Data Receive Data Nearly Embarrassingly Parallel Application

Low Level Image Processing Storage – A two dimensional array of pixels. – One bit, one byte, or three bytes may represent pixels – Operations may only involve local data Image Applications – Shift: newX=x+delta; newY=y+delta – Scale: newX = x*scale; newY = y*scale – Rotate a point about the origin newX = x cos  +y sin  newY=-xsin  +ycos  – Clip newX = x if min x <=x< max x ; 0 otherwise newY = y if min y <=y<=max y ; 0 otherwise Note: Does not include communication to a graphics adapter

Non-trivial Image Processing Smoothing – A function that captures important patterns, while eliminating noise or artifacts – Linear smoothing: Apply a linear transformation to a picture – Convolution: P new (x,y) = ∑ j=0,m-1 ∑ k=0,n-1 P(x,y,j,k) old f(j,k) Edge Detection – A function that searches for discontinuities or variations in depth, surface, or color – Purpose: Significantly reduce follow-up processing – Uses: Pattern recognition and computer vision – One approach: differentiate to identify large changes Pattern Matching – Match an image against a template or a group of features – Example: ∑i=0,X∑j=1,Y (Picture(x+I,y+i) – Template(x,y)) Note: This is another digital signal processing application

Array Storage The C language stores arrays in row-major order, Matlab and Fortran use column-major order Loops can be extremely slow in C if the outer loop processes columns due to the system memory cache operation int A[2][3] = { {1, 2, 3}, {4, 5, 6} }; In memory: int A[2][3][2] = {{{1,2}, {3,4}, {5,6}}, {{7,8}, {9,10}, {11,12}}}; In memory : Translate multi-dimension indices to single dimension offsets – Two Dimensions: offset = row*COLS + column – Three Dimensions: offset = i*DIM2*DIM3 + j*DIM3+ k – What is the formula for four dimensions? Cow –major (left most dimensions) are stored one after another Column-major (right most dimensions) are stored one after another

Process Partitioning Note: 128 rows per displayed cell Note: 128 columns per displayed cell Pixel 21 Rows 0: 0-7 2: :16-23 Partitioning might assign groups of rows or columns to processors Pixel 2053 Row 2, column 5 Rows 0: :

Typical Static Partitioning Master – Scatter or broadcast the image along with assigned processor rows – Gather the updated data back and perform final updates if necessary Slave – Receive Data – Compute translated coordinates – Perform collective gather operation Questions – How does the master decide how much to assign to each processor? – Is the load balanced (all processors working equally)? Notes on the Text shift example – Employs individual sends/receives, which is much slower – However, if coordinate positions change or results do not represent contiguous pixel positions, this might be required

Mandelbrot Set Z 0 = * i For each (x,y) from [-2,+2] Iterate z n until either The iteration stops when the iteration count reaches a limit (in the set) Z n is out of bounds ( |z n |>2 (not in the set) Save the iteration count which will map to a display color Definition: Those points C = (x,y) = x + iy in the complex plane that iterate with a function (normally: z n+1 = z n 2 + C) converge to a finite value Implementation Complex plane Display horizontal axis: real values vertical axis: imaginary values

Scaling and Zooming Display range of points – From c min = x min + iy min to c max = x max + iy max Display range of pixels – From the pixel at (0,0) to the pixel at (ROWS, COLUMNS) Pseudo code For row = row min to row end For col = 0 to COLUMNS c y = y min +(y max -y min )* row/ROWS c x = x min +(x max -x min )* col/COLUMNS color = mandelbrot(c x, c y ) picture[COLUMNS*row+col] = color

Pseudo code ( mandelbrot(c x,c y ) ) SET z = z real + i*z imaginary = 0 + i0 SET iterations = 0 DO SET z = z 2 + C // temp = z real ; z real =z real 2 –z imaginary 2 + c x // z imaginary = 2 * temp * z imaginary + c y SET value = z real 2 + z imaginary 2 iterations++ WHILE value<=4 and iterations<max RETURN iterations Notes: 1.The final iteration count determines each point’s color 2.Some points converge quickly; others slowly, and others not at all 3.Non-converging points are in the Mandelbrot Set (black on the previous slide) 4.Note 4 ½ = 2, so we don't need to compute the square root when setting value

Parallel Implementation Load-balancing – Algorithms used to avoid processors from becoming idle – Note: A balanced load does NOT require even same work loads Static Approach – The load is assigned once at the start of the run – Mandelbrot: assign each processor a group of rows – Deficiencies: Not load balanced Dynamic Approach – The load is dynamically assigned during the run – Mandelbrot: Slaves ask for work when they complete a section Both the Static and Dynamic algorithms are examples of load balancing

The Dynamic Approach 1.The Master's work is increased somewhat a.Must send rows when receive requests from slaves b.Must be responsive to slave requests. A separate thread might help or the master can make use of MPI's asynchronous receive calls. 2.Termination a.Slaves terminate when receiving "no work" indication in received messages b.The master must not terminate until all of the slaves complete 3.Partitioning of the load a.Master receives blocks of pixels, Slaves receive ranges of (x,y) ranges b.Partitions can be in columns or in rows. Which is better? 4.Refinement: Ask for work before completion (double buffering)

Monte Carlo Methods Pseudo-code (Throw darts to converge at a solution) 1.Compute a definite integral While more iterations needed pick a random point total += f(x) result = 1/iterations * total 2.Calculation of PI While more iterations needed Randomly pick a point If point is in circle within++ Compute PI = 4 * within / iterations Using the upper right quadrant eliminates the 4 in the equation Note: Parallel programs shouldn't use the standard random number generator 1/N ∑ 1 N f(pick.x) (x max – x min )

Computation of PI ∫(1-x 2 ) 1/2 dx = π/4; 0<=x<=1∫(1-x 2 ) 1/2 dx = π; -1<=x<=1 Total points/Points within = Total Area/Area in shape Questions: How to handle the boundary condition? What is the best accuracy that we can achieve? Within if (point.x 2 + point.y 2 ) ≤ 1

Parallel Random Number Generator Numbers of a pseudo random sequence are – Uniformly, large period, repeatable, statistically independent – Each processor must generate a unique sequence – Accuracy depends upon random sequence precision Sequential linear generator (a and m are prime; c=0) 1.X i+1 = (ax i +c) mod m (ex: a=16807, m=2 31 – 1, c=0) 2.Many other generators are possible Parallel linear generator with unique sequences 1.X i+k = (Ax i + C) mod m where k is the "jump" constant 2.A=a k, C=c (a k-1 + a k-2 + … + a 1 + a 0 ) 3.if k = P, we can compute A and C and the first k random numbers to get started x1x1 x2x2 x P-1 xPxP x P+1 x 2P-2 x 2P-1 Parallel Random Sequence