CPE779: Shared Memory and OpenMP Based on slides by Laxmikant V. Kale and David Padua of the University of Illinois

Introduction  Certain categories of programs  Use arrays and loops  Natural to do the loops in parallel  Such programs should be easy to express in parallel  OpenMP: Shared memory programming model  Some people think:  Easy to convert sequential programs to parallel via OpenMP  It works well on shared memory hardware  Others think: its hard to get good performance  Probably both right to some degree.. 2CPE 779 Parallel ComputingSpring 2010

OpenMP Spring 2010CPE 779 Parallel Computing3  Meant for C, C++ and FORTRAN  Standardizes 15 years of symmetric multiprocessing (SMP) practice  Set of compiler directives and library routines for parallel programmers

Programming Model Spring 2010CPE 779 Parallel Computing4  Fork-Join Parallelism: Single flow of control:  Master thread spawns slave threads as needed  Sequential sections should be small

Thread Interactions: Race conditions Spring 2010CPE 779 Parallel Computing5  Threads communicate by sharing variables  Unintended sharing can lead to race conditions  Race condition: when the program’s outcome changes as the threads are scheduled differently.  To control race conditions: use synchronization to protect data conflicts.  Synchronization is expensive so: change how data is stored to minimize the need for synchronization.

General Rules  Most of the constructs are compiler directives  For C and C++ #pragma omp construct [clause [clause]…]  For FORTRAN C$OMP construct [clause [clause]…] !$OMP construct [clause [clause]…] *$OMP construct [clause [clause]…]  Directives are ignored by non-OpenMP compilers 6CPE 779 Parallel ComputingSpring 2010

PARALLEL DO/FOR: Syntax Spring 2010CPE 779 Parallel Computing7  Specifies that the iterations of the immediately following do loop (Fortran) or for loop (C/C++) be executed in parallel. #pragma omp parallel for [clause [clause...]] for (index = first; test_expr; increment_expr) { for (index = first; test_expr; increment_expr) { body of the loop body of the loop }

Single precision a*x + y Spring 2010CPE 779 Parallel Computing8 subroutine saxpy (z, a, x, y, n) integer i, n real z(n), a, x(n), y(n) !$omp parallel do do i = 1, n z(i) = a * x(i) + y(i) enddoreturnend †SAXPY (Scalar Alpha X Plus Y) is one of the functions in the Basic Linear Algebra Subprograms (BLAS) package, and is a common operation in computations with vector processors. (saxpy†)

PARALLEL DO: Execution Spring 2010CPE 779 Parallel Computing9 TIME Master thread executes serial portion of code Master thread enters saxpy routine Master thread encounters parallel do directive. Creates slave threads (How many?) Master and slave threads divide iterations of parallel do loop and execute them concurrently (How?) Implicit synchronization: wait for all threads to finish their allocation of iterations Master thread resumes execution after the do loop. Slave threads disappear

Number of threads Spring 2010CPE 779 Parallel Computing10  Use environment variable  Use omp_set_num_threads() function setenv OMP_NUM_THREADS 8 (on Unix machines) subroutine saxpy (z, a, x, y, n) integer i, n real z(n), a, x(n), y(n) !$call omp_set_num_threads(4) !$omp parallel do do i = 1, n z(i) = a * x(i) + y(i) enddoreturnend

Loop Scheduling: User Control  The manner in which iterations of a parallel loop are assigned to threads is called the loop’s schedule.  Unless specified otherwise, the default schedule is used which assigns iterations to threads as evenly as possible (good enough for saxpy)  Alternative user-specified schedules possible - more on this later 11CPE 779 Parallel ComputingSpring 2010

Communication between threads Spring 2010CPE 779 Parallel Computing12  Six clause types allow the programmer to specify how data is shared between threads executing a parallel do (data scope clauses):  private: private (list)  shared: shared(list)  default: default (private | shared | none)  reduction:reduction(intrinsic operator : list)  firstprivate: firstprivate(list)  lastprivate: lastprivate(list)

Default data sharing Spring 2010CPE 779 Parallel Computing13  Most variables are shared by default  Global variables are shared among threads  Exception: loop index variables are private by default  Stack variables in function calls from parallel regions are thread-private

Example Spring 2010CPE 779 Parallel Computing14 double x, y; int i, j, m, n, maxiter; int depth[300][200]; extern int mandel_val(); n = 300; m = 200; maxiter = 200; #pragma omp parallel for private(j, x, y) for (i = 1; i <= m; i++) for (j = 1; j <= n; j++) { for (j = 1; j <= n; j++) { x = i/ (double) m; x = i/ (double) m; y = j/ (double) n; y = j/ (double) n; depth[j][i] = mandel_val(x, y, maxiter); depth[j][i] = mandel_val(x, y, maxiter); }

Private clause: Details  Each thread has its own copy of all variables declared private.  Private variables (implicit or explicit) are uninitialized when a thread starts. (some exceptions)  The value of a private variable is unavailable to the master thread after a parallel loop terminates.  firstprivate and lastprivate are solutions 15CPE 779 Parallel ComputingSpring 2010

Example Spring 2010CPE 779 Parallel Computing16  What’s wrong with this example: program wrong IS = 0 C$OMP PARALLEL DO PRIVATE(IS) DO J=1,1000 … … = IS … 800 CONTINUE print *, IS

Firstprivate and lastprivate Spring 2010CPE 779 Parallel Computing17  firstprivate (list) initializes each thread’s copy of a private variable to the value of the master thread’s copy, for all variables in list.  lastprivate (list) writes back to the master’s copy the value contained in the private copy belonging to the thread that executed the sequentially last iteration of the loop, for all variables in list.

Corrected Example Spring 2010CPE 779 Parallel Computing18 program right IS = 0 C$OMP PARALLEL DO FIRSTPRIVATE(IS) LASTPRIVATE(IS) DO J=1,1000 … … = IS … 800 CONTINUE print *, IS

Another example Spring 2010CPE 779 Parallel Computing19 real x(n, n), c(n, n), y(n), z(n) x(1, 1) = … x(2, 1) = … $omp parallel do firstprivate(x) lastprivate(i, x) do i = 1, n x(1, 2) = c(i, 1) * x(1, 1) x(2, 2) = c(i, 2) * x(2, 1) ** 2 y(i) = x(2, 2) + x(1, 2) z(i) = x(2, 2) - x(1, 2) enddo y[i-1] = x(1, 2) + x (2,2)

Reduction Example Spring 2010CPE 779 Parallel Computing20 subroutine sum (values, n, s) integer n real s, values(n) s = 0 s = 0 !$omp parallel do reduction (+:s) do i = 1, n s = s + values(i) enddoreturnend Performs a reduction on the variable “s”. A private copy of s is created for each thread. Iterations of the parallel loop will be distributed in equal sized blocks to each thread in the team

Reduction Example Spring 2010CPE 779 Parallel Computing21 subroutine sum (values, n, s) integer n real s, values(n) s = 0 s = 0 !$omp parallel do reduction (+:s) do i = 1, n s = s + values(i) enddoreturnend At the end of the parallel loop construct, all threads will add their values of “s" to update the master thread's global copy.

Assigning iterations to threads  Motivation: to balance the work per thread when the work per iteration is inherently unbalanced 22CPE 779 Parallel ComputingSpring 2010 !$omp parallel do private(xkind) do i = 1, n xkind = f(i) if (xkind.lt. 10 then) call smallwork(x[i]) else call bigwork(x[i]) endifenddo

Schedule clause and general form Spring 2010CPE 779 Parallel Computing23  schedule (type[, chunk])  type = static, dynamic, guided or runtime  optional chunk = scalar integer value  static: iterations are divided as evenly as possible among all threads - simple static  static, chunk: iterations are divided into chunks of size chunk. Chunks are then assigned in round robin fashion to threads - interleaved

Schedule clause: dynamic or guided scheduling Spring 2010CPE 779 Parallel Computing24  dynamic, chunk: iterations are divided into chunks of size chunk (1 if unspecified) and are assigned to threads dynamically after an initial round robin assignment - simple dynamic  guided, chunk: chunk size decreases exponentially from an implementation dependent value to chunk (1 if unspecified). Chunks are assigned dynamically - guided self scheduling