1. Bernstein’s Conditions

2. Techniques to Exploit Parallelism in Sequential Programming Hierarchy of levels of parallelism: Procedure or Methods Statements Expressions

3. Techniques to Exploit Parallelism in Sequential Programming Expression Level (Implicit Parallelism) Using the associative, commutative, and distributive properties of arithmetic expressions, object code will be produced where more than one instruction can be executed in parallel. Associativity: Z = ([(A+B)+C]+D)  Z = (A+B) + (C+D) + + + + + +

4. Techniques to Exploit Parallelism in Sequential Programming Statement Level This level is known as the D-structure level (for Dijkstra). Statements can be assigned into one of three categories: - Sequence: S1, S2 - Selection: IF b THEN S1 ELSE S2 - Iteration: (while, repeat, …) Bernstein, A.J. (1966), “Analysis of Programs for Parallel Processing, IEEE transactions of computers”, EC-15, pg. 757-762

5. Sequence Example S1: x  a+b Each instruction has an input space and output space, IS and OS respectively. IS is defined as the variables in the instruction that do not change after executing the instruction. For the example above IS= {a,b}. OS is defined as the variables in the instruction that change after executing the instruction. For the example above OS={x}

6. Berstein’s Conditions Given S1 and S2 IF IS(S1) & OS(S2) =  (Data Dependent) & OS(S1) & IS(S2) =  (Anti Dependent) & OS(S1) & OS(S2) =  (Output Dependent) THEN S1 and S2 can be executed in parallel

7. Iteration Example for i  1 to 6 do A[i]  B[i] + C[i] Using vectorization (expansion) we can write as A[1]  B[1]+C[1] A[2]  B[2]+C[2] A[3]  B[3]+C[3]… Applying Berstein’s laws tell us that the above instructions can be executed in parallel.

8. Selection Given A  B+1 IF A>5 C  A+1; Go to 5 ELSE D  A+C; Go to 7 END IF 5: E  A+1 6: F  A+1 To exploit parallelism the program can be written as A  B+1

9. Program Transformation FOR I  1 TO n do X  A[I] + B[I]. Y[I]  2 * X. X  C[I] / D[I]. P  X + 2 ENDFOR FOR I  1 TO n do X  A[I] + B[I]. Y[I]  2 * X. XX  C[I] / D[I]. P  XX + 2 ENDFOR removes data dependency data dependency 

10. Scalar Expansion FOR I  1 TO n do X  A[I] + B[I]. Y[I]  2 * X. ENDFOR FOR I  1 TO n do X  A[I] + B[I]. Y[I]  2 * X[I]. ENDFOR removes data dependency data dependency 

11. Loop Unrolling FOR I  1 TO n do X[I]  A[I] * B[I] ENDFOR  X[1]  A[1] * B[1] X[2]  A[2] * B[2]. X[n]  A[n] * B[n]

12. Loop Fusion or Jamming FOR I  1 TO n do X[I]  Y[I] * Z[I] ENDFOR FOR I  1 TO n do M[I]  P[I] + X[I] ENDFOR a) FOR I  1 TO n do X[I]  Y[I] * Z[I] M[I]  P[I] + X[I] ENDFOR b) FOR I  1 TO n do M[I]  P[I] + Y[I] * Z[I] ENDFOR 