CS 201 Compiler Construction Array Dependence Analysis & Loop Parallelization

Loop Parallelization Goal: Identify loops whose iterations can be executed in parallel on different processors of a shared-memory multiprocessor system. Matrix Multiplication for I = 1 to n do -- parallel for J = 1 to n do -- parallel for K = 1 to n do –- not parallel C[I,J] = C[I,J] + A[I,K]*B[K,J]

Data Dependences Flow Dependence: S1: X = …. S2: … = X Anti Dependence: S1: … = X S2: X = … Output Dependence: S1: X = … S1 δf S2 S1 δa S2 S1 δo S2 S1 S2 δf S1 S2 δa S1 S2 δo

Example: Data Dependences do I = 1, 40 S1: A(I+1) = …. S2: … = A(I-1) enddo S1: A(I-1) = … S2: … = A(I+1) S1: A(I+1) = … S2: A(I-1) = … S1 S2 δf S2 S1 δa S1 S2 δo

Sets of Dependences δf Due to A() do I = 1, 100 S: A(I) = B(I+2) + 1 T: B(I) = A(I-1) - 1 enddo S T δf Due to A() S(1): A(1) = B(3) + 1 T(1): B(1) = A(0) - 1 S(2): A(2) = B(4) + 1 T(2): B(2) = A(1) - 1 S(3): A(3) = B(5) + 1 T(3): B(3) = A(2) - 1 ………….. S(100): A(100) = B(102) + 1 T(100): B(100) = A(99) - 1 Set of iteration pairs associated with this dependence: {(i,j): j=i+1, 1<=i<=99} Dependence distance: j-i=1 constant in this case.

Nested Loops level 1: do I1 = 1, 100 level 2: do I2 = 1, 50 S: A(I1,I2) = A(I1,I2-1) + B(I1,I2) enddo Value computed by S in an iteration (i1,i2) is same as the value used in an iteration (j1,j2): A(i1,i2)A(j1,j2-1) iff i1=j1 and i2=j2-1 S is flow dependent on itself at level 2 (corresponds to inner loop) for fixed value of I1 dependence exists between different iterations of second loop (inner loop). iteration pairs: {((i1,i2),(j1,j2)): j1=i1, j2=i2+1, 1<=i1<=100, 1<=i2<=49}

Nested Loops Contd.. {((i1,i2),(j1,j2)): j1=i1, j2=i2+1, Iteration pairs: {((i1,i2),(j1,j2)): j1=i1, j2=i2+1, 1<=i1<=100, 1<=i2<=49} Dependence distance vector: (j1-i1,j2-i2) = (0,1) There is no dependence at level 1.

Computing Dependences - Formulation level 1: do I1 = 1, 8 level 2: do I2 = max(I1-3,1), min(I1,5) A(I1+1,I2+1) = A(I1,I2) + B(I1,I2) enddo Can we find iterations (i1,i2) & (j1,j2) such that i1+1=j1; i2+1=j2 and 1<=i1<=8 1<=j1<=8 i1-3<=i2<=i1 j1-3<=j2<=j1 i1-3<=i2<=5 j1-3<=j2<=5 1<=i2<=i1 1<=j2<=j1 1<=i2<=5 1<=j2<=5 i1, i2, j1, j2 are integers.

Computing Dependences Contd.. Dependence testing is an integer programming problem  NP-complete Solutions trade-off Speed and Accuracy of the solver. False positives: imprecise tests may report dependences that actually do not exist – conservative is to report false positives but never miss a dependence. DependenceTests: extreme value test; GCD test; Generalized GCD test; Lambda test; Delta test; Power test; Omega test etc…

Extreme Value Test Approximate test which guarantees that a solution exists but it may be a real solution not an integer solution. Let f: Rn  R st f is bounded in a set S contained in Rn Let b be a lower bound of f in S and B be an upper bound of f in S: b ≤ f(ā) ≤ B for any ā ε S contained in Rn For a real number c, the equation f(ā) = c will have solutions iff b ≤ c ≤ B.

Extreme Value Test Contd.. Example: f: R2  R; f(x,y) = 2x + 3y S = {(x,y): 0<=x<=1 & 0<=y<=1} contained in R2 lower bound, b=0; upper bound, B=5 1. Is there a real solution to the equation 2x + 3y = 4 ? Yes. (0.5,1) is a solution, there are many others. 2. Is there an integer solution in S ? No. f(0,0)=0; f(0,1)=3; f(1,0)=2; & f(1,1)=5. For none of the integer points in S, f(x,y)=4. If there are no real solutions, there are no integer solutions. If there are real solutions, then integer solutions may or may not exist.

Extreme Value Test Contd.. Example: DO I = 1, 10 DO J = 1, 10 A[10*I+J-5] = ….A[10*I+J-10]…. 10*I1+J1-5 = 10*I2+J2-10 10*I1-10*I2+J1-J2 = -5 f: R4  R; f(I1,I2,J1,J2) = 10*I1-10*I2+J1-J2 1<=I1,I2,J1,J2<=10; lower bound, b=-99; upper bound, B=+99 since -99 <= -5 <= +99 there is a dependence.

Extreme Value Test Contd..

Extreme Value Test Contd..

Extreme Value Test Contd..

Extreme Value Test Contd..

Nested Loops – Multidimensional Arrays

Nested Loops Contd..