1. Stanford University CS243 Winter 2006 Wei Li 1 Data Dependences and Parallelization

2. CS243 Winter 2006 Stanford University 2 Agenda Introduction Introduction Single Loop Single Loop Nested Loops Nested Loops Data Dependence Analysis Data Dependence Analysis

3. CS243 Winter 2006 Stanford University 3 Motivation DOALL loops: loops whose iterations can execute in parallel DOALL loops: loops whose iterations can execute in parallel New abstraction needed New abstraction needed Abstraction used in data flow analysis is inadequate Abstraction used in data flow analysis is inadequate Information of all instances of a statement is combined Information of all instances of a statement is combined for i = 11, 20 a[i] = a[i] + 3

4. CS243 Winter 2006 Stanford University 4 Examples for i = 11, 20 a[i] = a[i] + 3 Parallel for i = 11, 20 a[i] = a[i-1] + 3 Parallel?

5. CS243 Winter 2006 Stanford University 5 Examples for i = 11, 20 a[i] = a[i] + 3 Parallel for i = 11, 20 a[i] = a[i-1] + 3 for i = 11, 20 a[i] = a[i-10] + 3 Not parallel Parallel?

6. CS243 Winter 2006 Stanford University 6 Agenda Introduction Introduction Single Loop Single Loop Nested Loops Nested Loops Data Dependence Analysis Data Dependence Analysis

7. CS243 Winter 2006 Stanford University 7 Data Dependence of Scalar Variables True dependence True dependence Anti-dependence Anti-dependence a = = a a = Output dependence Output dependence Input dependence Input dependence a = = a

8. CS243 Winter 2006 Stanford University 8 Array Accesses in a Loop for i= 2, 5 a[i] = a[i] + 3 a[2] a[3] a[4] a[5] read write

9. CS243 Winter 2006 Stanford University 9 Array Anti-dependence for i= 2, 5 a[i-2] = a[i] + 3 a[2] a[0] a[3] a[1] a[4] a[2] a[5] a[3] read write

10. CS243 Winter 2006 Stanford University 10 Array True-dependence for i= 2, 5 a[i] = a[i-2] + 3 a[0] a[2] a[1] a[3] a[2] a[4] a[3] a[5] read write

11. CS243 Winter 2006 Stanford University 11 Dynamic Data Dependence Let o and o’ be two (dynamic) operations Let o and o’ be two (dynamic) operations Data dependence exists from o to o’, iff Data dependence exists from o to o’, iff either o or o’ is a write operation either o or o’ is a write operation o and o’ may refer to the same location o and o’ may refer to the same location o executes before o’ o executes before o’

12. CS243 Winter 2006 Stanford University 12 Static Data Dependence Let a and a’ be two static array accesses (not necessarily distinct) Let a and a’ be two static array accesses (not necessarily distinct) Data dependence exists from a to a’, iff Data dependence exists from a to a’, iff either a or a’ is a write operation either a or a’ is a write operation There exists a dynamic instance of a (o) and a dynamic instance of a’ (o’) such that There exists a dynamic instance of a (o) and a dynamic instance of a’ (o’) such that o and o’ may refer to the same location o and o’ may refer to the same location o executes before o’ o executes before o’

13. CS243 Winter 2006 Stanford University 13 Recognizing DOALL Loops Find data dependences in loop Find data dependences in loop Definition: a dependence is loop-carried if it crosses an iteration boundary Definition: a dependence is loop-carried if it crosses an iteration boundary If there are no loop-carried dependences then loop is parallelizable If there are no loop-carried dependences then loop is parallelizable

14. CS243 Winter 2006 Stanford University 14 Compute Dependence There is a dependence between a[i] and a[i-2] if There is a dependence between a[i] and a[i-2] if There exist two iterations i r and i w within the loop bounds such that iterations i r and i w read and write the same array element, respectively There exist two iterations i r and i w within the loop bounds such that iterations i r and i w read and write the same array element, respectively There exist i r, i w, 2 ≤ i r, i w ≤ 5, i r = i w -2 There exist i r, i w, 2 ≤ i r, i w ≤ 5, i r = i w -2 for i= 2, 5 a[i-2] = a[i] + 3

15. CS243 Winter 2006 Stanford University 15 Compute Dependence There is a dependence between a[i-2] and a[i-2] if There is a dependence between a[i-2] and a[i-2] if There exist two iterations i v and i w within the loop bounds such that iterations i v and i w write the same array element, respectively There exist two iterations i v and i w within the loop bounds such that iterations i v and i w write the same array element, respectively There exist i v, i w, 2 ≤ i v, i w ≤ 5, i v -2= i w -2 There exist i v, i w, 2 ≤ i v, i w ≤ 5, i v -2= i w -2 for i= 2, 5 a[i-2] = a[i] + 3

16. CS243 Winter 2006 Stanford University 16 Parallelization Is there a loop-carried dependence between a[i] and a[i-2]? Is there a loop-carried dependence between a[i] and a[i-2]? Is there a loop-carried dependence between a[i-2] and a[i-2]? Is there a loop-carried dependence between a[i-2] and a[i-2]? for i= 2, 5 a[i-2] = a[i] + 3

17. CS243 Winter 2006 Stanford University 17 Agenda Introduction Introduction Single Loops Single Loops Nested Loops Nested Loops Data Dependence Analysis Data Dependence Analysis

18. CS243 Winter 2006 Stanford University 18 Nested Loops Which loop(s) are parallel? Which loop(s) are parallel? for i1 = 0, 5 for i2 = 0, 3 a[i1,i2] = a[i1-2,i2-1] + 3

19. CS243 Winter 2006 Stanford University 19 Iteration Space An abstraction for loops An abstraction for loops Iteration is represented as coordinates in iteration space. Iteration is represented as coordinates in iteration space. for i1 = 0, 5 for i2 = 0, 3 a[i1,i2] = 3 i1 i2

20. CS243 Winter 2006 Stanford University 20 Execution Order Sequential execution order of iterations: Lexicographic order [0,0], [0,1], …[0,3], [1,0], [1,1], …[1,3], [2,0]… Sequential execution order of iterations: Lexicographic order [0,0], [0,1], …[0,3], [1,0], [1,1], …[1,3], [2,0]… Let I = (i 1,i 2,… i n ). I is lexicographically less than I’, I<I’, iff there exists k such that (i 1,… i k-1 ) = (i’ 1,… i’ k-1 ) and i k < i’ k Let I = (i 1,i 2,… i n ). I is lexicographically less than I’, I<I’, iff there exists k such that (i 1,… i k-1 ) = (i’ 1,… i’ k-1 ) and i k < i’ k i1i1 i2i2

21. CS243 Winter 2006 Stanford University 21 Parallelism for Nested Loops Is there a data dependence between a[i1,i2] and a[i1-2,i2-1]? Is there a data dependence between a[i1,i2] and a[i1-2,i2-1]? There exist i1 r, i2 r, i1 w, i2 w, such that There exist i1 r, i2 r, i1 w, i2 w, such that 0 ≤ i1 r, i1 w ≤ 5, 0 ≤ i1 r, i1 w ≤ 5, 0 ≤ i2 r, i2 w ≤ 3, 0 ≤ i2 r, i2 w ≤ 3, i1 r - 2 = i1 w i1 r - 2 = i1 w i2 r - 1 = i2 w i2 r - 1 = i2 w

22. CS243 Winter 2006 Stanford University 22 Loop-carried Dependence If there are no loop-carried dependences, then loop is parallelizable. If there are no loop-carried dependences, then loop is parallelizable. Dependence carried by outer loop: Dependence carried by outer loop: i1 r ≠ i1 w i1 r ≠ i1 w Dependence carried by inner loop: Dependence carried by inner loop: i1 r = i1 w i1 r = i1 w i2 r ≠ i2 w i2 r ≠ i2 w This can naturally be extended to dependence carried by loop level k. This can naturally be extended to dependence carried by loop level k.

23. CS243 Winter 2006 Stanford University 23 Nested Loops Which loop carries the dependence? Which loop carries the dependence? for i1 = 0, 5 for i2 = 0, 3 a[i1,i2] = a[i1-2,i2-1] + 3 i1 i2

24. CS243 Winter 2006 Stanford University 24 Agenda Introduction Introduction Single Loop Single Loop Nested Loops Nested Loops Data Dependence Analysis Data Dependence Analysis

25. CS243 Winter 2006 Stanford University 25 Solving Data Dependence Problems Memory disambiguation is un-decidable at compile-time. Memory disambiguation is un-decidable at compile-time. read(n) for i = 0, 3 a[i] = a[n] + 3

26. CS243 Winter 2006 Stanford University 26 Domain of Data Dependence Analysis Only use loop bounds and array indices which are integer linear functions of variables. Only use loop bounds and array indices which are integer linear functions of variables. for i1 = 1, n for i2 = 2*i1, 100 a[i1+2*i2+3][4*i1+2*i2][i1*i1] = … … = a[1][2*i1+1][i2] + 3

27. CS243 Winter 2006 Stanford University 27 Equations There is a data dependence, if There is a data dependence, if There exist i1 r, i2 r, i1 w, i2 w, such that There exist i1 r, i2 r, i1 w, i2 w, such that 0 ≤ i1 r, i1 w ≤ n, 2*i1 r ≤ i2 r ≤ 100, 2*i1 w ≤ i2 w ≤ 100, 0 ≤ i1 r, i1 w ≤ n, 2*i1 r ≤ i2 r ≤ 100, 2*i1 w ≤ i2 w ≤ 100, i1 w + 2*i2 w +3 = 1, 4*i1 w + 2*i2 w = 2*i1 r + 1 i1 w + 2*i2 w +3 = 1, 4*i1 w + 2*i2 w = 2*i1 r + 1 Note: ignoring non-affine relations Note: ignoring non-affine relations for i1 = 1, n for i2 = 2*i1, 100 a[i1+2*i2+3][4*i1+2*i2][i1*i1] = … … = a[1][2*i1+1][i2] + 3

28. CS243 Winter 2006 Stanford University 28 Solutions There is a data dependence, if There is a data dependence, if There exist i1 r, i2 r, i1 w, i2 w, such that There exist i1 r, i2 r, i1 w, i2 w, such that 0 ≤ i1 r, i1 w ≤ n, 2*i1 w ≤ i2 w ≤ 100, 2*i1 w ≤ i2 w ≤ 100, 0 ≤ i1 r, i1 w ≤ n, 2*i1 w ≤ i2 w ≤ 100, 2*i1 w ≤ i2 w ≤ 100, i1 w + 2*i2 w +3 = 1, 4*i1 w + 2*i2 w - 1 = i1 r + 1 i1 w + 2*i2 w +3 = 1, 4*i1 w + 2*i2 w - 1 = i1 r + 1 No solution → No data dependence No solution → No data dependence Solution → there may be a dependence Solution → there may be a dependence

29. CS243 Winter 2006 Stanford University 29 Form of Data Dependence Analysis Data dependence problems originally contains equalities and equalities Data dependence problems originally contains equalities and equalities Eliminate inequalities in the problem statement: Eliminate inequalities in the problem statement: Replace a ≠ b with two sub-problems: a>b or a b or a<b We get We get

30. CS243 Winter 2006 Stanford University 30 Form of Data Dependence Analysis Eliminate equalities in the problem statement: Eliminate equalities in the problem statement: Replace a =b with two sub-problems: a≤b and b≤a Replace a =b with two sub-problems: a≤b and b≤a We get We get Integer programming is NP-complete, i.e. Expensive Integer programming is NP-complete, i.e. Expensive

31. CS243 Winter 2006 Stanford University 31 Techniques: Inexact Tests Examples: GCD test, Banerjee’s test Examples: GCD test, Banerjee’s test 2 outcomes 2 outcomes No → no dependence No → no dependence Don’t know → assume there is a solution → dependence Don’t know → assume there is a solution → dependence Extra data dependence constraints Extra data dependence constraints Sacrifice parallelism for compiler efficiency Sacrifice parallelism for compiler efficiency

32. CS243 Winter 2006 Stanford University 32 GCD Test Is there any dependence? Is there any dependence? Solve a linear Diophantine equation Solve a linear Diophantine equation 2*i w = 2*i r + 1 2*i w = 2*i r + 1 for i = 1, 100 a[2*i] = … … = a[2*i+1] + 3

33. CS243 Winter 2006 Stanford University 33 GCD The greatest common divisor (GCD) of integers a 1, a 2, …, a n, denoted gcd(a 1, a 2, …, a n ), is the largest integer that evenly divides all these integers. The greatest common divisor (GCD) of integers a 1, a 2, …, a n, denoted gcd(a 1, a 2, …, a n ), is the largest integer that evenly divides all these integers. Theorem: The linear Diophantine equation Theorem: The linear Diophantine equation has an integer solution x 1, x 2, …, x n iff gcd(a 1, a 2, …, a n ) divides c has an integer solution x 1, x 2, …, x n iff gcd(a 1, a 2, …, a n ) divides c

34. CS243 Winter 2006 Stanford University 34 Examples Example 1: gcd(2,-2) = 2. No solutions Example 1: gcd(2,-2) = 2. No solutions Example 2: gcd(24,36,54) = 6. Many solutions Example 2: gcd(24,36,54) = 6. Many solutions

35. CS243 Winter 2006 Stanford University 35 Multiple Equalities Equation 1: gcd(1,-2,1) = 1. Many solutions Equation 1: gcd(1,-2,1) = 1. Many solutions Equation 2: gcd(3,2,1) = 1. Many solutions Equation 2: gcd(3,2,1) = 1. Many solutions Is there any solution satisfying both equations? Is there any solution satisfying both equations?

36. CS243 Winter 2006 Stanford University 36 The Euclidean Algorithm Assume a and b are positive integers, and a > b. Assume a and b are positive integers, and a > b. Let c be the remainder of a/b. Let c be the remainder of a/b. If c=0, then gcd(a,b) = b. If c=0, then gcd(a,b) = b. Otherwise, gcd(a,b) = gcd(b,c). Otherwise, gcd(a,b) = gcd(b,c). gcd(a 1, a 2, …, a n ) = gcd(gcd(a 1, a 2 ), a 3 …, a n ) gcd(a 1, a 2, …, a n ) = gcd(gcd(a 1, a 2 ), a 3 …, a n )

37. CS243 Winter 2006 Stanford University 37 Exact Analysis Most memory disambiguations are simple integer programs. Most memory disambiguations are simple integer programs. Approach: Solve exactly – yes, or no solution Approach: Solve exactly – yes, or no solution Solve exactly with Fourier-Motzkin + branch and bound Solve exactly with Fourier-Motzkin + branch and bound Omega package from University of Maryland Omega package from University of Maryland

38. CS243 Winter 2006 Stanford University 38 Incremental Analysis Use a series of simple tests to solve simple programs (based on properties of inequalities rather than array access patterns) Use a series of simple tests to solve simple programs (based on properties of inequalities rather than array access patterns) Solve exactly with Fourier-Motzkin + branch and bound Solve exactly with Fourier-Motzkin + branch and bound Memoization Memoization Many identical integer programs solved for each program Many identical integer programs solved for each program Save the results so it need not be recomputed Save the results so it need not be recomputed

39. CS243 Winter 2006 Stanford University 39 State of the Art Multiprocessors need large outer parallel loops Multiprocessors need large outer parallel loops Many inter-procedural optimizations are needed Many inter-procedural optimizations are needed Interprocedural scalar optimizations Interprocedural scalar optimizations Dependence Dependence Privatization Privatization Reduction recognition Reduction recognition Interprocedural array analysis Interprocedural array analysis Array section analysis Array section analysis

40. CS243 Winter 2006 Stanford University 40 Summary DOALL loops DOALL loops Iteration Space Iteration Space Data dependence analysis Data dependence analysis