1. Data Dependences CS 524 – High-Performance Computing

2. CS 524 (Wi 2003/04)- Asim Karim @ LUMS2 Data Dependences Fundamental execution ordering constraints: S1:a = b + c S2:d = a * 2 S3:a = c + 2 S4:e = d + c + 2  S1 must execute before S2 (flow-dependence)  S2 must execute before S3 (anti-dependence)  S1 must execute before S3 (output-dependence)  But, S3 and S4 can execute concurrently S1 S2 S3S4

3. CS 524 (Wi 2003/04)- Asim Karim @ LUMS3 Types of Dependences (1) Three types are usually defined:  Flow-dependence occurs when a variable which is assigned a value in one statement say S 1 is used in another statement, say S 2 later. Written as S 1 δ f S 2  Anti-dependence occurs when a variable which is used in one statement say S 1 is assigned a value in another statement, say S 2, later. Written as S 1 δ a S 2  Output dependence occurs when a variable which is assigned a value in one statement say S 1 is later reassigned in another statement say S 2. Written as S 1 δ o S 2

4. CS 524 (Wi 2003/04)- Asim Karim @ LUMS4 Types of Dependences (2) Type of dependence found by using IN and OUT sets for each statement  IN(S): the set of memory locations read by statement S  OUT(S): the set of memory locations written by statement S  A memory location may be included in both IN(S) and OUT(S) If S 1 is executed before S 2 in sequential execution, then  OUT(S 1 ) ∩ IN(S 2 ) ≠ {} ==> S 1 δ f S 2  IN(S 1 ) ∩ OUT(S 2 ) ≠ {} ==> S 1 δ a S 2  OUT(S 1 ) ∩ OUT(S 2 ) ≠ {} ==> S 1 δ o S 2

5. CS 524 (Wi 2003/04)- Asim Karim @ LUMS5 Data Dependence in Loops (1) Associate an instance to each statement and determine dependences between the instances  For example, we say S1(10) to mean the instance of S1 when i = 10 do i = 1, 50 S1: A(i) = B(i-1) + C(i) S2: B(i) = A(i+2) + C(i) end do

6. CS 524 (Wi 2003/04)- Asim Karim @ LUMS6 Data Dependence in Loops (2) do i = 1, 50 S1: A(i) = B(i-1) + C(i) S2: B(i) = A(i+2) + C(i) end do S1(1): A(1) = B(0) + C(1) S2(1): B(1) = A(3) + C(1) S1(2): A(2) = B(1) + C(2) S2(2): B(2) = A(4) + C(2) S1(3): A(3) = B(2) + C(3) S2(3): B(3) = A(5) + C(3)... S1(50): A(50) = B(49) + C(50) S2(50): B(50) = A(52) + C(50)

7. CS 524 (Wi 2003/04)- Asim Karim @ LUMS7 Data Flow Dependence Data written by some statement instance is later read by some statement instance, in the serial execution of the program. This is written as S 1 δ f S 2. do i = 3, 50 S1: A(i+1) =... S2:... = A(i-2)... end do S1 S2

8. CS 524 (Wi 2003/04)- Asim Karim @ LUMS8 Data Anti-Dependence Data read by some statement instance is later written by some statement instance, in the serial execution of the program. This is written as S 1 δ a S 2. do i = 1, 50 S1: A(i-1) =... S2:... = A(i+1)... end do S1 S2

9. CS 524 (Wi 2003/04)- Asim Karim @ LUMS9 Iteration Space Graph (1) Nested loops define an iteration space: do i = 1, 4 do j = 1, 4 A(i,j) = B(i,j) + C(j) end do Sequential execution (traversal order): j i

10. CS 524 (Wi 2003/04)- Asim Karim @ LUMS10 Iteration Space Graph (2) Dimensionality of iteration space = loop nest level  Not restricted to be rectangular  Triangular iteration spaces are common in scientific codes do i = 1, 5 do j = i, 5 A(i,j) = B(i,j) + C(j) end do Sequential execution (traversal order): j i

11. CS 524 (Wi 2003/04)- Asim Karim @ LUMS11 Sequential Execution Ordering of execution  Given two iterations (i 1, j 1 ) and (i 2, j 2 ) (with positive loop steps): we say (i 1, j 1 ) PCD (i 2, j 2 ) if and only if either (i 1 < i 2 ) or [(i 1 = i 2 ) AND (j i < j 2 )].  This rule can be extended to multi-dimensional iteration spaces. A vector (d 1, d 2 ) is positive, if (0, 0) PCD (d 1, d 2 ) i.e., its first (leading) non-zero component is positive. PCD = precedes = ordering symbol for vectors

12. CS 524 (Wi 2003/04)- Asim Karim @ LUMS12 Dependences in Loop Nests do i 1 = L 1, U 1 do i 2 = L 2, U 2... do i n = L n, U n BODY(i 1, i 2,..., i n ) end do end do... end do There is a dependence in a loop nest if there are iterations I = (i 1, i 2,...,i n ) and J = (j 1, j 2,...,j n ) and some memory location M such that 1. I PCD J 2. BODY(I) and BODY(J) reference M 3. There is no intervening iteration K that accesses M, I PCD K PCD J

13. CS 524 (Wi 2003/04)- Asim Karim @ LUMS13 Distance and Direction Vectors Assume a dependence from BODY(I = (i 1, i 2,...,i n )) and BODY(J = (j 1, j 2,...,j n )). The distance vector d = (j 1 – i 1, j 2 – i 2,…, j n – i n ) Define the sign function sgn(x 1 ) of scalar x 1 : -if x 1 < 0 sgn(x 1 ) = 0if x 1 = 0 +if x 1 > 0 The direction vector = (sgn(d 1 ), sgn(d 2 ),…,sgn(d n ) where d k = j k - i k for k = 1,…,n.

14. CS 524 (Wi 2003/04)- Asim Karim @ LUMS14 Example of Dependence Vectors do i = 1, N do j = 1, N A(i, j) = A(i, j-3) + A(i-2, j) + A(i-1, j+2) + A(i+1, j-1) end do RHS reference TypeDistance vector Direction vector A(i, j-3)Flow(0, 3)(0, +) A(i-2, j)Flow(2, 0)(+, 0) A(i-1, j+2)Flow(1, -2)(+, -) A(i+1, j-1)Anti(1, -1)(+, -)

15. CS 524 (Wi 2003/04)- Asim Karim @ LUMS15 Validity of Loop Permutation (1) Before interchange do i = 1, N do j = 1, N... end do After interchange do j = 1, N do i = 1, N... end do j i j i (+, -) prevents interchange (-+)

16. CS 524 (Wi 2003/04)- Asim Karim @ LUMS16 Validity of Loop Permutation (2) Loop permutation is valid if  all dependences are satisfied after interchange  Geometric view: source (of depended statements) still executed before sink  Algebraic view: permuted dependence direction vector is lexicographically non-negative

17. CS 524 (Wi 2003/04)- Asim Karim @ LUMS17 Iteration Space Blocking (Tiling) A tile in an n-dimensional iteration space is an n- dimensional subset of the iteration space  A tile is defined by a set of boundaries regularly spaced apart  Each tile boundary is an (n – 1)-dimensional plane j i

18. CS 524 (Wi 2003/04)- Asim Karim @ LUMS18 Validity of Loop Blocking Loop blocking is valid if  All dependences still satisfied in tiled execution order of iteration space graph (i.e. source before sink)  Geometric view: No two dependences cross any tile boundary in opposite directions  Algebraic view: Dot product of all dependence distance vectors with normal to tile boundary has same sign

19. CS 524 (Wi 2003/04)- Asim Karim @ LUMS19 Summary Data dependency analysis is essential to avoid changing the meaning of the program when performance optimization transformations are done Data dependency analysis is essential in design of parallel algorithms from sequential code Data dependences must be maintained in all loop transformations; otherwise the transformation is illegal Data dependency exist in a loop nest when dependence vector is lexicographically non-negative