1. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley1 CS 284a Lecture Tuesday, 18 November, 1997

2. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley2 Flags for Mutual Exclusion and Ordering { sthread_flag flag[n + 1]; float sum; sum = 0.0; for (i = 0; i < n; i++) sthread_flag_intialize(&flag[i]); sthread_flag_set(&flag[0]); #pragma multithreadable for (i = 0; i < n; i++) { float subsum = f(i); sthread_flag_check(&flag[i]); sum = sum + subsum; sthread_flag_set(&flag[i + 1]); } for (i = 0; i < n; i++) sthread_flag_finalize(&flag[i]); } One flag for each synchronization operation.

3. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley3 Need Fewer Counters Than Flags { sthread_counter counter; float sum; sum = 0.0; sthread_counter_intialize(&counter); #pragma multithreadable for (i = 0; i < n; i++) { float subsum = f(i); sthread_counter_check(&counter, i); sum = sum + subsum; sthread_counter_increment(&counter, 1); } sthread_counter_finalize(&counter); } One counter for all n synchronization operations.

4. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley4 Synchronization Counter Operations typedef... sthread_counter; int sthread_counter_initialize(sthread_counter *counter); /* Initialize before any other operation on counter. */ /* Counter value is initialized to zero. */ int sthread_counter_finalize(sthread_counter *counter); /* Finalize after all other operations on counter. */ int sthread_counter_increment(sthread_counter *counter, int amount); /* Increment counter by amount. */ int sthread_counter_check(sthread_counter *counter, int value); /* Suspend until counter value reaches at least the given value. */ int sthread_counter_reset(sthread_counter *counter); /* Set counter value to zero. */ Return code is error value ( STHREAD_ERROR_NONE for no error detected).

5. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley5 Example: LU Factorization A LU 1 1 1 1 1 1 1 1 1 3 74 4-26 3728 25 42 6 18 42 24 18 12 36 18 3 13 37 4 26 30 29 37 5 18 45 2 32 31 25 38 37 9 23 52 81 65 11 56 29 86 4 9 23 68 43 18 94 5 41 7 22 50 15 82 81 11 78 50 8 28 71 23 91 70 25 80 94 3 11 37 58 24 62 2 64 4 9 20 54 67 40 18 24 39 -345 637 7-4 35 62-23 7452-322-4 635947834 43 -3142 -257-384 747528 64134 5697 43 64 3

6. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley6 void LU_factorize(double A[N][N]) { int i, j, k; double sum; for (i = 0; i < N; i++) { for (j = 0; j < i; j++) { sum = 0.0; for (k = 0; k < j1; k++) sum = sum + A[i][k]*A[k][j]; A[i][j] = (A[i][j] - sum)/A[j][j]; } for (j = i; j < N; j++) { sum = 0.0; for (k = 0; k < i; k++) sum = sum +A[i][k]*A[k][j]; A[i][j] = A[i][j] - sum; } Sequential Program

7. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley7 Data Dependencies LU 0 N-1 0

8. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley8 Sequential Execution Order 0N-1 0 LU

9. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley9 Goal: Multithreaded Execution Subject to Data Dependencies 0N-1 0 LU

10. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley10 Fine-Grained Implementation Alternatives N 2 flags - one for every cell. 2N counters - one for every row and every column.

11. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley11 void LU_factorize(double A[N][N]) { counter row_count[N], column_count[N]; int i; for (i = 0; i < n; i++) { sthread_counter_initialize(&row_count[i]); sthread_counter_initialize(&column_count[i]); } #pragma multithreadable for (i = 0; i < N; i++) { int j1, j2; #pragma multithreadable { #pragma multithreadable for (j1 = 0; j1 < i; j1++) { int k; double sum = 0.0; check(row_count[i], j1); check(column_count[j1], j1 + 1); for (k = 0; k < j1; k++) sum = sum + A[i][k]*A[k][j1]; A[i][j1] = (A[i][j1] - sum)/A[j1][j1]; increment(row_count[i]); } #pragma multithreadable for (j2 = i; j2 < N; j2++) { int k; double sum = 0.0; check(row_count[i], i); check(column_count[j2], i); for (k = 0; k < i; k++) sum = sum +A[i][k]*A[k][j2]; A[i][j2] = A[i][j2] - sum; increment(column_count[j2]); } for (i = 0; i < n; i++) { sthread_counter_finalize(&row_count[i]); sthread_counter_finalize(&column_count[i]); }

12. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley12 Efficient Solution Synchronization on every cell is too fine-grained. Need to group cells into blocks. Need counter per row and column of blocks. Still too many threads? Use fewer threads than blocks. Which order of block execution? Which mapping of blocks onto threads?

13. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley13 Rules for Using Flags: Did John Lie to Us? Want multithreaded  sequential. Multithreaded block {... S;... T;... } or multithreaded for loop with iterations... S;... T;... For each variable x such that either: –S writes x and T reads x, –S reads x and T writes x, or –S writes x and T writes x. Require flag x_ready such that: –x_ready is set after operation on x in S. –x_ready is checked before operation on x in T. Flag x_ready maintains order of operations on x.

14. CS 284a, 18 November 1997 Copyright (c) 1997-98, John Thornley14 Question What are the rules for using counters to “guard” shared variables?