1. Parallel Prefix

2. The Parallel Prefix Method
This is our first example of a parallel algorithm
Watch closely what is being optimized for
Parallel steps
Beautiful idea with surprising uses
Not sure if the parallel prefix method is used much in the real world
Might maybe be inside MPI scan
Might be used in some SIMD and SIMD like cases
The real key: What is it about the real world that differs from the naïve mental model of parallelism?

3. Students early mental models
Look up or figure out how to do things in parallel
Then we get speedups!
NOT!

4. Parallel Prefix Algorithms
A theoretical (may or may not be practical) secret to turning serial into parallel
Suppose you bump into a parallel algorithm that surprises you “there is no way to parallelize this algorithm” you say
Probably a variation on parallel prefix!

5. Example of a prefix Sum Prefix Input x = (x1, x2, . . ., xn)
Output y = (y1, y2, . . ., yn)
yi = Σj=1:I xj
Example
x = ( 1, 2, 3, 4, 5, 6, 7, 8 )
y = ( 1, 3, 6, 10, 15, 21, 28, 36)
Prefix Functions-- outputs depend upon an initial string

6. What do you think? Can we really parallelize this?
It looks like this sort of code:
y=0;
for i=2:n, y(i)=y(i-1)+x(i); end
The ith iteration of the loop is not at all decoupled from the (i-1)st iteration.
Impossible to parallelize right?

7. A clue?
x = ( 1, 2, 3, 4, 5, 6, 7, 8 )
y = ( 1, 3, 6, 10, 15, 21, 28, 36)
Is there any value in adding, say, ?
Note if we separately have 1+2+3, what can we do?
Suppose we added 1+2, 3+4, etc. pairwise, what could we do?

8. MATLAB command: y=cumsum(x) MATLAB matmul: y=tril(ones(n))*x
Prefix Functions -- outputs depend upon an initial string
Suffix Functions -- outputs depend upon a final string
Other Notations
+\ “plus scan” APL (“A Programming Language” source of the very name “scan”, an array based language that was ahead of its time)
MPI_scan
MATLAB command: y=cumsum(x)
MATLAB matmul: y=tril(ones(n))*x

9. Parallel Prefix Recursive View
Pairwise sums
Recursive prefix
Update “odds”
Any associative operator
1 0 0
1 1 0
1 1 1

10. MATLAB simulation function y=prefix(x) n=length(x); if n==1, y=x; else
w=x(1:2:n)+x(2:2:n); % Pairwise adds
w=prefix(w); % Recur
y(1:2:n)= x(1:2:n)+[0 w(1:end-1) ]; y(2:2:n)=w; % Update Adds
end
What does this reveal? What does this hide?

11. Operation Count Notice # adds = 2n # required = n
Parallelism at the cost of more work!

12. Any Associative Operation works
(a b) c = a (b c) + + + +
Sum (+)
Product (*)
Max
Min
Input: Reals
All (=and)
Any (= or)
Input: Bits (Boolean)
MatMul
Inputs: Matrices

13. Fibonacci via Matrix Multiply Prefix
Fn+1 = Fn + Fn-1
Can compute all Fn by matmul_prefix on
[ , , , , , , , , ]
then select the upper left entry

14. Arithmetic Modulo 2 (binary arithmetic)
0+0= *0=0
0+1= *1=0
1+0= *0=0
1+1= *1=1
Add =
exclusive or
Mult =
and

15. Carry-Look Ahead Addition (Babbage 1800’s)
Example
Carry
First Int
Second Int
Sum
Goal: Add Two n-bit Integers

16. Carry-Look Ahead Addition (Babbage 1800’s)
Goal: Add Two n-bit Integers
Example Notation
Carry c2 c1 c0
First Int a3 a2 a1 a0
Second Int a3 b2 b1 b0
Sum s3 s s1 s0

17. Carry-Look Ahead Addition (Babbage 1800’s)
Goal: Add Two n-bit Integers
Example Notation
Carry c2 c1 c0
First Int a3 a2 a1 a0
Second Int a3 b2 b1 b0
Sum s3 s s1 s0
c-1 = 0
for i = 0 : n-1
si = ai + bi + ci-1
ci = aibi + ci-1(ai + bi)
end
sn = cn-1
(addition mod 2)

18. Carry-Look Ahead Addition (Babbage 1800’s)
Goal: Add Two n-bit Integers
Example Notation
Carry c2 c1 c0
First Int a3 a2 a1 a0
Second Int a3 b2 b1 b0
Sum s3 s s1 s0
c-1 = 0
for i = 0 : n-1
si = ai + bi + ci-1
ci = aibi + ci-1(ai + bi)
end
sn = cn-1
(addition mod 2)
ci ai + bi aibi ci-1 =

19. Carry-Look Ahead Addition (Babbage 1800’s)
Goal: Add Two n-bit Integers
Example Notation
Carry c2 c1 c0
First Int a3 a2 a1 a0
Second Int a3 b2 b1 b0
Sum s3 s s1 s0
c-1 = 0
for i = 0 : n-1
si = ai + bi + ci-1
ci = aibi + ci-1(ai + bi)
end
sn = cn-1
(addition mod 2)
ci ai + bi aibi ci-1 =
Matmul prefix with binary arithmetic is equivalent to carry-look ahead!
Compute ci by prefix, then
si = ai + bi +ci-1 in parallel

20. [ ] Tridiagonal Factor a1 b1 c1 a2 b2 Determinants (D0=1, D1=a1)
[ ]
a1 b1
c1 a2 b2
c2 a3 b3
c3 a4 b4
c4 a5
Tridiagonal Factor
Determinants (D0=1, D1=a1)
(Dk is the det of the kxk upper left):
Dn-1 Dn
T =
Dn = an Dn-1 - bn-1 cn-1 Dn-2
Compute Dn by matmul_prefix
Dn an -bn-1cn Dn-1
Dn Dn-2 =
d1 b1
l d2 b2
l d3
dn = Dn/Dn-1
ln = cn/dn
T =
3 embarassing
Parallels + prefix

21. The “Myth” of log n
The log2 n parallel steps is not the main reason for the usefulness of parallel prefix.
Say n = 1000p (1000 summands per processor)
Time = (2000 adds) + (log2P message passings)
fast & embarassingly parallel
(2000 local adds are serial for each processor of course)

22. Myth of log n Example 80, 000 10, 000 adds + 3 communication hops
total speed is as if there is no communication
Myth of log n Example
40, 000
20, 000
10, 000
log2n = number of steps to add n numbers (NO!!)

23. Any Prefix Operation May Be Segmented!

24. Segmented Operations Inputs = Ordered Pairs (operand, boolean)
e.g. (x, T) or (x, F)
Change of segment indicated by switching T/F
2 (y, T) (y, F)
(x, T) (x y, T) (y, F)
(x, F) (y, T) (xÅy, F)
e. g
T T F F F T F T + +
Result

25. Copy Prefix: x y = x (is associative)+
Segmented
T T F F F T F T

26. High Performance Fortran
SUM_PREFIX ( ARRAY, DIM, MASK, SEG, EXC)
T T T T T
A = M = F F T T T
T F T F F
SUM_PREFIX(A) =
SUM_SUFFIX(A)
SUM_PREFIX(A, DIM = 2) =
SUM_PREFIX(A, MASK = M) =

27. More HPF Segmented T T | F | T T | F F 1 2 3 4 5 A = 6 7 8 9 10
A =
T T F F F
S = F T T F F
T T T T T
Sum_Prefix (A, SEGMENTS = S)
T T | F | T T | F F

28. Example of Exclusive A = 1 2 3 4 5 Sum_Prefix(A) 1 3 6 10 15
Sum_Prefix(A, EXCLUSIVE = TRUE)
(Exclusive: Don’t count myself)

29. Parallel Prefix prefix( [1 2 3 4 5 6 7 8])=[1 3 6 10 15 21 28 36]
Pairwise sums
Recursive prefix
Update “evens”
Any associative operator
AKA: +\ (APL), cumsum(Matlab), MPI_SCAN,
1 0 0
1 1 0
1 1 1

30. Variations on Prefix
exclusive( [ ])=[ ]
1)Pairwise Sums
2)Recursive Prefix
3)Update “odds”

31. Variations on Prefix The Family... 1 2 3 4 5 6 7 8 3 7 11 15 0 3 10 21
exclusive( [ ])=[ ]
1)Pairwise Sums
2)Recursive Prefix
3)Update “odds”
The Family...
Directions
Left
Inclusive
Exc=0
Prefix
Exclusive
Exc=1
Exc Prefix

32. Variations on Prefix The Family... 1 2 3 4 5 6 7 8 3 7 11 15 0 3 10 21
exclusive( [ ])=[ ]
1)Pairwise Sums
2)Recursive Prefix
3)Update “evens”
The Family...
Directions
Left
Right
Inclusive
Exc=0
Prefix
Suffix
Exclusive
Exc=1
Exc Prefix
Exc Suffix

33. Variations on Prefix The Family... 1 2 3 4 5 6 7 8 3 7 11 15
reduce( [ ])=[ ]
1)Pairwise Sums
2)Recursive Reduce
3)Update “odds”
The Family...
Directions
Left
Right
Left/Right
Inclusive
Exc=0
Prefix
Suffix
Reduce
Exclusive
Exc=1
Exc Prefix
Exc Suffix
Exc Reduce

34. Variations on Prefix The Family... 1 2 3 4 5 6 7 8 3 7 11 15 0 3 10 21
exclusive( [ ])=[ ]
1)Pairwise Sums
2)Recursive Prefix
3)Update “evens”
The Family...
Directions
Left
Right
Left/Right
Inclusive
Exc=0
Prefix
Suffix
Reduce
Exclusive
Exc=1
Exc Prefix
Exc Suffix
Exc Reduce
Neighbor Exc
Exc=2
Left Multipole
Right " " "
Multipole

35. Multipole in 2d or 3d etc The Family... Directions Left Right
Notice that left/right generalizes more readily to higher dimensions
Ask yourself what Exc=2 looks like in 3d
The Family...
Directions
Left
Right
Left/Right
Inclusive
Exc=0
Prefix
Suffix
Reduce
Exclusive
Exc=1
Exc Prefix
Exc Suffix
Exc Reduce
Neighbor Exc
Exc=2
Left Multipole
Right " " "
Multipole

36. Not Parallel Prefix but PRAM
Only concerned with minimizing parallel time (not communication)
Arbitrary number of processors
One element per processor

37. Csanky’s (1977) Matrix Inversion
Lemma 1: ( ) in O(log2n) (triangular matrix inv)
Proof Idea: A A
C B -B-1CA B-1 =
Lemma 2: Cayley - Hamilton
p(x) = det (xI - A) = xn + c1xn cn (cn = det A)
0 = p(A) = An + c1An cnI
A-1 = (An-1 + c1An cn-1)(-1/cn)
Powers of A via Parallel Prefix

38. Lemma 3: Leverier’s Lemma
1 c s1
s c s2
s2 s c s3 sk= tr (Ak)
: : : :
sn s1 n cn sn
Csanky 1) Parallel Prefix powers of A
2) sk by directly adding diagonals ) ci from lemas 1 and 3
4) A-1 obtained from lemma 2
= -
Horrible for A=3I and n>50 !!

39. Matrix multiply can be done in log n steps on n3 processors with the pram model
Can be useful to think this way, but must also remember how real machines are built!
Parallel steps are not the whole story
Nobody puts one element per processor