1. 12/20091 Functional Collection-Oriented Programming Guy Blelloch Carnegie Mellon University

2. Collection-oriented programming Programmer emphasis is on operations over collections of values. (Data Driven) Array based: APL, Nial, FP, Matlab Database: SQL, Linq Scripting: SETL, Python Data parallel: *Lisp, HPF, Nesl, Id, ZPL Map-reduce All of these support some form of Map and some form of reduce. 12/20092

3. Collection-oriented programming Concise code Promotes a functional style of programming Has become popular even without parallelism (matlab, python, sql, …) Parallelism Map is naturally parallel Many collection operations are parallel: reduce, scan, collect, flatten, transpose, … Most often DETERMINISTIC 12/20093

4. Collection-oriented programming “Concurrency” (Non-deterministic environment) On its own not really useful for “concurrent” applications (e.g. operating systems, or front- end of a web server). 12/20094 The environment sequentialconcurrent Parallelism serial Traditional programming Traditional OS parallel Deterministic parallelism General parallelism

5. Flat vs. Nested Can collections contain collections? Can arbitrary functions be mapped? Flat languages APL, SQL, Map-reduce, HPF, Matlab Nested Languages SETL, Python, Nesl, Id I conjecture that flat CO languages will never be general purpose—not good for trees, divide-and-conquer, … 12/20095

6. 6 Quicksort in NESL function quicksort(S) = if (#S <= 1) then S else let a = S[rand(#S)]; S1 = {e in S | e < a}; S2 = {e in S | e = a}; S3 = {e in S | e > a}; R = {quicksort(v) : v in [S1, S3]}; in R[0] ++ S2 ++ R[1];

7. 12/20097 Quicksort in X10 double[] quicksort(double[] S) { if (S.length < 2) return S; double a = S[rand(S.length)]; double[] S1,S2,S3; finish { async { S1 = quicksort(lessThan(S,a));} async { S2 = eqTo(S,a);} S3 = quicksort(grThan(S,a)); } append(S1,append(S2,S3)); }

8. Matrix Multiplication Fun A*B { if #A < k then baseCase.. A 11,A 12,A 21,A 22 = QuadSplit(A) B 11,B 12,B 21,B 22 = QuadSplit(B) Parallel { C 11 = A 11 *B 11 + A 12 *B 21 C 12 = A 11 *B 12 + A 12 *B 22 C 21 = A 21 *B 11 + A 22 *B 21 C 22 = A 21 *B 12 + A 22 *B 22 } return QuadJoin(C 11,C 12,C 21,C 22 ) } 12/20098 Need to be able to program for locality.

9. Question: How general is functional CO programming? Advantages High-level/concise Natural/Intuitive Deterministic Parallelism (for all partial results) No need for annotations, commutativity, regions No speculation Simple cost model (even including locality) Potential Disadvantages Performance Major rewriting of code Does not support “concurrency” on its own 12/20099

10. Barnes Hut function bTree(Pts,box as (x0,y0,s)) = if #pts = 0 then EMPTY else if #pts = 1 then LEAF(p[0]) else let xm = x0 + s/2; ym = y0 + s/2; parallelLet T1 = bTree({(x,y,d) in pts | x<xm & y<ym}, (x0,y0,s/2)); T2 = bTree({(x,y,d) in pts | x =ym}, (x0,y0+s/2,s/2));.. in NODE(cmass(T1,T2,T3,T4),box,T1,T2,T3,T4) 12/200910

11. Barnes Hut function force(p,LEAF(p’)) = force(p,p’) | force(p,EMPTY) = 0 | force(p,(c,box,T1,T2,T3,T4) if far(p,box) then forceC(p,c) else force(p,T1)+force(p,T2)+force(p,T3) +force(p,T4) function forces(Points,T) = {move(p,force(p,T)) : p in Points}; 12/200911

12. “Algorithms in the Real World” Compression: JPEG 12/200912 TopicAlgorithmCOParallelFunctional* CompressionJPEG445 BW445 Error CorrectingReed Solomon444 CodesParity Check555^ CryptographyReijdael455 Bignum445 Comp BiologyBlast444 Clustall335 N-body CodesBarnes Hut555 Callahan Kosaraju555 *Easily expressed with no shared writeable state ^Depends on algorithm

13. Compression: JPEG 12/200913 TopicAlgorithmCOParallelFunctional* Linear/IntegerInterior Point444^ ProgrammingBranch and Bound333 Web IndexingIndex Building555 Page Rank555 GeometryDelaunay454^ Nearest Neighbors445 DimensionalitySVD444 ReductionJohnson Lindenstraus555 String SearchingSuffix Trees444^ Graph SeparatorsContraction444 ^Depends on algorithm

14. Barnes Hut function bTree(Pts,box as (x0,y0,s)) = if #pts = 0 then EMPTY else if #pts = 1 then LEAF(p[0]) else let xm = x0 + s/2; ym = y0 + s/2; parallelLet T1 = bTree({(x,y,d) in pts | x<xm & y<ym}, (x0,y0,s/2)); T2 = bTree({(x,y,d) in pts | x =ym}, (x0,y0+s/2,s/2));.. in NODE(cmass(T1,T2,T3,T4),box,T1,T2,T3,T4) 12/200914

15. Barnes Hut function force(p,LEAF(p’)) = force(p,p’) | force(p,EMPTY) = 0 | force(p,(c,box,T1,T2,T3,T4) if far(p,box) then forceC(p,c) else force(p,T1)+force(p,T2)+force(p,T3) +force(p,T4) function forces(Points,T) = {force(p,T) : p in Points}; 12/200915

16. Graph Connectivity/ Spanning Tree 12/200916

17. 12/200917 Graph Connectivity Edge List Representation: Edges = [(0,1), (0,2), (2,3), (3,4), (3,5), (3,6), (1,3), (1,5), (5,6), (4,6)] 0 1 3 2 4 56

18. 12/200918 Graph Contraction 0 1 3 2 4 56 0 1 3 2 4 56 1 1 1 2 6 16 1 2 61 2 6 1 1 1 1 Form stars relabel contract

19. 12/200919 Graph Connectivity Edge List Representation: Edges = [(0,1), (0,2), (2,3), (3,4), (3,5), (3,6), (1,3), (1,5), (5,6), (4,6)] 0 1 3 2 4 56 Hooks = [(0,1), (1,3), (1,5), (3,6), (4,6)]

20. 12/200920 Graph Connectivity L = Vertex Labels, E = Edge List function connectivity(L, E) = if #E = 0 then L else let FL = {coinToss(.5) : x in [0:#L]}; H = {(u,v) in E | Fl[u] and not(Fl[v])}; L = L <- H; E = {(L[u],L[v]): (u,v) in E | L[u]\=L[v]}; in connectivity(L,E);

21. 12/200921 Conclusions/Questions Perhaps Functional Programming is adequate for most/all parallel applications. Collections seems to encourage a functional style even in non functional languages Give fully deterministic results/and partial results

22. 12/200922 Quicksort in Multilisp (defun quicksort (L) (qs L nil)) (defun qs (L rest) (if (null L) rest (let ((a (car L)) (L1 (filter (lambda (b) (< b a)) (cdr L))) (L3 (filter (lambda (b) (>= b a)) (cdr L)))) (qs L1 (future (cons a (qs L3 rest))))))) (defun filter (f L) (if (null L) nil (if (f (car L)) (future (cons (car L) (filter f (cdr L)) (filter f (cdr L))))

23. 12/200923 Quicksort in Multilisp (futures) Span = O(n) Work = O(n log n) Not a very good parallel algorithm

24. 12/200924 Scan code function addscan(A) = if (#A <= 1) then [0] else let sums = {A[2*i] + A[2*i+1] : i in [0:#a/2]}; evens = addscan(sums); odds = {evens[i] + A[2*i] : i in [0:#a/2]}; in interleave(evens,odds);,

25. Fourier Transform function fft(a,w) = if #a == 1 then a else let r = {fft(b, w[0:#w:2]): b in [a[0:#a:2],a[1:#a:2]} in {a + b * w : a in r[0] ++ r[0]; b in r[1] ++ r[1]; w in w}; 12/200925

26. Sparse Vector Matrix Multiply function sparseMxV(M,v) = {sum({v[i]*w : i,w in row}) : row in M}; 12/200926

27. MapReduce function mapReduce(MAP,REDUCE,documents) = let temp = flatten({MAP(d) : d in documents}); in flatten({REDUCE(k,vs) : (k,vs) in collect(temp)}); 12/200927 function wordcount(docs) = mapReduce(d => {(w,1) : w in wordify(d)}, (w,c) => [(w,sum(c))], documents); wordcount(["this is is document 1”, "this is document 2"]);  [(“1”,1),(“this”,2),(“is”,3),(“document”,2),(“2”,1)]