1. Cluster Computing with Dryad Mihai Budiu, MSR-SVC LiveLabs, March 2008

2. Goal 2

3. The Dryad Project http://research.microsoft.com/research/sv/dryad Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly European Conference on Computer Systems (EuroSys), Lisbon, Portugal, March 21-23, 2007 3

4. Dryad Design Implementation Policies as Plug-ins Building on Dryad 4

5. Design Space 5 ThroughputLatency Internet Private data center Data- parallel Shared memory

6. Data Partitioning 6 RAM DATA

7. 2-D Piping Unix Pipes: 1-D grep | sed | sort | awk | perl Dryad: 2-D grep 1000 | sed 500 | sort 1000 | awk 500 | perl 50 7

8. Dryad = Execution Layer 8 Job (Application) Dryad Cluster Pipeline Shell Machine

9. Dryad Design Implementation Policies as Plug-ins Building on Dryad 9

10. Virtualized 2-D Pipelines 10

11. Virtualized 2-D Pipelines 11

12. Virtualized 2-D Pipelines 12

13. Virtualized 2-D Pipelines 13

14. Virtualized 2-D Pipelines 14 2D DAG multi-machine virtualized

15. Dryad Job Structure 15 grep sed sort awk perl grep sed sort awk Input files Vertices (processes) Output files Channels Stage grep 1000 | sed 500 | sort 1000 | awk 500 | perl 50

16. Channels 16 X M Items Finite Streams of items distributed filesystem files (persistent) SMB/NTFS files (temporary) TCP pipes (inter-machine) memory FIFOs (intra-machine)

17. Architecture 17 Files, TCP, FIFO, Network job schedule data plane control plane NSPD V VV Job managercluster

18. JM code vertex code Staging 1. Build 2. Send.exe 3. Start JM 5. Generate graph 7. Serialize vertices 8. Monitor Vertex execution 4. Query cluster resources Cluster services 6. Initialize vertices

19. Fault Tolerance

20. Dryad Design Implementation Policies and Resource Management Building on Dryad 20

21. Policy Managers 21 RR XXXX Stage R RR Stage X Job Manager R managerX Manager R-X Manager Connection R-X

22. X[0]X[1]X[3]X[2] Completed vertices Slow vertex Duplicate vertex Duplicate Execution Manager Duplication Policy = f(running times, data volumes)

23. SSSS AAA SS T SSSSSS T # 1# 2# 1# 3 # 2 # 3# 2# 1 static dynamic rack # Aggregation Manager 23

24. Data Distribution (Group By) 24 Dest Source Dest Source Dest Source m n m x n

25. TT [0-?)[?-100) Range-Distribution Manager S DDD SS SSS T static dynamic 25 Hist [0-30),[30-100) [30-100)[0-30) [0-100)

26. Goal: Declarative Programming 26 X T S XX SS TTT X staticdynamic

27. Dryad Design Implementation Policies as Plug-ins Building on Dryad 27

28. Software Stack 28 Windows Server Cluster Services Distributed Filesystem Dryad Distributed Shell PSQL DryadLINQ Perl SQL server C++ Windows Server C++ CIFS/NTFS legacy code sed, awk, grep, etc. SSIS Queries C# Vectors Machine Learning C# Job queueing, monitoring

29. SkyServer Query 18 29 select distinct P.ObjID into results from photoPrimary U, neighbors N, photoPrimary L where U.ObjID = N.ObjID and L.ObjID = N.NeighborObjID and P.ObjID < L.ObjID and abs((U.u-U.g)-(L.u-L.g))<0.05 and abs((U.g-U.r)-(L.g-L.r))<0.05 and abs((U.r-U.i)-(L.r-L.i))<0.05 and abs((U.i-U.z)-(L.i-L.z))<0.05

30. 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 0246810 Number of Computers Speed-up (times) Dryad In-Memory Dryad Two-pass SQLServer 2005 SkyServer Q18 Performance 30

31. DryadLINQ 31 Declarative programming Integration with Visual Studio Integration with.Net Type safety Automatic serialization Job graph optimizations static dynamic Conciseness

32. 32 LINQ Collection collection; bool IsLegal(Key); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

33. Collection collection; bool IsLegal(Key k); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; 33 DryadLINQ = LINQ + Dryad C# collection results C# Vertex code Query plan (Dryad job) Data

34. Sort & Map-Reduce in DryadLINQ 34 S DDD SS Sort Sampl [0-30),[30-100) [30-100)[0-30) [0-100)

35. PLINQ 35 public static IEnumerable DryadSort (IEnumerable source, Func keySelector, IComparer comparer, bool isDescending) { return source.AsParallel().OrderBy(keySelector, comparer); }

36. Machine Learning in DryadLINQ 36 Dryad DryadLINQ Large Vector Machine learning Data analysis

37. Very Large Vector Library PartitionedVector 37 T Scalar TT T

38. Operations on Large Vectors: Map 1 38 U T T U f f f preserves partitioning

39. V Map 2 (Pairwise) 39 T U f V U T f

40. Map 3 (Vector-Scalar) 40 T U f V V U T f

41. Reduce (Fold) 41 UUU U f fff f UUU U

42. Linear Algebra 42 T U V =,, T

43. Linear Regression Data Find S.t. 43

44. Analytic Solution 44 X×X T Y×X T Σ X[0]X[1]X[2]Y[0]Y[1]Y[2] Σ [ ] -1 * A Map Reduce

45. Linear Regression Code Vectors x = input(0), y = input(1); Matrices xx = x.PairwiseOuterProduct(x); OneMatrix xxs = xx.Sum(); Matrices yx = y.PairwiseOuterProduct(x); OneMatrix yxs = yx.Sum(); OneMatrix xxinv = xxs.Map(a => a.Inverse()); OneMatrix A = yxs.Map( xxinv, (a, b) => a.Multiply(b)); 45

46. Expectation Maximization (Gaussians) 46 160 lines 3 iterations shown

47. Conclusions Dryad = distributed execution environment Application-independent (semantics oblivious) Supports rich software ecosystem – Relational algebra – Map-reduce – LINQ – Etc. DryadLINQ = A Dryad provider for LINQ This is only the beginning! 47

48. Backup Slides 48

49. Many similarities Exe + app. model Map+sort+reduce Few policies Program=map+reduce Simple Mature (> 4 years) Widely deployed Hadoop Dryad Map-Reduce Execution layer Job = arbitrary DAG Plug-in policies Program=graph gen. Complex ( features) New (< 2 years) Still growing Internal 49

50. Small Cluster Support 50 Sort Merge Sort Merge Sort Merge Grouping vertices Sort Merge Fast channels

51. SkyServer DB query Took SQL plan Manually coded in Dryad Manually partitioned data u: objid, color n: objid, neighborobjid [partition by objid] select u.color,n.neighborobjid from u join n where u.objid = n.objid (u.color,n.neighborobjid) [re-partition by n.neighborobjid] [order by n.neighborobjid] [distinct] [merge outputs] select u.objid from u join where u.objid =.neighborobjid and |u.color -.color| < d

52. Optimization D M S Y X M S M S M S UN U

53. D M S Y X M S M S M S UN U

54. Query histogram computation Input: log file (n partitions) Extract queries from log partitions Re-partition by hash of query (k buckets) Compute histogram within each bucket

55. Naïve histogram topology Pparse lines D hash distribute S quicksort C count occurrences MSmerge sort

56. Efficient histogram topology Pparse lines D hash distribute S quicksort C count occurrences MSmerge sort M non-deterministic merge Q' is:Each R is: Each MS C M P C S Q' RR k T k n T is: Each MS D C

57. Final histogram refinement 1,800 computers 43,171 vertices 11,072 processes 11.5 minutes