Presentation is loading. Please wait.

Presentation is loading. Please wait.

The DryadLINQ Approach to Distributed Data-Parallel Computing Yuan Yu Microsoft Research Silicon Valley.

Similar presentations


Presentation on theme: "The DryadLINQ Approach to Distributed Data-Parallel Computing Yuan Yu Microsoft Research Silicon Valley."— Presentation transcript:

1 The DryadLINQ Approach to Distributed Data-Parallel Computing Yuan Yu Microsoft Research Silicon Valley

2 Distributed Data-Parallel Computing Dryad talk: the execution layer – How to reliably and efficiently execute distributed data-parallel programs on a compute cluster? This talk: the programming model – How to write distributed data-parallel programs for a compute cluster?

3 The Programming Model Sequential, single machine programming abstraction Same program runs on single-core, multi-core, or cluster Preserve the existing programming environments – Modern programming languages (C# and Java) are very good Expressive language and data model Strong static typing, GC, generics, … – Modern IDEs (Visual Studio and Eclipse) are very good Great debugging and library support Legacy code could be easily reused

4 Dryad and DryadLINQ DryadLINQ provides automatic query plan generation Dryad provides automatic distributed execution

5 Outline Programming model DryadLINQ Applications Discussions and conclusion

6 LINQ Microsoft’s Language INtegrated Query – Available in.NET3.5 and Visual Studio 2008 A set of operators to manipulate datasets in.NET – Support traditional relational operators Select, Join, GroupBy, Aggregate, etc. – Integrated into.NET programming languages Programs can invoke operators Operators can invoke arbitrary.NET functions Data model – Data elements are strongly typed.NET objects – Much more expressive than relational tables For example, nested data structures

7 LINQ Framework PLINQ Local machine.Net program (C#, VB, F#, etc) Execution engines Query Objects LINQ-to-SQL DryadLINQ LINQ-to-Obj LINQ provider interface Scalability Single-core Multi-core Cluster

8 A Simple LINQ Query IEnumerable babies =...; var results = from baby in babies where baby.Name == queryName && baby.State == queryState && baby.Year >= yearStart && baby.Year <= yearEnd orderby baby.Year ascending select baby;

9 A Simple PLINQ Query IEnumerable babies =...; var results = from baby in babies.AsParallel() where baby.Name == queryName && baby.State == queryState && baby.Year >= yearStart && baby.Year <= yearEnd orderby baby.Year ascending select baby;

10 A Simple DryadLINQ Query PartitionedTable babies = PartitionedTable.Get (“BabyInfo.pt”); var results = from baby in babies where baby.Name == queryName && baby.State == queryState && baby.Year >= yearStart && baby.Year <= yearEnd orderby baby.Year ascending select baby;

11 DryadLINQ Data Model Partition Partitioned Table.Net objects Partitioned table exposes metadata information – type, partition, compression scheme, serialization, etc.

12 Demo It is just programming – The same familiar programming languages, development tools, libraries, etc.

13 K-means Execution Graph C0C0 ac P1P1 P2P2 P3P3 cc C1C1 ac cc C2C2 ac cc C3C3

14 K-means in DryadLINQ public static Vector NearestCenter(Vector v, IEnumerable centers) { return centers.Aggregate((r, c) => (r - v).Norm2() < (c - v).Norm2() ? r : c); } public static IQueryable Step(IQueryable vectors, IQueryable centers) { return vectors.GroupBy(v => NearestCenter(v, centers)).Select(group => group.Aggregate((x,y) => x + y) / group.Count()); } var vectors = PartitionedTable.Get ("dfs://vectors.pt"); var centers = vectors.Take(100); for (int i = 0; i < 10; i++) { centers = Step(vectors, centers); } centers.ToPartitionedTable (“dfs://centers.pt”); public class Vector { public double[] entries; [Associative] public static Vector operator +(Vector v1, Vector v2) { … } public static Vector operator -(Vector v1, Vector v2) { … } public double Norm2() { …} }

15 PageRank Execution Graph N01N01 ae E1E1 E2E2 E3E3 cc N02N02 N03N03 D D D N11N11 N12N12 N13N13 ae cc D D D N21N21 N22N22 N23N23 ae cc D D D N31N31 N32N32 N33N33

16 PageRank in DryadLINQ var pages = PartitionedTable.Get (“dfs://pages.pt”); var ranks = pages.Select(page => new Rank(page.name, 1.0)); // repeat the iterative computation several times for (int iter = 0; iter < n; iter++) { ranks = Step(pages, ranks); } ranks.ToPartitionedTable (“dfs://ranks.pt”); public struct Page { public UInt64 name; public Int64 degree; public UInt64[] links; public Page(UInt64 n, Int64 d, UInt64[] l) { name = n; degree = d; links = l; } public Rank[] Disperse(Rank rank) { Rank[] ranks = new Rank[links.Length]; double score = rank.rank / this.degree; for (int i = 0; i < ranks.Length; i++) { ranks[i] = new Rank(this.links[i], score); } return ranks; } public struct Rank { public UInt64 name; public double rank; public Rank(UInt64 n, double r) { name = n; rank = r; } } public static IQueryable Step(IQueryable pages, IQueryable ranks) { // join pages with ranks, and disperse updates var updates = from page in pages join rank in ranks on page.name equals rank.name select page.Disperse(rank); // re-accumulate. return from list in updates from rank in list group rank.rank by rank.name into g select new Rank(g.Key, g.Sum()); }

17 MapReduce in DryadLINQ MapReduce(source, // sequence of Ts mapper, // T -> Ms keySelector, // M -> K reducer) // (K, Ms) -> Rs { var map = source.SelectMany(mapper); var group = map.GroupBy(keySelector); var result = group.SelectMany(reducer); return result; // sequence of Rs } // Ex: Count the frequencies of words in a book MapReduce(book, line => line.Split(' '), w => w.ToLower(), g => g.Count())

18 Dryad DryadLINQ System Architecture DryadLINQ Client machine (11) Distributed query plan.NET program Query Expr Cluster Output Tables Results Input Tables Invoke Query plan Output Table Dryad Execution.Net Objects LINQ query foreach Vertex code

19 DryadLINQ Distributed execution plan generation – Static optimizations: pipelining, eager aggregation, etc. – Dynamic optimizations: data-dependent partitioning, dynamic aggregation, etc. Vertex runtime – Single machine (multi-core) implementation of LINQ – Vertex code that runs on vertices – Data serialization code – Callback code for runtime dynamic optimizations – Automatically distributed to cluster machines

20 A Simple Example Count the frequencies of words in a book var map = book.SelectMany(line => line.Split(' ')); var group = map.GroupBy(w => w.ToLower()); var result = group.Select(g => g.Count());

21 Naïve Execution Plan M D MG G R X M D G R X M D Map Distribute Merge GroupBy Reduce Consumer map reduce MG G R X..... … g.Count() line.Split(' ')

22 Execution Plan Using Partial Aggregation M G1 D M D M D MG G2 C MG G2 C GroupBy InitialReduce Merge GroupBy Combine Merge GroupBy FinalReduce Consumer map aggregation tree reduce IR MG G3 F X MG G3 F X Map Distribute g.Count() g.Sum()

23 Challenge: Program Analysis Support The main sources of difficulty – Complicated data model – User-defined functions all over the places Requires sophisticated static program analysis at byte-code level – Mainly some form of flow anal ysis Possible with modern programming languages and runtimes, such as C#/CLR

24 Inferring Dataset Property Useful for query optimizations Select(x => new { A=x.a+x.b, B=f(x.c) } Hash(y => y.a+y.b) Hash(z => z.A) GroupBy(x => x.A) No data partitioning at GroupBy

25 Inferring Dataset Property Useful for query optimizations Select(x => Foo(x) } Hash(y => y.a+y.b) Hash(x => x.A)? GroupBy(x => x.A) Need IL-level data-flow analysis of Foo

26 Caching Query Results A cluster-wide caching service to support – Reuse of common subqueries – Incremental computations Cache: [Key  Value] – Key is, Value is the result of q(d) – Key requires a pretty good reachability analysis For correctness, Key must include “everything” reachable from q For performance, Key should only contain things q depends on

27 More Static Analysis Purity checking – All the functions called in DryadLINQ queries must be side-effect free – DryadLINQ (and PLINQ) doesn’t enforce it Metadata validity checking – Partitioned table’s metadata contains the record type, partition scheme, serialization functions, … – Need to determine if the metadata is valid – DryadLINQ doesn’t fully enforce it Static enforcement of program properties for security and privacy mechanisms

28 Examples of DryadLINQ Applications Data mining – Analysis of service logs for network security – Analysis of Windows Watson/SQM data – Cluster monitoring and performance analysis Graph analysis – Accelerated Page-Rank computation – Road network shortest-path preprocessing Image processing – Image indexing – Decision tree training – Epitome computation Simulation – light flow simulations for next-generation display research – Monte-Carlo simulations for mobile data eScience – Machine learning platform for health solutions – Astrophysics simulation

29 Decision Tree Training Mihai Budiu, Jamie Shotton et al Machine learning 1M images x 10,000 pixels x 2,000 features x 2 21 tree nodes Complexity >10 20 objects Decision Tree labelimage Learn a decision tree to classify pixels in a large set of images

30 Read, preprocess Redistribute Histogram Regroup histograms on node Compute new tree layer Compute new tree Broadcast new tree Initial empty tree Final tree Image inputs, partitioned Sample Execution plan 30

31 Application Details Workflow = 37 DryadLINQ jobs 12 hours running time on 235 machines More than 100,000 processes More than 100 days of CPU time Recovers from several failures daily 34,000 lines of.NET code

32 Windows SQM Data Analysis Michal Strehovsky, Sivarudrappa Mahesh et al

33 The Language Integration Approach Single unified programming environment – Unified data model and programming language – Direct access to IDE and libraries – Different from SQL, HIVE, Pig Latin Multiple layers of languages and data models Works out very well, but requires good programming language supports – LINQ extensibility: custom operators/providers –.NET reflection, dynamic code generation, …

34 Combining with PLINQ 34 Query DryadLINQ PLINQ subquery The combination of PLINQ and DryadLINQ delivers computation to every core in the cluster

35 Acyclic Dataflow Graph Acyclic dataflow graph provides a very powerful computation model – Easy target for higher-level programming abstractions such as DryadLINQ – Easy expression of many data-parallel optimizations We designed Dryad to be general and flexible – Programmability is less of a concern – Used primarily to support higher-level programming abstractions – No major changes made to Dryad in order to support DryadLINQ

36 Expectation Maximization (Gaussians) 36 Generated by DryadLINQ 3 iterations shown

37 Decoupling of Dryad and DryadLINQ Separation of concerns – Dryad layer concerns scheduling and fault-tolerance – DryadLINQ layer concerns the programming model and the parallelization of programs – Result: powerful and expressive execution engine and programming model Different from the MapReduce/Hadoop approach – A single abstraction for both programming model and execution engine – Result: very simple, but very restricted execution engine and language

38 Image Processing Cosmos DFSSQL Servers Software Stack 38 Windows Server Cluster Services (Azure, HPC, or Cosmos) Azure DFS Dryad DryadLINQ Windows Server CIFS/NTFS Machine Learning Graph Analysis Data Mining Applications …… eScience

39 Conclusion Single unified programming environment – Unified data model and programming language – Direct access to IDE and libraries An open and extensible system – Many LINQ providers out there Existing ones: LINQ-to-XML, LINQ-to-SQL, PLINQ, … Very easy to write one for your app domain – Dryad/DryadLINQ scales out all of them!

40 Availability Freely available for academic use – http://connect.microsoft.com/DryadLINQ – DryadLINQ source, Dryad binaries, documentation, samples, blog, discussion group, etc. Will be available soon for commercial use – Free, but no product support


Download ppt "The DryadLINQ Approach to Distributed Data-Parallel Computing Yuan Yu Microsoft Research Silicon Valley."

Similar presentations


Ads by Google