1. Large-scale Machine Learning using DryadLINQ Mihai Budiu Microsoft Research, Silicon Valley Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop Stanford, June 11, 2019

2. Goal of DryadLINQ 2

3. Software Stack 3 Windows Server Cluster services Cluster storage Dryad DryadLINQ Windows Server Applications.Net + LINQ

4. Dryad = Execution Layer 4 Job (application) Dryad Cluster Pipeline Unix Shell Machine ≈

5. Collection.NET objects of type T LINQ Data Model

6. LINQ Language Summary 6 Where (filter) Select (map) GroupBy OrderBy (sort) Aggregate (fold) Join Input

7. LINQ 7 Dryad => DryadLINQ

8. DryadLINQ Data Model 8 Partition Collection.Net objects

9. Collection collection; static bool IsLegal(Key c); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; 9 DryadLINQ = LINQ + Dryad C# collection results C# Code Dryad job Data

10. Example: Natal Training 10

11. Natal Problem 11 Recognize players from depth map At frame rate Low resource usage

12. Learn from Data 12 Motion Capture (ground truth) Classifier Training examples Machine learning Rasterize

13. Running on Xbox 13

14. Cluster-based training 14 Classifier Training examples Dryad DryadLINQ Machine learning

16. You can have it! Dryad+DryadLINQ available for download – Academic license – Commercial evaluation license Runs on Windows HPC platform Dryad is in binary form, DryadLINQ in source Requires signing a 3-page licensing agreement http://connect.microsoft.com/site/sitehome.aspx?SiteID=891

17. Conclusions 17 =