1. LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation

2. Censored 7 years ago Little recent innovation Where’s the cloud?

3. Why? What? How? Monads

4. IEnumerable IQueryable IEnumerable IQueryable new[] { 42 } SelectMany IEnumerable SelectMany ( this IEnumerable source, Func > selector) Could there be more? Also see www.codeplex.com/LINQSQO for “Project MinLINQ”

5. from _ in name from s in _.ToUpper() select s from _ in name from s in _.ToUpper() select s name.SelectMany( _ => _.ToUpper(), s => s) name.SelectMany( _ => _.ToUpper(), s => s) Compiler Null-propagating dot string s = name?.ToUpper(); Null-propagating dot string s = name?.ToUpper(); Syntactic sugar One single library function suffices

6. demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Essential LINQ

7. var src = new Source (); var res = from p in src where p.Price > 100m group p by p.Category; foreach (var p in res) Console.WriteLine(p); Implements IQueryable Implements IQueryable Compiles fine Does it really have to be a runtime check?

8. var res = from p in src where p.Price > 100m group p by p.Category; var res = src.Where(p => p.Price > 100m).GroupBy(p => p.Category); Syntactic sugar Can be instance methods Source Filtered Projected WhereSelect No GroupBy “edge”

9. var res = from tweet in twitter where tweet. About From Location == “Bart” From From About Location == “LINQ” About where tweet. About Query “learns” class Twitter { public TwitterByFrom Where(Func filter); // Other filter methods } Recipe Method overloads Type state machine Operator overloads Recipe Method overloads Type state machine Operator overloads “Has a” type From operator overloading class TwitterByFrom { public TwitterByAboutFrom Where(Func filter); // Other filter methods // Fields with current filters } select tweet; Custom syntax trees class TweetAboutFromLoc { public FromString From; public AboutString About; public LocString Location; } class FromString { FilterFrom operator ==( FromString f, string s) } class TweetAboutFromLoc { public FromString From; public AboutString About; public LocString Location; } class FromString { FilterFrom operator ==( FromString f, string s) } class TweetAboutLoc { public AboutString About; public LocString Location; } class AboutString { FilterAbout operator ==( AboutString f, string s) } class TweetAboutLoc { public AboutString About; public LocString Location; } class AboutString { FilterAbout operator ==( AboutString f, string s) }

10. demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Query Providers Revisited

11. + + 1 1 * * 2 2 3 3 Source domain Rule Set Patterns with rewrite actions Rule Set Patterns with rewrite actions Target domain BURS Tree Rewriting Engine E.g. LINQ Expression TreesE.g. T-SQL Expression Trees Const(2) Const(3) Mul Inc Currently built at runtime, e.g. once per LINQ provider that gets loaded

12. x 1 $ Const(x) Leaf node ? : int1  Final states 1  1 $ Any source leaf node with an int constant value can be reduced using rule 1. Final states have a cost (incremental, see further).

13. x + ab 1 $ Const(x) 2 $ Add(a,b) Leaf node ? : int1  Wildcard a : int2 * + 3  Final states 1  1 $ 3  2 $ Wildcard b doesn’t get its own entry, since it has the same type as wildcard a. This reduces space in the BURS tables. Wildcard b doesn’t get its own entry, since it has the same type as wildcard a. This reduces space in the BURS tables. 2D table for state transition of the + operator. Wildcard matches any subtree that’s assignable to int.

14. x + ab * ab 1 $ Const(x) 2 $ Add(a,b) 3 $ Mul(a,b) Leaf node ? : int1  Wildcard a : int2 * + 3  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ What’s the meaning of costs? The cost of 3$ is incremental and has to be added to the cost of all the wildcard matches. In this case, a match of the Mul -rule (4  ) costs 3$ + cost a + cost b. What’s the meaning of costs? The cost of 3$ is incremental and has to be added to the cost of all the wildcard matches. In this case, a match of the Mul -rule (4  ) costs 3$ + cost a + cost b.

15. x + ab * ab + c* ab 1 $ Const(x) 2 $ Add(a,b) 3 $ Mul(a,b) 4 $ AddMul(a,b,c) Leaf node ? : int1  Wildcard a : int2 * + 3  4  5  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ 5  4 $ A pattern forest is kept to map subtrees on already existent states. Here (a * b) matches state 4.

16. x + ab * ab + c* ab + a1 1 $ Const(x) 2 $ Add(a,b) 3 $ Mul(a,b) 4 $ AddMul(a,b,c) 1 $ Inc(a) Leaf node ? : int1  16 Wildcard a : int2 * + 6 3  7  4  5  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ 5  4 $ 7  1 $

17. 23 * 1 + Leaf node ? : int1  16 Wildcard a : int2 * + 6 3  7  4  5  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ 5  4 $ 7  1 $ 5 $ 6 $ Constant leaf node 1 doesn’t contribute cost (part of pattern).

18. demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Bottom Up Rewriting

19. Environment MoveNext Got next? Application OnNext Have next! IEnumerable IEnumerator IEnumerable IEnumerator IObservable IObserver IObservable IObserver Interactive Reactive

20. Observable Subscribe Observer

21. Event Stream Processing Interfaces namespace System { public interface IObservable { IDisposable Subscribe(IObserver observer); } public interface IObserver { void OnNext(T value); void OnError(Exception error); void OnCompleted(); }

22. Fixed (MSIL) Translatable (Expression trees) ToQueryable ToObservable ToEnumerable AsQueryable AsEnumerable AsQbservable AsObservable Pull (interactive) Push (reactive) Concurrency (IScheduler) LINQ to *.* What? Where? How? Worker pools Message loops Threads Distributed Duality Homo-iconic IQbservable LINQ to WMI Events ToQbservable IQueryable LINQ to SQL IEnumerable LINQ to Objects IObservable LINQ to Events

23. IQbservable - The Dual of IQueryable public interface IQbservable : IObservable { Expression Expression { get; } Type ElementType { get; } IQbservableProvider Provider { get; } } public interface IQbservableProvider { IQbservable CreateQuery (Expression expression); } Extended role for some operators No Execute method Homoiconic naming IQbservable IObservable Homoiconic naming IQbservable IObservable

24. demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team LINQ to Twitter

25. Baked in notion of “where”? foreach (var p in res.RemoteOn(new SqlScheduler(“server”))) // Process product Decoupled “what” from “where” Entry-point for the schema Custom schedulers could be very rich (e.g. server farm) var ctx = new NorthwindDataContext(); var res = from product in ctx.Products where product.Price > 100m select product.Name;

26. Rx.NET RxJS stocks.Where(function (t) { return t.Symbol == “MSFT”; }).Select(function (t) { return t.Quote; }) JSON serializer Observable data source Retargeting to AJAX from ticker in stocks where ticker.Symbol == “MSFT” select ticker.Quote

27. Query Server Alice F9 F10 chat talk var chat = svc.GetStream (“chat”); var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”); talk.Subscribe(s => ShowMessage(s)); F5 chat Where Select = = msg “Bob” From msg Text Translate chat Where Select = = msg “Bob” From msg Text

28. Query Server Alice chat talk var chat = svc.GetStream (“chat”); var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”); talk.Subscribe(s => ShowMessage(s)); Bob F9 chat F10 var chat = svc.GetStream(“chat”); Rx.Observable.FromDOMEvent(txt, “Changed”).Select(function (e) { return txt.Text; }).Throttle(.5 /* 500 ms */).Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); }); F5 chat Where Select = = msg “Bob” From msg Text Translate chat

29. Query Server Alice chat talk Bob chat Hello Alice! F5 var chat = svc.GetStream(“chat”); Rx.Observable.FromDOMEvent(txt, “Changed”).Select(function (e) { return txt.Text; }).Throttle(.5 /* 500 ms */).Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); }); Bob says: Hello Alice! Bonjour Alice! var chat = svc.GetStream (“chat”); var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”); talk.Subscribe(s => ShowMessage(s)); F5 Bob says: Bonjour Alice! Where Select = = msg “Bob” From msg Text Translate “Bonjour Alice!” From: “Bob” Text: “Hello Alice!” From: “Bob” Text: “Hello Alice!” From: “Bob” Text: “Hello Alice!” From: “Bob” Text: “Hello Alice!” Translate “Bonjour Alice!” “Hello Alice!”

30. demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Expression Tree Serialization

31. announcing Now with support for hosting expression tree based queries (enumerable and observable) StreamInsight v2.1

32. Model[ Decisions[Reals, SA, VZ], Goals[ Minimize[20 * SA + 15 * VZ] ], Constraints[ C1 -> 0.3 * SA + 0.4 * VZ >= 2000, C2 -> 0.4 * SA + 0.2 * VZ >= 1500, C3 -> 0.2 * SA + 0.3 * VZ >= 500, C4 -> SA <= 9000, C5 -> VZ <= 6000, C6 -> SA >= 0, C7 -> VZ >= 0 ] from m in ctx.CreateModel(new { vz = default(double), sa = default(double) }) where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500 where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000 orderby 20 * m.sa + 15 * m.vz select m To compute call Solve Symbolic, non-persisted Cost / barrel Max # barrels Max # barrels Min # barrels Min # barrels Refinement %

33. from costSA in GetPriceMonitor("SA") from costVZ in GetPriceMonitor("VZ") from m in ctx.CreateModel(new { vz = default(double), sa = default(double) }) where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500 where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000 orderby costSA * m.sa + costVZ * m.vz select m Observable data sources SelectMany Parameters to decision Subscribe here!

34. demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team LINQ to Constraints

35. DEV413 – Curing your Event Processing Blues using Reactive Extensions (Rx) Find Me Later The Week At The Ask The Experts

36. Connect. Share. Discuss. http://europe.msteched.com Learning Microsoft Certification & Training Resources www.microsoft.com/learning TechNet Resources for IT Professionals http://microsoft.com/technet Resources for Developers http://microsoft.com/msdn

37. Evaluations http://europe.msteched.com/sessions Submit your evals online