1. SQL Server 2008 R2 StreamInsight Complex Event Processing Event Stream Processing

2. Who Am I? SQL Server MVP SQL Server Consultant Joint author on Wrox Professional SSIS book www.SQLDTS.com and www.SQLIS.com www.SQLDTS.comwww.SQLIS.com Specialise in Moving Data @allanSQLIS (twitter)

3. Agenda Use Cases & Challenges Formulating Declarative Queries Windows in Time Event Flow Debugging Demos

4. Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. What is CEP?

5. Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries request response Event output stream input stream What is CEP?

6. Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less request response Event output stream input stream What is CEP?

7. Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less Data RateHundreds of events/sec> Tens of thousands of events/sec request response Event output stream input stream What is CEP?

8. Who might need CEP Fraud Detection Real-Time Trade Risking Algorithmic Trading/Betting Meter throughputs – Oil, Gas, Water, Electricity – Use to drive alarms, alerts etc

9. Query Logic Event sources Event targets Devices, Sensors Web servers Event stores & Databases Stock ticker, news feeds Event stores & Databases Pagers & Monitoring devices KPI Dashboards, SharePoint UI Trading stations StreamInsight Application at Runtime.NET C# LINQ StreamInsight Application Development Input Adapters Output Adapters StreamInsight Engine Query Logic StreamInsight Platform

10. Query Expressiveness Selection of events (filter) Calculations on the payload (project) Correlation of streams (join) Stream partitioning (group and apply) Aggregation (sum, count, …) over event windows Ranking over event windows (topK)

11. Query Expressiveness var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 }; var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 }; Projection

12. Query Expressiveness var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 }; var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 }; Projection Filter

13. Query Expressiveness var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w }; var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w }; Projection Filter Correlation (Join)

14. Query Expressiveness var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) }; var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) }; Projection Filter Correlation (Join) Aggregation over windows

15. Projection Filter Correlation (Join) Aggregation over windows Group and Aggregate Query Expressiveness var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) }; var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };

16. Windowing Hopping – Tumbling Snapshot Count

17. Time Windows Hopping Window

18. Time Windows Tumbling Window

19. Snapshot Windows

21. Debugger Has its own debugger You cannot live without it (believe me) Set breakpoints Filter A wonderful tool

22. DEMOS Tour Trace Reader Live Trace File Reader Query Demo (Joining) Debugger Tool (How it will save you)