INTEGRATION DAY 2015 Sam Vanhoutte Azure Event Hubs, Stream Analytics & Power BI

Nice to meet you Sam VANHOUTTE CTO 7 year - BizTalk V-TSP 1st year - Integration MVP be.linkedin.com/in/samvanhoutte/ > 60 Active integration customers International Focus - HQ in BE Focused on integration solutions 2000 Belgium 2004 France 2013 Portugal 60 employees > 50 consultants BizTalk certified e-news + SoMe 2012 & 2013 Partner of the Year Award Finalist Application Integration

The Internet of Things 3 The Internet of Everything M2M communicationSpecial purpose devicesSmart things

The IoT value chain Machine Learning

Demo scenario 6 Event HubsStream AnalyticsPower BI Ingest Speeding tickets Traffic jam detection

Power BI tools 7 Power BI designer (when no office license) PowerQuery >> Get data PowerPivot>> Prepare data PowerView>> Present data

Event Hubs an Azure Service Bus service (collection & ingestion)

Azure Service Bus RelayQueue Topic Notification Hub Event Hub

Event Hub – IOT at Scale Event Sources Cloud Services Storage & Analytics Custom Code & 3 rd Party Services Web/Mobile User Interfaces Integration Services Event Hub - Hyper Scale - - Fully Managed - - Interoperable - - Secure - - Cost Effective -

Introducing Azure Event Hubs Event Producers

Segmentation of the event stream for scale-out – Parallelism for consumers pulling events for processing – Parallelism for producers sub mitting events Default 16, minimum 8, self-service maximum 32 – Azure Support can enable up to 1024 (or more under special conditions) – Maximum 10 Event Hubs per namespace Sender usage of partitions – Direct targeting with partition-id allowing for sender controlled segmentation – Automatic hash-based distribution by PartitionKey or Publisher Identity – Automatic random distribution Partitions

Throughput Unit (TU): Quota and Billing Concept – Write: Lesser of 1MByte/sec or 1000 message operations/sec (incl. management) – Read: 2MByte/sec – Included retention: 84GByte/day (24h at full ingress rate) Retention can be expanded w/ self-service up to 7 days, via Azure support up to 30 days Local-redundant Azure storage pricing for overages applies Number of Partitions ≥ Throughput Units – At most one throughput unit per partition, minimum is one – 10 partitions have 10 TU = 10 MByte/sec throughput ceiling TUs are applied and enforced at the namespace level, i.e. across Event Hubs – Maximum of 20 TUs per account in self-service. – Further w/ commitments through Azure support (blocks of 20 up to 100, blocks of 100) Billing – TUs are billed by the hour (!) they are applied to a namespace (more on pricing later) Throughput Units

Event Hubs an Azure Service Bus service Producers (sending data to Event Hubs)

Publishers 15 Event Producers Very many publishers Short-lived, low throughput: HTTPS Long-lived, high throughput: AMQP Long lived AMQP connections are billable, HTTPS requests are not; AMQP connection allowance included in tier Publish to … PartitionId Direct PartitionKey PartitionKey selecting PartitionId Publisher Policy ( /publishers/ ) overriding PartitionKey

Publishers 16 Event Producers Very many publishers Short-lived, low throughput: HTTPS Long-lived, high throughput: AMQP Long lived AMQP connections are billable, HTTPS requests are not; AMQP connection allowance included in tier Publish to … PartitionId Direct PartitionKey PartitionKey selecting PartitionId Publisher Policy ( /publishers/ ) overriding PartitionKey Demo: Create Event Hubs Send events to Event Hub

Securing publishers 17 ➔ SAS Policies are defined in the portal ➔ Not linked to publishers and limited in numbers ➔ Create Signatures per device ➔ Have a timespan ➔ Are unique to a publisher ➔ Revoking or blacklisting publishers ➔ Manage Rights needed for this

Securing publishers 18 ➔ SAS Policies are defined in the portal ➔ Not linked to publishers and limited in numbers ➔ Create Signatures per device ➔ Have a timespan ➔ Are unique to a publisher ➔ Revoking or blacklisting publishers ➔ Manage Rights needed for this Demo: Generating SAS Signatures Using Signatures by publishers Revoking publishers Reminder ! Sam, increase your instances

Event Hubs an Azure Service Bus service Consumers (reading data to Event Hubs)

Like subscriptions Consumer groups Receivers read from a consumer groups Checkpointing per CG Create receivers per partition Maximum 20 CG Consumer Grp {Default}

Like subscriptions Consumers Partitioned consumer model (not competing consumers!) Each at their own pace ID, Time, [Data]

EventProcessorHost Out of the box Lease management 22 Nuget package Managed cursors by client Uses storage for state and CP IEventProcesso r

EventProcessorHost Out of the box Lease management 23 Nuget package Managed cursors by client Uses storage for state and CP IEventProcesso r Demo: Implementing EventProcessorHost Dashboarding demo

Securing consumers 24 ➔ Using SAS key / values or ACS with SBAZTool ➔ Full SAS support for consumer groups will come

Event Hubs an Azure Service Bus service Pricing & Tips + tricks

Pricing 26 Basic: Up to 100 connections, no extension Standard: 1000 connections incl. Price (US Dollars) Throughput Unit Hour (Basic)0.015TU per hour Throughput Unit Hour (Standard)0.03TU per hour Ingress Events0.028per 1,000,000 events Cost Brokered Connections (1k-100k) connection/hour Cost Brokered Connections (100k-500k) connection/hour Cost Brokered Connections (500k+) connection/hour Storage Overage >TUs*84GB local-redundant Azure storage charge- through

Tips & tricks 27 Multiple hubs per namespace Reuse senders & factories Use consumer groups

Azure Stream Analytics complex event processing. Introduction (set up a basic ASA job)

Scenarios

Real time analytics 30 Millions events / sec Continuous stream of data Correlate Fast time to value High availability SQL Syntax Easy test & debug in portal PowerBI !.999% SQL

End to end Architecture overview 31 Data Source CollectProcessConsumeDeliver Event Inputs -Event Hub -Azure Blob Transform -Temporal joins -Filter -Aggregates -Projections -Windows -Etc. Enrich Correlate Outputs -SQL Azure -Azure Blobs -Event Hub - Table storage - PowerBI Azure Storage Temporal Semantics Guaranteed delivery Guaranteed up time Azure Stream Analytics Reference Data -Azure Blob

End to end Architecture overview 32 Data Source CollectProcessConsumeDeliver Event Inputs -Event Hub -Azure Blob Transform -Temporal joins -Filter -Aggregates -Projections -Windows -Etc. Enrich Correlate Outputs -SQL Azure -Azure Blobs -Event Hub Azure Storage Temporal Semantics Guaranteed delivery Guaranteed up time Azure Stream Analytics Reference Data -Azure Blob Demo: Set up Stream Analytics The first job

Azure Stream Analytics complex event processing. Query Syntax (SQL like a pro)

Functions & supported types Aggregate functions Count, Min, Max, Avg, Sum Scalar functions Cast Date and time: Datename, Datepart, Day, Month, Year, Datediff, Dateadd String: Len, Concat, Charindex, Substring, Patindex Types TypeDescription bigintIntegers in the range -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807). floatFloating point numbers in the range E+308 to -2.23E-308, 0, and 2.23E-308 to 1.79E+308. nvarchar(max)Text values, comprised of Unicode characters. Note: A value other than max is not supported. datetimeDefines a date that is combined with a time of day with fractional seconds that is based on a 24-hour clock and relative to UTC (time zone offset 0).

Tumbling windows Group events, based on timestamping 35 Tumbling window Aggregate per time interval Hopping window Schedule overlapping windows Sliding window Windows constant re-evaluated

End to end Architecture overview 36 Data Source CollectProcessConsumeDeliver Event Inputs -Event Hub -Azure Blob Transform -Temporal joins -Filter -Aggregates -Projections -Windows -Etc. Enrich Correlate Outputs -SQL Azure -Azure Blobs -Event Hub Azure Storage Temporal Semantics Guaranteed delivery Guaranteed up time Azure Stream Analytics Reference Data -Azure Blob Important: For PowerBI you need organizational account

End to end Architecture overview 37 Data Source CollectProcessConsumeDeliver Event Inputs -Event Hub -Azure Blob Transform -Temporal joins -Filter -Aggregates -Projections -Windows -Etc. Enrich Correlate Outputs -SQL Azure -Azure Blobs -Event Hub Azure Storage Temporal Semantics Guaranteed delivery Guaranteed up time Azure Stream Analytics Reference Data -Azure Blob Demo: Traffic management (speed tickets) Traffic management (average speed) Traffic management (suspected cars)

Tips & tricks 38 Use Timestamp by Chose your windowing logic Consumer groups Don’t delete blobs while job runs