Shubha Vijayasarathy Program Manager, Azure Event Hubs - Microsoft Azure Event Hubs: the world’s most widely used telemetry service

A Big Data Scenario Websites Devices Map reduce system Real time monitoring Data warehouse Devices Pipe to other system

A Big Data Scenario Azure Event Hubs Websites Devices Map reduce system Websites Real time monitoring Data warehouse Devices Pipe to other system

Big Data Design Elements Design considerations, “How do we…” Azure Event Hubs handle variety? handle volume? handle variety? Heterogeneous inputs - devices, web pages, VMs, Docker containers, multi-language clients handle volume? Huge amount of data needs to be ingested handle high velocity? Data delivered at different speeds scale out/in on demand? Workloads increase during a busy week Event Hubs Auto-Inflate handle high velocity? scale out/in on demand? Auto-Inflate

Big Data Design Elements Design considerations, “Can we…” Azure Event Hubs …regional disaster? GeoDR do near real-time stream and batch? Capture …regional disaster? Event Hubs GeoDR do near real-time stream and batch Capture for batch estimate costs accurately? Simple billing model estimate costs accurately? Simple billing model rapidly move from proto  prod?

Big Data Design Elements Design considerations, “How do we…” Azure Event Hubs get support? use the best solutions? Event Hubs - world class team behind you! get world class support Fully managed service with SLAs. Huge community support too. use best solutions You don’t have to be big data experts Event Hubs - world class team behind you who are working on bleeding edge of big-data + streaming technology

Big Data Design Elements Design considerations, “Integrate story…” Azure Event Hubs Integrate? Simplified integration with Azure Event Grid Azure Stream Analytics Logic Apps Event Hubs connector Azure Functions Apache Spark Apache Storm Apache Nifi … easily integrate with other systems Simplified integration with Azure Event Grid Azure Stream Analytics Logic Apps Event Hubs connector Azure Functions Apache Spark Apache Storm Apache Nifi … Many more…

Big Data Design Elements Design considerations, “Do we…” Azure Event Hubs manage the service? diagnose and monitor? manage the service? Complete PaaS solution, we manage the OS upgrades, patching, you just use the service Integrated with Azure Monitor for alerting and monitoring Azure Monitor for Event Hubs metrics Data governance Compliant services - https://azure.microsoft.com/overview/trusted-cloud/ follow data governance practices?

Azure Event Hubs for Big Data Event producers Collection Ingress Stream Processing Long-term storage Presentation / action Applications Fast Data Service bus Azure DBs HDInsight Azure Storage Search and query Cortana Analytics PowerBI Dashboards Cloud gateways (web APIs) Event hubs Legacy IOT (custom protocols) Stream processing IP-capable systems (containers/VMs/servers) Slow Data Devices Azure Data Lake Field gateways Low-power devices (RTOS) Devices to take action

Event Hubs  Data Warehouse - using Event Grid (Demo) Stream big data into a data warehouse GitHub sample Deploy the infrastructure with this sample template.

Coming soon! Azure Geo DR General Availability IP filtering VNet integration Dedicated clusters