Observation Pattern Theory Hypothesis What will happen? How can we make it happen? Predictive Analytics Prescriptive Analytics What happened? Why did it happen? Descriptive Analytics INFORMATION Diagnostic Analytics OPTIMIZATION Confirmation Theory Hypothesis Observation

Implement Data Warehouse Physical Design ETL Development Reporting & Analytics Development Install and Tune Reporting & Analytics Design Dimension Modelling ETL Design Setup Infrastructure Understand Corporate Strategy Data sources ETL BI and analytic Data warehouse Gather Requirements Business Requirements Technical Requirements

Ingest all data regardless of requirements Store all data in native format without schema definition Do analysis Using analytic engines like Hadoop Interactive queries Batch queries Machine Learning Data warehouse Real-time analytics Devices

What happened? What is happening? Why did it happen? What are key relationships? What will happen? What if? How risky is it? What should happen? What is the best option? How can I optimize? Data sources

Massive Compute and Storage Deployment expertise Data of all Volume Variety, Velocity Speed Scale Economics Always Up, Always On Open and flexible Time to value

Azure Facts >4 trillion objects in Azure 300,000-1M+ requests per second Double compute and storage every 6 months Azure Storage HDInsight Data Factory ML Stream Analytics Database DocumentDB Search Event Hubs

Microsoft’s cloud Hadoop offering 100% open source Apache Hadoop Built on the latest releases across Hadoop (2.6) Up and running in minutes with no hardware to deploy Harness existing.NET and Java skills Utilize familiar BI tools for analysis including Microsoft Excel

Use CasesWhere? Active Archive / Compliance Reporting Restricted data = “down here”. “Up there” could be considered for other scenarios. ETL / Data Warehouse Optimization Often has “down here” gravity, but cloud-based ETL offload has big payout Smart Meter AnalysisTypically born “up there” Single View of Customer May have heavy “down here” gravity; unless you’re using SaaS apps, then why not “up there”? New Data for Product Management Restricted data = “down here”. “Up there” could be considered for many scenarios. Vehicle Data for Transportation/LogisticsWhy not “up there”? Vehicle Data for InsuranceMay have heavy “down here” gravity (ex. join w/risk data, etc.)

Rockwell Automation is partnered with one of the six oil and gas super majors to build unmanned internet-connected gas dispensers. Each dispenser emits real-time management metrics allowing them to detect anomalies and predict when proactive maintenance needs to occur. Store sensor data every 5 minutes Temperature, pressure, vibration, etc. Tens of thousands of data points / second Azure Blobs Azure HDInsight Hive, Pig, Azure SQL DB Power BI for O365 Mobile Notification Hub Mobile Device Real-time notification

JustGiving wanted to harness the power of their data by using network science to map people’s connections and relationships so that they could connect people with the causes they care about. Based on 15 years of data, the JustGiving GiveGraph is the world’s largest ecosystem of giving behavior. It contains more than 81 million person nodes, thousands of causes and 285 million connections and is the engine that drives JustGiving’s social platform, enabling levels of personalization and engagement that a traditional infrastructure would be unable to deliver. SQL Server On-premises Agent Azure Blobs Azure HDInsight Give Graph Azure Tables Web API Website + Event store Service Bus Serves results Azure Cache Activity Feeds

Use Cases Ad Placement and Offers Active Archive ETL Offload Single View of Customer Recommendation Engine Customer Targeting and Acquisition New Data for Product Management Vehicle Data Web Personalization and Experience

Microsoft contribution to Apache code Hadoop 2.0 Sample Query Hive 10HDP 1.3 / Hive 11 HDP x Speedup 40X Speedup HDP x Speedup

Data Node Task Tracker Name Node Job Tracker HMaster Coordination Region Server

Stream processing Search and query Data analytics (Excel) Web/thick client dashboards Devices to take action

A hyper scale repository for big data analytic workloads