Big Data for the SQL Eye Cindy

Look, it’s SQL! SELECT score, fun FROM toDo WHERE type = 'they pay me for

And yet it’s more! CREATE EXTERNAL TABLE IF NOT EXISTS toDo (fun STRING, rank INT COMMENT 'rank the greatness', type STRING) COMMENT 'two tables walk into a bar....' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION

A mix of old and new -- read some data SELECT 'you cannot make me ', score, fun, type FROM toDo WHERE score <= 0 ORDER BY score; SELECT 'when can we ', score, fun, type FROM toDo WHERE score > 0 DISTRIBUTE BY score SORT BY

Visual job

That’s Hive folks! Hive on Hadoop on HDInsight on Azure Big Data in the

Back to You Have you used Big Data? Azure? What questions do you have? What do you want to know by the end of this talk? What makes your projects go right or wrong? Will you use Big

Big Data

Part 3: Single Slide A leading game development studio that creates, develops, produces, and publishes a number of popular video games needed to analyze large amounts of in-game data that were unstructured. They chose Azure HDInsight, Data Factory, SQL Server on-premises, Power View, Power Query to do in- game analytics and understanding what gamers do during game-play and what campaigns they can run to influence in-game purchases. Finally, twitter sentiment is collected to correlate with sales.

Game Development Company Gaming A predominantly mobile-based game development company. While they are a mid-sized organization, they have partnered with media giants on various gaming projects Part 1: What They Did | In-game Analytics Challenge As a game development studio, they wanted to do in-game analytics to understand their players more and what they do in the games Solution Azure HDInsight (MapReduce and Storm), Service Bus, SQL Server for reporting Collects telemetry and logging data to gain in-game analytics: How many players using the game How many players invited their friends How far along did players get into the tutorial How many attempts did they make on one level/stage In-game Analytics Media tonic

BK1 Game Development Company Part 2: How They Did It | In-game Analytics How They Did It Collect data from games in Azure Blobs Game sends telemetry/logging data as JSON files Contains every action of user in the game Data is pushed to Azure Service Bus as real-time Tens of Gigabytes of data captured daily HDInsight picks up real-time data and processes From Service Bus, HDInsight processes using Apache Storm and MapReduce Constantly running experiments to determine insight A/B testing In-game metrics and analytics Spin up 32-node cluster nightly for four hours Output sent to SQL Server for BI Transfer data to SQL Server for BI In-game Analytics Service Bus SQL Server On-premises

Part 3: Single Slide A game development studio that wanted to do in-game analytics to understand their players more and what they do in their games. They chose Azure HDInsight including Storm in HDInsight so they can do near real-time in-game analytics of their users. Now, they can understand how many players are playing, how many are referring the game, how difficult a game level is, etc.

Typical Big Data Use Cases Smart meter monitoring Equipment monitoring Advertising analysis Life sciences research Fraud detection Healthcare outcomes Weather forecasting Natural resource exploration Social network analysis Churn analysis Traffic flow optimization Legal discovery Telemetry IT infrastructure optimization

It depends It’s mostly true

Hadoop Shines When…. Data exploration, analytics and reporting, new data-driven actionable insights Rapid iterating Unknown unknowns Flexible scaling Data driven actions for early competitive advantage or first to market Low number of direct, concurrent users Low cost data archival

Hadoop Anti-Patterns…. Replace system whose pain points don’t align with Hadoop’s strengths OLTP needs adequately met by an existing system Known data with a static schema Many end users Interactive response time requirements (becoming less true) Your first Hadoop project + mission critical system

You tell me… What is Big Data? What is Hadoop? What specific scenario would you use it

Hortonworks

Business Scenarios Recommendations, customer churn, forecasting, etc. Perceptual Intelligence Face, vision Speech, text Personal Digital Assistant Cortana Dashboards and Visualizations Power BI Machine Learning and Analytics Azure Machine Learning Azure Stream Analytics Cortana Analytics Suite DATA Business apps Custom apps Sensors and devices INTELLIGENCEACTION People Automated Systems Big Data Stores Azure SQL Data Warehouse Information Management Azure Data Factory Azure Data Catalog Azure Event Hub Azure Data Lake Store Azure HDInsight (Hadoop) Azure Data Lake Analytics

Azure Data Lake - The

Azure Data Lake store Azure Data Lake managed clusters Hadoop Cluster Azure Data Lake: store & managed clusters HDFS/WebHDFS API

HDInsight “managed Hadoop clusters” Input File Output File n1n2n3 n4 Hive/Pig/etc. job ADLA Account lots of containers YARN Layer U-SQL/Hive/Pig job ADLA “analytics service” HDInsight Cluster Storage (Blob or ADLS) Analytics: Two Form Factors 45

Azure Data Lake - The Action

Azure has so much more Go straight to the business code Scale storage and compute separately Open Source Linux Managed and unmanaged services Hybrid On-demand and 24x7 options SQL

It’s a Polyglot Stream your data into a lake Pick the best compute for each

And it’s

Big Data for the SQL Eye Cindy

And back to

What is Big Data? It Is Scale Out Enables elasticity Encourages exploration Faster data ingestion Lower TCO Empowers self-service BI and analytics Rapid time to insight It Is NOT A well-defined thing About volume, size A replacement for everything The answer to every problem

What is Hadoop? Conceptual View It Is A type of Big Data Just another data source A loose collection of open source code Distributed by many Handles loosely structured data Write once, read many It Is Not Actually a thing! The only way to do Big Data

Business Scenarios Recommendations, customer churn, forecasting, etc. Perceptual Intelligence Face, vision Speech, text Personal Digital Assistant Cortana Dashboards and Visualizations Power BI Machine Learning and Analytics Azure Machine Learning Azure Stream Analytics Cortana Analytics Suite DATA Business apps Custom apps Sensors and devices INTELLIGENCEACTION People Automated Systems Big Data Stores Azure SQL Data Warehouse Information Management Azure Data Factory Azure Data Catalog Azure Event Hub Azure Data Lake Store Azure HDInsight (Hadoop) Azure Data Lake Analytics

Microsoft Azure Data Lake YARN U-SQL Analytics Service HDInsight Store HDFS