1. 5/9/2018 7:28 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Demystifying Cloud Data Services for an App Developer
5/9/2018 7:28 AM
P4016
Demystifying Cloud Data Services for an App Developer
Romit Girdhar
Software Engineer
Wee Hyong Tok
Program Manager
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Evolving Requirements
Evolving Data Processing Requirements
VALUE
Growth of data
Machine learning and AI
Any data
In-memory
Internet of Things
INTERNET CONNECTED
Hadoop
Dashboards
Ad hoc analysis
Operational reporting
CLOUD
DIGITAL
ANALOG
Enterprise data warehouse
OLAP
ETL
MOBILE
[wh]
Good day, everyone!
Thank you for tuning in to this session.
As we work with customers, one of the things we have been hearing a lot is the evolving data processing requirements.
From transactional systems, to doing ETL and bringing the data into an enterprise data warehouse
To wanting to reduce the end to end latency to derive insights (from days to minutes, to even seconds).
And to be able to do scale… working on big data
And wanting to build intelligent systems using Machine Learning/AI on data of various shapes and forms…
This session will help demystify the data services that you can use to achieve all of this…
Super exciting!
Complex implementations
Spreadmarts
Siloed data
Transactional systems
Evolving Requirements

4. Transforming Data into Intelligent Action
5/9/2018 7:28 AM
Transforming Data into Intelligent Action
EXTRACT
Original data
TRANSFORM
ETL tool
(SSIS, etc.)
LOAD
(SQL Sever, Teradata, etc.)
EDW
Transformed data
OLTP, ERP, LOB, ...
BI tools
Data marts
Apps
Dashboards
(On-premise and in the cloud)
INGEST
Original data
Scale-out, storage, and compute
(HDFS, Blob storage, etc.)
TRANSFORM AND LOAD
Devices, social, sensors, web
#MSBuild
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Common Components of Data-driven Solutions
Microsoft Build 2017
5/9/2018 7:28 AM
Common Components of Data-driven Solutions
Ingestion – Incoming data
Processing – Analyze/Act on your data
Storing – Access your raw/processed data
Serving – Serve your data
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Microsoft Build 2017
5/9/2018 7:28 AM
Ingestion Patterns
Real-time Ingestion - Capable of ingesting millions of messages/sec
Near-real time Ingestion – Data received from web forms, etc., ingesting less than 1000msg/sec
One-time Load
Scheduled Periodic Loads
[romit]
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Microsoft Build 2017
5/9/2018 7:28 AM
Processing Patterns
Real-time Processing – Need to process each event as it arrives.
Batch Processing – Process large amounts of data on a periodic schedule
[romit]
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Data Serving Patterns Dashboarding/Reporting
Microsoft Build 2017
5/9/2018 7:28 AM
Data Serving Patterns
Dashboarding/Reporting
Ad-hoc Data Access – Access your data in your app
Interactive Data Access – Slice/Dice, Drill-down
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Microsoft Build 2017
5/9/2018 7:28 AM
Data Storage Patterns
OLTP/Hot Store – Great for quick access of data & light-weight Interactive queries
Warm Store – Cheap to store ; Great for batch processing
Archival/Cold Store – Cheap to store ; Data only needs to be accessed as an exception.
OLAP/Analytical Store – Similar to a hot store, but, provides fast and interactive access to your data
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Types of Data Stores Requirements Transient / Staging Big Data
Microsoft Build 2017
5/9/2018 7:28 AM
Types of Data Stores
Transient / Staging
Big Data
Relational
Non-Relational
JSON Documents
Column
Key-Value
Graph
Time-series
Requirements
Data Volume and Velocity
Real time vs Batch
Read/Write Latency
Types of data to be stored
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Solution Patterns 5/9/2018 7:28 AM [wh]
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Solution Pattern #1 - Real-time Serving
Microsoft Build 2017
5/9/2018 7:28 AM
Solution Pattern #1 - Real-time Serving
Hot/
OLTP Store
Application
Inbound
Data
Application
Cold/ Archival
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Solution Pattern #1 - Real-time Serving
Microsoft Build 2017
5/9/2018 7:28 AM
Solution Pattern #1 - Real-time Serving
Hot/
OLTP Store
Application
Inbound
Data
Application
Cold/ Archival
Ingest
Serve and
Consume
Store
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Solution Pattern #2 – Real-time Processing
Microsoft Build 2017
5/9/2018 7:28 AM
Solution Pattern #2 – Real-time Processing
Warm Store
(for later processing)
Inbound
Data
Message Queue
Near real-time processing engine
Dashboarding/Reporting
Hot/OLTP Store
Application
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Solution Pattern #2 – Real-time Processing
Microsoft Build 2017
5/9/2018 7:28 AM
Solution Pattern #2 – Real-time Processing
Warm Store
(for later processing)
Inbound
Data
Message Queue
Near real-time processing engine
Dashboarding/Reporting
Hot/OLTP Store
Application
Serve and
Consume
Ingest
Process
Store
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Solution Pattern #3 - Batch Processing
Microsoft Build 2017
5/9/2018 7:28 AM
Solution Pattern #3 - Batch Processing
Reports and
Dashboard
Other Data Sources
Data Processing
Advanced
Analytics
OLAP/Analytical Store
Application
Inbound
Data
Data in the
warm store
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. OLAP/Analytical Store
Microsoft Build 2017
5/9/2018 7:28 AM
Solution Pattern #3
Reports and
Dashboard
Other Data Sources
Data Processing
[romit]
Advanced
Analytics
OLAP/Analytical Store
Application
Inbound
Data
Data in the
warm store
Process
Ingest
Store
Serve and
Consume
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. What are my Technology Options?
5/9/2018 7:28 AM
What are my Technology Options?
[romit]
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. What are my tech options?
Microsoft Build 2017
5/9/2018 7:28 AM
What are my tech options?
Callouts:
This is not an exhaustive list (we’ve listed the common ones)
SQL DW is a big data processing tool
Hot store and Analytical stores are similar, though some like AAS are better for interactive querying
Benefit that Hadoop & Spark bring to the table
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Let’s test our knowledge…
Microsoft Build 2017
5/9/2018 7:28 AM
Let’s test our knowledge… ? ?
[romit]
Support Call Logs
Service Telemetry Data
Social Data
Ingest
Process
Store
Serve and
Consume
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Let’s test our knowledge…
Microsoft Build 2017
5/9/2018 7:28 AM
Let’s test our knowledge…
Power BI
Event Hubs
Stream Analytics
Real-time Insights and Alerts
Federated U-SQL Join Query (Reference Data)
Scale Built-in Cognition Capabilities
Azure Data Lake Store and Analytics
Azure SQL Database
Azure SQL
Data Warehouse
PolyBase over Azure Data Lake Store
Azure Analysis Services
[romit]
->
Support Call Logs
Service Telemetry Data
Social Data
Ingest
Process
Store
Serve and
Consume
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. How should I get started?
5/9/2018 7:28 AM
How should I get started?
[romit]
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Data Processing Flow – Real-time processing
Microsoft Build 2017
5/9/2018 7:28 AM
Data Processing Flow – Real-time processing
Call-outs:
For ASA, the only queue that currently works is EventHubs
Storm/Spark as supported as a part of HDInsight
“this” data means data in the current state/whether raw or processed
Difference/similarities between “Hot” store and “Serving” Store
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Data Processing Flow – Batch Processing
Microsoft Build 2017
5/9/2018 7:28 AM
Data Processing Flow – Batch Processing
Yes No No No
Callouts:
- Partition your “big” data. For the most part, by Date-time
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Call to action Get started in minutes http://aka.ms/cisolutions
Reference architecture for common scenarios
Built on best practice design patterns
Automated deployment on your Azure subscription
Customizable for your needs
Supported by a global partner ecosystem
Code Labs – Building your First App
Get started in minutes

26. Related sessions Learn more about building Data-driven Apps for Scale:
Microsoft Build 2017
5/9/2018 7:28 AM
Related sessions
Learn more about building Data-driven Apps for Scale:
B8040: How JCI built next-gen Data-driven applications at scale
Thursday, May 11th, 2017: 4.00pm – 5.00pm PST
B8081: How to serve AI with Data: The future of the data platform
Wednesday, May 10th, 2017: 2.00pm – 3.00pm PST [Live streaming available!]
B8018: How to build global scale IoT applications with Azure SQL DB
Thursday, May 11th, 2017: 3.30pm – 4.30pm PST
Code Stories Theater sessions
#MSBuild
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. 5/9/2018 7:28 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.