1. Cloud Systems of Intelligence design patterns Making it real for Product Engineering Session: BRK3321 Larry Persaud – Principal Lead : AI+R Abhinav Mithal – Senior Manager : AI + R September 26th, 2017

2. here? Why should you care?
AI Powered Organizations are
getting ahead – Fast!
Why are you
here? Why should you care?
Massive Compute, Powerful
Algorithms, DNN in Cloud
Massive amount of info
fueling powerful applications
How to lead and sustain
digital transformation

3. Business is being transformed by three trends
Cloud
Intelligence
Big Data

4. Successful Companies Achieve Broad Market Adoption
Find It
Nail It
Scale It
Beachhead
Early Adopters
Broad Market
Solution Design Patterns
Path toward
Market Maker Products

5. Data Science Solution Workflow
Data Sources
Ingest
Prepare
Analyze
Serve
Consume
Understand Business Goals and ?’s
Debug, Fix, Enhance, etc.
Discover & Gather Data
Ingest Data
Under-stand Data
Transform Data
Create Model
Deploy Model
Monitor & Maintain Model
Apps
Services
Respond to changes/lessons
Data Engines
Share Results with Business Owners
Collaboration & Version Control
Documentation

6. Building Systems of Intelligence on Azure
Information Management
Big Data Stores
Machine Learning and Analytics
Intelligence
People
Data Sources
Azure Storage
Data Lake Store
Machine Learning
Cognitive Services
Data Factory
SQL Data
Warehouse
Data Lake Analytics
Bot Framework
Apps
Web
Mobile
Bots
Data Catalog
Apps
HDInsight
(Hadoop and Spark)
Event Hubs
Cosmos DB
Cortana
Kafka for HDI
Sensors and devices
Stream Analytics
Dashboards & Visualizations
Automated Systems
Analysis Services
Power BI
Intelligence
Data
Action 6

7. Demo: Interactive Voice Response Bot

9. Starting with Customers

10. Industry and Use Cases Industry Use Cases Manufacturing Finance Retail
Quality Assurance
Predictive Maintenance
Fault detection
Manufacturing
Real time fraud prevention
Financial Forecasting
Financial Analysis
Finance
Personalized Ad Recommendations
Computer Vision
AI Bots
Retail
Insurance fraud prevention
Patient Population Management
Equipment PM etc.
Health Care

11. Design Patterns Design Patterns Use Cases
Modern DW (e.g. telemetry reporting, revenue reporting)
Data Prep & Integration for Data Driven SaaS
Data Prep for ML Scoring (e.g. Predictive Maintenance, Prod Recommendations)
Batch Big Data Processing
@Scale
IoT: Streaming vehicle/driver insights & reporting
Real time fraud prevention
Stream Processing
@ Scale
IoT: Device monitoring / anomaly detection
Personalized Ad Recommendations
Fraud Detection
Computer Vision
Cloud Experimentation
and Operationalization
AI Bots

12. Several Architectures…`
Several Architectures…
(too many doing the same thing)

13. Big Data Processing @ Scale Approach
Prepare &
Transform
Ingest
Store
Anglyze
Stage
Consume
Import/Export
Rest
Cosmos DB
ADF
Blob
ADF
BI/Exploration Tools
AzCopy
ADLS
HDI Spark
HDI Spark
SQL DW
Application Code/Store
Kalfa
SQL DW
ADLA
ADLA
SQL DB
Archive(Blob, ADLS, etc.)
Streaming Data
IOT Hub
ASA
Cosmos DB
[Batch] VM
[Batch] VM AS
Events Hubs
HDI/Storm
SQL DW
SQL DW
SQL/IaaS
Service Bus
Event Proc. Host
AML
SQL Orcas
MRS
Orch. & Monitor across stages:
Data Flow

14. Another Approach Prepare & Transform Ingest Store Anglyze Stage
Consume
Import/Export
Rest
ADF
Blob
ADF
Cosmos DB
BI/Exploration Tools
AzCopy
ADLS
HDI Spark
HDI Spark
SQL DW
Application Code/Store
Kalfa
SQL DW
ADLA
ADLA
SQL DB
Archive(Blob, ADLS, etc)
Streaming Data
IOT Hub
ASA
Cosmos DB
[Batch] VM
[Batch] VM AS
Events Hubs
HDI/Storm
SQL DW
SQL DW
SQL/laaS
Service Bus
Event Proc. Host
AML
SQL Orcas
MRS
Orch. & Monitor across stages:
Data Flow

15. Yet Another Approach
Prepare &
Transform
Ingest
Store
Anglyze
Stage
Consume
Import/Export
Rest
ADF
Cosmos DB
Blob
ADF
BI/Exploration Tools
AzCopy
ADLS
HDI Spark
HDI Spark
SQL DW
Application Code/Store
Kalfa
SQL DW
ADLA
ADLA
SQL DB
Archive(Blob, ADLS, etc)
Streaming Data
IOT Hub
ASA
Cosmos DB
[Batch] VM
[Batch] VM AS
Events Hubs
HDI/Storm
SQL DW
SQL DW
SQL/laaS
SQL Orcas
Service Bus
Event Proc. Host
AML
MRS
Orch. & Monitor across stages:
Data Flow
Mode
Original Transactions
Processing (Spark)
Distribution (to SQLDB – P15)
Stream
7 Million Rows
5 node
4-5 mins
Scale
300 Million Rows
12 node
2 Hr (ADF)
8 Hr (Direct from Spark)
Batch Massive Scale
3000 Million Rows
50 node
20+ Hours (ADF)
Note: Dark Blue boxes form a optimal path for the big data processing at scale

16. Pattern 1: Batch Big Data + ML @ Scale

17. What is Big Data Processing
Stage
Processes
Connect and collect data from various data sources
Capture data from real time data streams
Data Ingestion
Schedule and monitor Data collection.
Add structure of raw data
Combine data in various format and structure
Create a common data schema for down stream consumption
Harmonize the data
Prepare data aggregates and summary
Create time series lags
Create feature vectors
Featurize the Data
Create ML predictions
Create Alerts
Retrain the ML models
Analyze and Predict

18. Big Data Processing @ Scale
Lambda Architecture
Big Data Scale
Use Cases
Built a modern DW
Data preparation for data-driven SaaS Apps
Data preparation for ML scoring
Prepare and transform any data, on demand or reoccurring
Azure Services
The Pattern
Data Factory
Event Hub
Stream Analytics
Data Lake Store / Blob
HDInsight / ADLA / Batch
SQL DW / SQL DB / AS / Cosmos DB
Ingest: move data from any source (on prem or cloud) reliably at scale
Store: enables large scale batch processing, low $/GB (vs. traditional DWs), support [un and/or semi]structured data
Prepare & Transform: normalize & clean scale. Shape data to target (e.g. Dims & Facts)
Analyze: Enrich data (via joins, ML model execution, etc)
Stage: “consumption ready” data into the optimal store for the scenario (e.g. DW for BI, OLTP store for app, etc.)
Default path:
Prepare &
Transform
Ingest
Store
Anglyze
Stage
Consume
Disk Imp/Exp
Rest
ADF
Blob
HDI Spark
HDI Spark
ADF
DW / AS
BI/Exploration Tools
Streaming Data
Events Hubs
ASA VM
Cosmos DB
Application Code/Store
Blob
Archive(Blob, ADLS, etc)
Orch. & Monitor across stages:
ADF 18

19. Pattern 2: Cognitive Search (AI Powered Content Match)

20. IVR Bot Architecture Ingest Prepare Analyze Publish Consume
normalize, clean
text analysis
bot logic
incoming calls
Language Understanding(LUIS)
Bot
Connector
Skype Client
Language Understanding Intelligent System) 5 1 2
Bing Speech
Web App
speech-to-text
bot SDK 4
Real Time 3
Azure SQL
Azure Search 6
Cosmos DB
Batch ETL
product inventory
product matching + synonyms
session state

21. AI Stack for Intelligent Match and Search
Technologies
Cognitive APIs, AML
Azure Search + Custom Analyzers
Bot Framework + Azure web Apps
Skype Client 1
Incoming query 6
Final search results
Query/Result Orchestration (Azure Web App) 3
Query enrichment
Preprocess search query
Result enrichment
Post-process matching documents
Execute query 4
Matched documents 2
Enrich query
Enrich/rank results 5
Cognitive APIs (one or more)
Technical Properties
Scalable e2e system indexing 300MM documents with 60QPS.
Pluggable query enrichment and Result enrichment techniques to solve cognitive search and match and rank problems
Custom Recognizer
Speech-to-text
LUIS
Detect product entities
Azure ML
Classification, topics, etc.
LUIS
Detect product entities
Custom Recognizer
Speech-to-text
Azure Search
Analyzers
Custom text processing
Keyword match
Textual similarity
Filter
Structured metadata

22. Data and AI

23. Cloud AI Stack AI Applications Services Frameworks Processing
Cognitive Services
AML Web Services
BOT Framework
Inferencing
Frameworks
Docker
Machine Learning Toolkits
CNTK
Tensorflow
ML Server
Scikit-Learn
Other Libs.
Spark, SQL, Other Engines
Processing
Model & Experimentation Management
Spark
DSVM
AI Batch Training
ACS
PROSE Data Wrangling
Infrastructure
EDGE
Storage
CPUs
FPGA
COSMOS DB
ADLS
SQL DB
SQL DW
BLOB
GPUs
IOT

24. Data & AI | Building AI Cloud System of Intelligence
APIs
Frameworks
COGNITIVE SERVICES | BOT FRAMEWORK | AML WEB SERVICE
Batch
ML Toolkits
Interface & Interop
Services
CNTK
Tensorflow
ML Server
Scikit-Learn
Trigger-based
Machine Learning
Model Management & Experimentation
ML Experiment &Train
Processing
Spark
AI Batch
DSVM
Real-time
Storage
Data Preparation
BLOB
SQL DB
ADLS
COSMOS DB
Infrastructure
SQL DW
OTHER
Batch Data
Stream Data …
Data Ingest

25. Conclusion / Key Takeaways

26. Key Takeaways MSFT recommended architectures
MSFT 1st party enterprise solutions
MSFT solution journey
AI powered orgs are reaping benefits in cost reduction, improved quality, increasing the bottom line.

27. 9/12/ :21 PM
© 2017 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Objective Product Building Guide: “Find, Nail, Scale”
Customer Journey for Data Big and Small
Deep Dive AI Solutions
AI Stack, Data and Product Journey
Conclusion

29. Pattern Design Processes
Ingest
Transform
Prepare
Analyze
Serve
Consume
Pattern Design Processes