1. WPC047 Data ON THE ROAD: the Azure part
Jessica Tibaldi Tech Evangelist Microsoft
@_jetiba

2. Understand how to build a scalable and performant backend for your IoT solution to store and analyze data in the cloud using Azure services
Azure Machine Learning Studio
Azure HDInsight
Azure Data Factory
(extra) Azure Service Fabric
Agenda –

3. Demo Architecture mydriving-archive mydriving-hourlypbi
mydriving-sqlpbi
mydriving-vinlookup
mydrivingDB
mydrivingAnalyticsDB
IoT Hub
Power BI
Data Factory
HDInsight
Storage - Blob
Machine Learning
Event Hub
Service Fabric
Car (Sensor)
Xamarin App
(device) –

4. Build, deploy, and publish predictive analytics solutions
Machine Learning and Analytics
Machine Learning
Simple, scalable, cutting edge. A fully managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions.
Deploy in minutes. Azure Machine Learning means business. You can deploy your model into production as a web service that can be called from any device, anywhere and that can use any data source.
Publish, share, monetize. Share your solution with the world in the Gallery or on the Azure Marketplace. –

5. Machine Learning Flow and Algorithms
Define Objective
Collect Data
Prepare/ Clean Data
Construct/ Train Models
Score/ Evaluate Models
Publish
Manage
Type of Algorithms
Classification
Regression
Recommendation
Anomaly Detection
Clustering
Integrate –

6. Azure Machine Learning service
Data
Clients
API
ML STUDIO
Model is now a web service that is callable
Blobs and Tables
Hadoop (HDInsight)
Relational DB (Azure SQL DB)
Integrated development environment for Machine Learning
Monetize the API through our marketplace –

7. Azure Machine Learning DEMO–

8. Machine Learning and Analytics
BigData Analysis
HDInsight is a cloud implementation on Microsoft Azure of Apache Hadoop technology stack that is the go-to solution for big data analysis
Machine Learning and Analytics
Batch
Map Reduce
Script
Pig
SQL
Hive
NoSQL
HBase
Streaming
Storm
In-Memory
Spark
HDInsight
(Hadoop and Spark)
Core Engine
Scale to petabytes on demand
Process unstructured and semi-structured data
Develop in Java, .NET, and more
Skip buying and maintaining hardware
Deploy in Windows or Linux
Spin up an Apache Hadoop cluster in minutes
Visualize your Hadoop data in Excel
Easily integrate on-premises Hadoop clusters –

9. Hive
SQL-Like query syntax – if you know SQL, you’ll be able to use Hive
Relational set algebra mixed with row-oriented manipulation
Declare tables (internal and external) and views
Query processor optimizes MapReduce job –

10. Compose and orchestrate data services at scale
SQL
DATA SOURCES
{ }
PREPARE
TRANSFORM & ANALYZE
PUBLISH
SQL
DATA CONSUMPTION
INGEST
<>
Information Management
Data Factory
Create, schedule, orchestrate, and manage data pipelines
Visualize data lineage
Connect to on-premises and cloud data sources
Monitor data pipeline health
Automate cloud resource management
Move relational data for Hadoop processing
Transform with Hive, Pig, or custom code –

11. Data Factory Elements
Pipelines a grouping of logically related activities that performs a task
Linked Services define the information needed for Data Factory to connect to external resources
Activities define the actions to perform on your data
data store
compute resource
Data transformation
Datasets Datasets identify data within different data stores, such as tables, files, folders, and documents.
Data movement –

12. Example Datasets Pipeline Activities
Pipeline (Active Period: July 2016 to July 2017)
Datasets
Pipeline
Activities –

13. Activity type, properties
& parameters (if required)
Inputs & outputs
Policy & schedule

14. DEMO Azure Data Factory–

15. Service Fabric - Microservices apporach
Compute
Service
Fabric
High scalability
High reliability
High availability
Constant application evolution
Deployment and update speed
Development agility
Resource optimization and cost reduction –

16. Service Fabric in the demo application
VINLookupService (stateless) looks up additional vehicle information and saves that to a SQL db
IoTHubPartitionMap (stateful) obtains the Event Hub partition key which VINLookupService uses to connect –

17. Additional extension routes…
Real-time identification of nearby points of interested based on GPS coordinates
Identification of the driver identity in a vehicles fleet management scenario –

18. Some scenarios… Vehicle diagnostic Fleet management
The front brakes are needing to be serviced sooner than expected
Fleet management
24 vehicles are shown on a map, showing status
Engine emission control
Vehicle B204 is driving in eco-mode 78% of the time
Roadside assistance
Tow truck is on its way to vehicle B204
Eco-driving
14/15 vehicles meet standards and 1 is scheduled for maintenance
Engine performance
Temperature is beyond the ideal range for 13 vehicles
Usage-based insurance
3 vehicles have daily mileage that qualify them for reduced rates –

19. Q&A
Domande e Risposte –

20. MyDriving Docs https://azure. microsoft
Useful Links –

21. Contatti OverNet Education
Tel
@overnete
Contatti OverNet Education –

22. Appendix –

23. Input data
Data Transformation
Define model
Train model

24. Score model => Prediction
Evaluate model => Prediction

30. Azure Event Hub partitions system Azure Event Hub
IEventProcessor
Event Processor Host
Direct
Partition 1
Consumer Group
Event Receivers
> 1M Producers > 1GB/sec Aggregate Throughput
Partition 2
Partitions
Partition 3
Partition 4
PartitionKey
Partition 5
Event Producers
AMQP
HTTPS
Hash
Partition 6
Partition 7
Consumer Group 2
Partition 8
Throughput Units:
1 ≤ TUs ≤ Partition Count
1 MB/sec or 1,000 events/sec ingress
2 MB/sec or 2,000 events/sec egress
TUs are billed by the hour
Up to 32 partitions via portal, more on request up to 1024
Define the maximum degree of downstream parallelism
Consumer Groups:
A view on the event stream
Can create up to 20 named consumer groups
AMQP

31. Dashboards & Visualizations
Keep a pulse on your business with live, interactive dashboards
Stream Analytics
Dashboards & Visualizations
Power BI
Event Hubs
Power BI
Machine Learning
Power BI
Storage
SQL database
HDInsight
Power BI
Analytics for everyone, even non-data experts
Your whole business on one dashboard
Create stunning, interactive reports
Drive consistent analysis across your organization
Embed visuals in your applications
Get real-time alerts when things change

32. Data Sources Ingest Prepare Analyze Publish Consume Power BI
Machine Learning
Power BI
Diagnostic Streaming
Event
Hubs
Stream Analytics
Sensors and devices
Stream Analytics
Machine Learning
HDInsight
HDInsight
SQL Data
Warehouse
Business apps
Azure Blob storage
Data Factory: Move data, orchestrate, schedule and monitor
Enterprise
data sources
Data Catalog: Register, annotate, understand, discover data sets