1. Design big data applications using Azure Storage
11/20/2018 3:02 PM
Design big data applications using Azure Storage
Chris Brooks
Principal PM Manager
Microsoft Azure Storage
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Infrastructure Services
Platform Services
Infrastructure Services
Compute
Storage
Datacenter Infrastructure (42 Regions Announced, 36 Online)
Application Platform
Web
Apps
Mobile
API
Notification
Hubs
Hybrid
Cloud
Backup
StorSimple
Azure Site
Recovery
Import/Export
Networking
Data
SQL
Database
CosmosDB
Redis
Cache
Azure
Search
Tables
SQL Data
Warehouse
Azure AD
Health Monitoring
Virtual Network
Express
Route
Blob
Files
Disks
Virtual Machines
AD Privileged
Identity
Management
Traffic Manager
App
Gateway
Operational
Analytics
Compute Services
Services
Batch
RemoteApp
Service
Fabric
Developer Services
Visual Studio
Application
Insights
VS Team Services
Containers
DNS
VPN Gateway
Load Balancer
Domain Services
Analytics & IoT
HDInsight
Machine
Learning
Stream Analytics
Factory
Event
Data Lake
Analytics Service
IoT Hub
Catalog
Security & Management
Azure Active
Directory
Multi-Factor
Authentication
Automation
Portal
Key Vault
Store/
Marketplace
VM Image Gallery
& VM Depot
B2C
Scheduler
Xamarin
HockeyApp
Power BI
Embedded
SQL Server
Stretch Database
Engagement
Functions
Intelligence
Cognitive Services
Bot Framework
Cortana
Security Center
Container
Queues VM
Scale Sets
Data Lake Store
Dev/Test Lab
Integration
BizTalk
Service Bus
Logic
Media & CDN
Content Delivery
Network
Media Services
Media Analytics

3. Microsoft Ignite 2016
11/20/2018 3:02 PM
Azure Scale
36 GA, 6 coming soon – Storage is available in every region
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Azure Storage Services
Microsoft Build 2016
11/20/2018 3:02 PM
Azure Storage Services
IaaS
Virtual machines
Storage
Networking
PaaS
Existing frameworks
Web and mobile
Serverless Compute
Microservices
Disks
Persistent disks for Azure IaaS VMs
Standard Storage Disks:
Magnetic disk based, low IOPS, moderate latency
Premium Storage Disks: SSD based, high IOPS, low latency
Managed Disks
Files
Fully Managed File Shares in the Cloud
SMB and REST access
“Lift and shift” legacy apps
Blobs
Highly scalable, REST based cloud object store
Block Blobs: Sequential I/O, Hot, Cool and Archive Tiers
Page Blobs: Random-write pattern data
Append Blobs
Tables
Massive auto-scaling NoSQL store
Dynamic scaling based on load
Scale to PBs of table data
Fast key/value lookups
Queues
Reliable queues at scale for cloud services
Decouple and scale components
Message visibility timeout and update message to protect against unreliable dequeuers
Built on a unified Distributed Storage System
Durability, Encryption at Rest, Strongly Consistent Replication, Fault Tolerance, Auto Load-Balancing
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Big Data from IoT, Genomics, etc
Microsoft Build 2017
11/20/2018 3:02 PM
Blob Storage Service
App and Web scale data
Backups and Archive
Big Data from IoT, Genomics, etc
Store and serve
Block Blobs
Most object storage scenarios
Append Blobs
Multi-writer append only scenarios
Page Blobs
Page aligned random reads &writes
Writer 1
Writer 2
Writer 3
Block 1
Block 4
Block 3
Block 2
Image.jpg
Sparse File
512 byte aligned
512
1024
1536
2048
2560
Block 1
Block 2
Block 3
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Big Data Ecosystem

7. Big Data with Azure Blob Storage
Microsoft Build 2017
11/20/2018 3:02 PM
Big Data with Azure Blob Storage
Big Data Use Cases
Ingest & ETL
Streaming
Analytics & Machine Learning
Data Aggregation
Presentation
Machine
Learning
Stream Analytics
Batch
Functions
Data Lake Analytics
Data Factory
App Insights
Log Analytics
Monitor
IoT Hub
Event Hubs
Data Warehouse
CDN
Search
Power BI
Cognitive services
Azure HDInsight
Blob Storage
Blob Storage Pillars
Open & Interoperable
Manageable & Cost Efficient
Scalable & Performant
Secure & Compliant
Durable & Available
Storage
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Demo: Storage + Spark + Search
Gaurav Mantri
Microsoft Azure MVP
CEO, Cynapta Software

9. SDK/Tools Developers Tools Open Source client libraries on GitHub
Retry/resume, Parallel upload/downloads, etc.
Mobile platform support
Data movement library
High performance data transfers
Cross platform compliant – CoreCLR based
Tools
Storage Powershell – Linux support
CLI – Python based, Linux friendly
AZCopy – Robocopy for Azure Storage
Storage Explorer – Cross platform UX

10. Azcopy for Linux Familiar Unix syntax
11/20/2018 3:02 PM
Azcopy for Linux
Familiar Unix syntax
Built on CoreCLR and Storage Data Movement Library
Fast data transfer with pause/resume
Preview available now -
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Demo: AZCopy on Linux

12. Introducing Azure Data Box PREVIEW
Fast and Easy
Rent an Azure Data Box and transfer about 100 TB of data to Azure in around a week. Data Box uses standard NAS protocols.
Safe and Secure
Azure Data Box is tamper-resistant and ruggedized for shipping. Data is protected with AES-256 encryption for safe transit.
Partner Supported
Azure Data Box is integrated with a global array of industry-leading Azure partners.
Sign up today:

13. Coming Soon: FUSE adapter for Blob Storage
Mount a container as a filesystem with blobfuse
Basic POSIX File operations supported (others coming soon)
Shared storage for Compute nodes; scales up to the Storage account limits
Local cache on the ephemeral disk to speed up subsequent reads

14. Demo: BlobFuse

15. Hadoop File System and Azure Storage
Microsoft Build 2017
Hadoop File System and Azure Storage
11/20/2018 3:02 PM
Applications
Hadoop Shell Commands
WASB Improvements
Vastly reduce ramp up time for workloads from Azure HDInsight clusters minimizing throttling/timeouts due to ramp up
Improved support for HDFS Flush and Sync semantics
Improved handling of intermittent errors without failing HDI jobs
Java Storage SDK version upgraded to provide larger blob size support
Performance improvements for data reads / writes
Hadoop FileSystem API Interface (Extensible)
HDFS
Azure Storage Integration (WASB) …
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Demo: HDInsight with Azure Storage

17. Blob Storage events through Azure Event Grid
Event publishers
Event handlers
Resource Groups
Event Hubs
Azure Subscriptions
Custom Events
Blob Storage
Azure Functions
Azure Automation
Logic Apps
WebHooks

18. GEOSYS on Azure

19. For agriculture For insurance For commodities
Monitor and benchmark fields, optimize inputs
For insurance
Minimize risk with unbiased, third-party data
For commodities
Objectively monitor and
analyze crop production

20. GEOSYS Azure Data Pipeline
Ingress
Raw
Image Processing
Processed
Blob Storage
Indexing
Blob Storage
ScyllaDB
Custom
HDInsight

21. Big Data Economics

22. Lower Transaction cost
11/20/2018 3:02 PM
Tiered Storage
Introducing Blob-Level Tiering
Individual blob can move between tiers
All tiers of blobs co-exist in the a storage account
Blob REST API
Hot Tier
Lower Transaction cost
Cool Tier
Lower Capacity cost
Blob-Level Tiering
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Lower Transaction cost
11/20/2018 3:02 PM
Tiered Storage
Introducing Blob-Level Tiering
Individual blob can move between tiers
All tiers of blobs co-exist in the a storage account
New Storage Tier – Archive Storage
Cold storage for long term data
Retrieval latency is hours
Consistent API Among Storage Tiers
Access through Blob REST API
Support direct writes to Archive
Blob REST API
Hot Tier
Lower Transaction cost
Cool Tier
Lower Capacity cost
Archive Tier
Lowest Capacity cost
Blob-Level Tiering
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Big Data Scalability

25. Individual Blob Performance & Scale Improvements
Microsoft Build 2017
11/20/2018 3:02 PM
Individual Blob Performance & Scale Improvements
Capacity
Single Block Blob max size increased 25x to 5TB
Max block size increased 25x to 100MB
Throughput
Read throughput improvements by 3x for single/ few readers
Read throughput increase up to 2GB/s with 50+ parallel readers.
Write throughput improvements by ~3x
Improved read latency for blobs <4MB
5 TB
200 GB
2 GB/s
60 MB/s
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. New - Storage Account Scalability
Microsoft Build 2017
11/20/2018 3:02 PM
New - Storage Account Scalability
Scale Target increases per account
500TB  5 PB storage capacity
20K  50K Requests/sec
20 Gbps  50 Gbps Bandwidth
Contact Azure Support to enable
More coming in late CY17 & CY18
Optimized for rapid storage scale out for HDInsight & HPC workloads
5 PB
500 TB
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Disk Scale Roadmap Up to 32TB Up to 4TB Now
11/20/2018 3:02 PM
Disk Scale Roadmap
Up to
32TB
Up to 4TB
Now
Premium: 32GBNew, 64GBNew, 128GB, 512GB, 1TB, 2TBNew, 4TBNew
(7,500IOPS, 250MBps for Premium)
Standard: 32GB, 64GB, 128GB, 256GBNew
512GB, 1TB, 2TBNew, 4TBNew
Early CY2018
Max 32TB Disk Size for Premium & Standard Managed Disks
Higher IOPS and Throughput
Premium Managed Disks: 256GB
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Mojio on Azure
The Leading Open Platform for Connected Cars

29. Mojio unlocks hidden data from nearly any vehicle
Extended context and value-added services
HARDWARE AGNOSTIC
Mojio unlocks hidden data from nearly any vehicle
Main message for this slide goes here

30. Launched Nov ’16, over 300,000 Subscribers $10/month
Main message for this slide goes here

31. Unlocking valuable data that matters
Trip Start/End
GPS Location
Frequency & Patterns
Harsh Braking
Rapid Acceleration
Accidents
Weather
Traffic
Speed Limits
Road Types
Bumps & disturbances
VIN
Ignition ON/OFF
VSS Speed
Battery Voltage
Check Engine Light
DTCs and PIDs
Recalls & TSB’s
OEM Services
Odometer
Unlocking valuable data that matters

32. Mojio Azure Data Pipeline
Collection
Ingress
Storage
Enrichment & Analytics
Sharing
Consume
Event Hub
Blob Storage
HDInsight

33. Demo: Configure Event Hubs Capture to Azure Storage

34. Big Data Security and Compliance

35. Security - Encryption Storage Service Encryption at Rest
Microsoft Build 2017
11/20/2018 3:02 PM
Security - Encryption
Data disclosure prevention from physical disk compromise
Available now for Blob & File Storage with MS managed keys
Customer managed encryption keys - Preview H2 2017
Encryption on for all accounts - H2 2017
Storage Service Encryption at Rest
Storage Service Encryption in transit
Storage REST APIs support HTTPS
SAS Tokens can be restricted for HTTPS only
(New) “Secure Transfer” option limits all access to HTTPS only.
AES based, CBC mode source with MS provided or KeyVault based keys
Available in C#, Java, Python
Range downloads supported
Client side Encryption at source
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. Security – Network Layered security for Storage
Microsoft Build 2017
11/20/2018 3:02 PM
Security – Network
Layered security for Storage
Protection from key disclosure threats
Limit access to specific Azure VNets or public internet IP address ranges
Public Preview available now
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

37. Security - AAD Authentication and RBAC
Microsoft Build 2017
11/20/2018 3:02 PM
Security - AAD Authentication and RBAC
Currently support AAD, OAuth and RBAC on Storage Resource Provider via ARM
Adding AAD and OAuth to Blob Storage REST API
Integrating with Azure RBAC for Authorization
Role assignments down to container scope
Preview in H2 2017
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

38. Security - Apache Ranger Integration
Microsoft Build 2017
11/20/2018 3:02 PM
Security - Apache Ranger Integration
Allows granular user access control to Azure Storage Blobs in Hadoop.
Available in HDInsight Premium Clusters
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

39. Azure has the most comprehensive compliance coverage
11/20/2018
Compliance
Azure has the most comprehensive compliance coverage
CSA STAR Attestation
CSA STAR Certification
CSA STAR Self-Assessment
ISO 22301
ISO 27001
ISO 27017
ISO 27018
SOC 1 Type 2
SOC 2 Type 2
Global
U.S. Government
CJIS
DoD DISA SRG Level 2
DoD DISA SRG Level 4
DoD DISA SRG Level 5
FedRAMP
FIPS 140-2
High JAB P-ATO
IRS 1075
ITAR
Moderate JAB P-ATO
Section 508 VPAT SP
Industry
CDSA
FACT UK
FERPA
FFIEC
FISC Japan
GLBA
GxP 21 CFR Part 11
HIPAA/HITECH
HITRUST
IG Toolkit UK
MARS-E
MPAA
PCI DSS Level 1
Shared Assessments
Regional
Argentina PDPA
Australia IRAP/CCSL
Canada Privacy Laws
China DJCP
China GB 18030
China TRUCS
ENISA IAF
EU Model Clauses
EU-US Privacy Shield
Germany IT Grundschutz
India MeitY
Japan CS Mark Gold
Japan My Number Act
New Zealand GCIO
Singapore MTCS
Spain DPA
Spain ENS
UK G-Cloud
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

40. PureCars on Azure Mac White Director of Software Development

41. PureCars
Creating software for the automotive industry to drive higher sales for our customers
Started in 2008, with a third party used car product for dealer websites
Expanded into digital advertising in 2013 and has grown from an initial 20 dealer AdWords advertising portfolio to over 1500
From 2015 to today, we have expanded to customers using multiple digital advertising platforms

42. Multiple Big Data Problems
Inventory Data Ingestion
We need to take in roughly 2.5 million vehicles data daily and process it into usable advertising
Advertising Performance Data
We retrieve data from multiple platforms with multiple dimension and metric break downs.
Dealer Website Analytics
Dealer websites are increasingly the only auto shopping locations used by consumers.

43. Website Analytics
Problem: Dealer websites are increasingly the only auto shopping locations used by consumers.
The average showroom visit before purchasing a car is approximately 1.5
Increases the importance of interactions on the website
Digital Advertising even more vital to drive traffic to the site
Leverage consumer engagement to drive advertising and show value to dealers.

44. Web Analytics Architecture with Azure Storage
Dealer Websites
Collection
Ingress
Stream Processing
Storage
Warehouse
Consumer
ROI
Analytics
Events API
Stream Analytics
Blob Storage
Event Hub
Retarget

45. Next for PureCars Increase use of PaaS Azure Offerings
Further data collection and analysis
Cloud Services to Microservices
PureCars Atlanta
Atlanta, GA

46. Azure Storage Sessions @ Ignite
11/20/2018 3:02 PM
SessionID
When/Where
Attend if you are interested in
Talk title
BRK2286
Tue 9/26 10: :00
Hyatt Plaza Intl G
Azure Files
Azure File Sync
Microsoft Azure File Sync - seamlessly extend file services across servers and cloud
BRK2161
Tue 9/26 12:30 - 1:45
OCCC W307
Windows File Server features
Maximize storage efficiency and conquer distributed file access…
BRK2371
Tue 9/26 2:15 - 3:30
Hyatt Windermere Y
Hybrid/Tiered Storage with Blobs
Partner solutions for Hybrid Storage
Best of both worlds: The benefits of cloud object storage…
BRK2256
Wed 9/27 09: :15
Hyatt Windermere X
Blob Storage (Many) New Features
Tiered Storage and Archive Tier
Azure Blob Storage: Scalable, efficient storage for PBs of unstructured data
BRK2265
Wed 9/27 10: :00
Backup/DR partners on Azure
Cost effective Backup using Blobs
Resilient, cost effective backup and disaster recovery with Azure Blob Storage
BRK2273
Wed 9/27 2:15 - 3:30
Hadoop and Big Data using Blobs
New Features in Blobs for Big Data
Design big data applications using Azure Storage
BRK2372
Thu 9/28 10: :00
Azure Managed Disks
Azure VM Disks Roadmap
Azure Disks: Scale, performance, and ease of use
BRK2288
Thu 9/28 12:30 - 1:45
Industry use of object storage (Blobs)
Many customer/partner case studies
Azure Storage: Digital transformation in key industries
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

47. What’s next at Ignite with Azure Storage?
11/20/2018 3:02 PM
What’s next at Ignite with Azure Storage?
Follow-up talks
Backup/DR solutions on Blobs  10:45-12p Windermere X
Big Data on Blobs  2p-3:15p Windermere X
Vertical Industry solutions on Blobs  12:30-1:45
File/Hybrid Storage using Blobs  Recorded talk
We’d love to connect with you
Stop by the Azure Storage booth
Sign up for a 30 min 1:1 time slot this week:
now or after the conference
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

48. 11/20/2018 3:02 PM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

49. Please evaluate this session
Tech Ready 15
11/20/2018
Please evaluate this session
From your
Please expand notes window at bottom of slide and read. Then Delete this text box.
PC or tablet: visit MyIgnite
Phone: download and use the Microsoft Ignite mobile app
Your input is important!
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.