1. ADF & SSIS: New Capabilities for Data Integration in the Cloud
6/19/ :43 AM
BRK2254
ADF & SSIS: New Capabilities for Data Integration in the Cloud
Mike Flasko
Principal Group Program Manager
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Agenda Current State Feedback & Target Scenarios What’s New Roadmap
6/19/ :44 AM
Agenda
Current State
Feedback & Target Scenarios
What’s New
Roadmap
Q & A
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Current State: Data Integration in Azure
Azure Data Factory (ADF)
Provides orchestration, data movement and monitoring services
Orchestration model: time series processing
Hybrid Data movement as a Service w/ many connectors
Programmatic authoring, visual monitoring (.NET, Powershell)
SSIS: server software for ETL
Focused on ETL to/from SQL Server
Scale up data transformation engine
Visual authoring of control and data flow
Rich ecosystem (BIML, Task Libraries, etc)

4. Feedback: Do any of these sound familiar?
I need to create on-demand and/or event-triggered pipelines
I need to create a delta-processing pipeline
I need rich orchestration constructs to model my unique requirements (e.g. facilitate efficient restatements)
I need to reliably work with all my data across cloud, on prem, SaaS apps, etc
Data movement at scale
Data pipelines spanning services, servers, cloud/onprem, etc
ISVs: Need full programmatic access, using the languages and runtimes we are comfortable with
My data integration team are not developers, I need visual tools
Productivity of ETL is key, it is still 70%+ of overall solution time
Scale to 1000’s pipelines
How do I leverage my existing SSIS investments in the cloud?

5. Consumption Friendly Data
Modern Data Warehouse
Extract & Load
Prepare
Transform/Analyze
Extract & Load
Data Source 1
Data Source 2
DW & DM
BI Tools
Staging
Prepared Data
Consumption Friendly Data
“Data Lake”
Data Source N
DS Tools

6. Consumption Friendly Data
Modern Data Warehouse
Extract & Load
Prepare
Transform/Analyze
Extract & Load
Data Source 1
Data Source 2
DW & DM
BI Tools
Staging
Prepared Data
Consumption Friendly Data
“Data Lake”
Data Source N
DS Tools
Data-driven SaaS Application
App Storage
SaaS
App
Browser/Device

7. Consumption Friendly Data
Modern Data Warehouse
Extract & Load
Prepare
Transform/Analyze
Extract & Load
Data Source 1
Data Source 2
DW & DM
BI Tools
Staging
Prepared Data
Consumption Friendly Data
“Data Lake”
Data Source N
DS Tools
Data-driven SaaS Application
App Storage
SaaS
App
Browser/Device
Lift my existing SSIS packages to the cloud
Data Source 1
ETL: SSIS
SQL Server
ETL: SSIS
SQL Server
Data Source N

8. Managed Data Integration Service
ADF v2 Public Preview
New Pipeline Model
Rich pipeline orchestration
Triggers – ondemand, schedule, event
Data Movement as a Service
Cloud, Hybrid
30 connectors provided
SSIS Package Execution
In a managed cloud environment
Use familiar tools, SSMS & SSDT
Author & Monitor
Programmability (Python, .NET, Powershell, etc)
Visual Tools (coming soon)
Data Factory
Managed Data Integration Service

9. On Prem Apps & Data
Cloud Svcs, Apps & Data

10. Azure Data Factory v2 Service
Command and Control
Data
Data Factory
A data integration account
Location of orchestration, service metadata
UX & SDK
Authoring | Monitoring/Mgmt
Azure Data Factory v2 Service
Scheduling | Orchestration | Monitoring
Integration Runtime (IR)
ADF’s execution engine
Three core capabilities:
data movement
pipeline activity execution
SSIS package execution
Self Hosted
Integration Runtime
Azure
Integration
Runtime
On Prem Apps & Data
Cloud Svcs, Apps & Data

11. Azure Data Factory v2 Service
Command and Control
Data
Data Factory
A data integration account.
Location of orchestration, service metadata
UX & SDK
Authoring | Monitoring/Mgmt
Azure Data Factory v2 Service
Scheduling | Orchestration | Monitoring
Integration Runtime (IR)
ADF’s execution engine
Three core capabilities:
data movement
pipeline activity execution
SSIS package execution
Pipeline
SSIS Package
Self Hosted
Integration Runtime
Azure
Integration
Runtime
On Prem Apps & Data
Cloud Svcs, Apps & Data

12. Building Data Pipelines
6/19/ :44 AM
Building Data Pipelines
Modern DW
Data Driven SaaS Apps
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. ADFv2 Pipelines Trigger Activity 1 Activity 2 “On Error” Activity 1
My Pipeline 1
My Pipeline 2
For Each…
Trigger
Event
Wall Clock
On Demand
params
params
Activity 3
Activity 1
Activity 2
params
Activity 4
“On Error” Activity 1 …
params

14. ADFv2 Pipelines Trigger Activity 1 Data Flow “On Error” Activity 1
My Pipeline 1
My Pipeline 2
For Each…
Trigger
Event
Wall Clock
On Demand
params
params
Activity 3
Activity 1
Data Flow
params
Activity 4
“On Error” Activity 1 …
params

15. Gain insights from ADLA pipeline & recurring jobs
New Pipeline Jobs View
New
Superset of original jobs view
Adds grouping of jobs by pipelines & recurrences
Jobs and consumption trends per pipeline
Quickly identify pipelines and jobs to troubleshoot
Quickly compare failed jobs with “last known good” instance
Manage pipeline cost, improve efficiency and predict future cost
How to use
Create ADF v2 pipelines containing ADLA U-SQL activities
Pipelines and Recurrences automatically appear in ADLA portal
Submit and monitor pipeline/recurring jobs using Azure PowerShell, ADLA SDK and REST APIs

16. Data Movement Scalable Simple Access all your data per job elasticity
Up to 1 GB/s
Simple
Visually author or via code (Python, .Net, etc)
Serverless, no infrastructure to manage
Access all your data
30+ connectors provided and growing (cloud, on premises, SaaS)
Data Movement as a Service: 17 points of presence world wide
Self-hostable Integration Runtime for hybrid movement
Data Movement

17. Demo Copy data from Amazon S3 to Azure Blob
Process/transform data using Spark, on error
Demo
Ingest and Transform Pipeline
Populate DW Pipeline
Send
error
On Demand
Trigger
Copy
Spark
Copy S3
Blob
Azure DW
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Azure Data Factory has simplified the integration of data from multiple hybrid sources at scale to drive meaningful insights for our customers

19. “Azure Data Factory has enabled us to integrate heterogenous data from multiple hospitals allowing us to leverage big data and analytics offerings in Azure at scale to drive better health outcomes for our customers” David B. McAuley, CTO, Lumedx

20. SSIS in ADFv2 Lift existing SSIS projects to the cloud
6/19/ :44 AM
SSIS in ADFv2
Lift existing SSIS projects to the cloud
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Integration Runtime for SSIS
Managed Cloud Environment
Pick # nodes & node size
Resizable
SQL Standard Edition, Enterprise coming soon
Compatible
Same SSIS runtime across Windows, Linux, Azure Cloud
SSIS + SQL Server
SQL Managed instance + SSIS (in ADFv2)
Access on premises data via VNet
Get Started
Hourly pricing (no SQL Server license required)
Use existing license (coming soon)
Integration Runtime for SSIS
Azure
Integration
Runtime
SSIS Project

22. Demo Integration Runtime for SSIS in ADFv2 6/19/2018 10:44 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Looking Forward Visual Tools
6/19/ :44 AM
Looking Forward Visual Tools
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Roadmap 2017 2018 SDKs (Python, .Net, Powershell)
New control-flow/data-flow based app model
Familiar to existing SSIS, etc users.
Serverless, pay per use
SSIS runtime (in ADFv2)
For lifting existing on prem SSIS solutions to cloud
“Use existing license” (coming soon)
Data movement
More connectors, scale out, highly available self-hosted integration runtime
Visual experience (coming soon)
For control flow, data movement & monitoring
2018
Visual experiences (data transform)
Data movement (connectivity, scale, … )
Further SSIS integration
Data Discovery

27. Please evaluate this session
Tech Ready 15
6/19/2018
Please evaluate this session
From your
Please expand notes window at bottom of slide and read. Then Delete this text box.
PC or tablet: visit MyIgnite
Phone: download and use the Microsoft Ignite mobile app
Your input is important!
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Q & A mike.flasko@microsoft.com @mflasko

29. 6/19/ :44 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.