ADF & SSIS: New Capabilities for Data Integration in the Cloud 6/19/2018 10:43 AM BRK2254 ADF & SSIS: New Capabilities for Data Integration in the Cloud Mike Flasko Principal Group Program Manager © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Agenda Current State Feedback & Target Scenarios What’s New Roadmap 6/19/2018 10:44 AM Agenda Current State Feedback & Target Scenarios What’s New Roadmap Q & A © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Current State: Data Integration in Azure Azure Data Factory (ADF) Provides orchestration, data movement and monitoring services Orchestration model: time series processing Hybrid Data movement as a Service w/ many connectors Programmatic authoring, visual monitoring (.NET, Powershell) SSIS: server software for ETL Focused on ETL to/from SQL Server Scale up data transformation engine Visual authoring of control and data flow Rich ecosystem (BIML, Task Libraries, etc)

Feedback: Do any of these sound familiar? I need to create on-demand and/or event-triggered pipelines I need to create a delta-processing pipeline I need rich orchestration constructs to model my unique requirements (e.g. facilitate efficient restatements) I need to reliably work with all my data across cloud, on prem, SaaS apps, etc Data movement at scale Data pipelines spanning services, servers, cloud/onprem, etc ISVs: Need full programmatic access, using the languages and runtimes we are comfortable with My data integration team are not developers, I need visual tools Productivity of ETL is key, it is still 70%+ of overall solution time Scale to 1000’s pipelines How do I leverage my existing SSIS investments in the cloud?

Consumption Friendly Data Modern Data Warehouse Extract & Load Prepare Transform/Analyze Extract & Load Data Source 1 Data Source 2 DW & DM BI Tools Staging Prepared Data Consumption Friendly Data “Data Lake” Data Source N DS Tools

Consumption Friendly Data Modern Data Warehouse Extract & Load Prepare Transform/Analyze Extract & Load Data Source 1 Data Source 2 DW & DM BI Tools Staging Prepared Data Consumption Friendly Data “Data Lake” Data Source N DS Tools Data-driven SaaS Application App Storage SaaS App Browser/Device

Consumption Friendly Data Modern Data Warehouse Extract & Load Prepare Transform/Analyze Extract & Load Data Source 1 Data Source 2 DW & DM BI Tools Staging Prepared Data Consumption Friendly Data “Data Lake” Data Source N DS Tools Data-driven SaaS Application App Storage SaaS App Browser/Device Lift my existing SSIS packages to the cloud Data Source 1 ETL: SSIS SQL Server ETL: SSIS SQL Server Data Source N

Managed Data Integration Service ADF v2 Public Preview New Pipeline Model Rich pipeline orchestration Triggers – ondemand, schedule, event Data Movement as a Service Cloud, Hybrid 30 connectors provided SSIS Package Execution In a managed cloud environment Use familiar tools, SSMS & SSDT Author & Monitor Programmability (Python, .NET, Powershell, etc) Visual Tools (coming soon) Data Factory Managed Data Integration Service

On Prem Apps & Data Cloud Svcs, Apps & Data

Azure Data Factory v2 Service Command and Control Data Data Factory A data integration account Location of orchestration, service metadata UX & SDK Authoring | Monitoring/Mgmt Azure Data Factory v2 Service Scheduling | Orchestration | Monitoring Integration Runtime (IR) ADF’s execution engine Three core capabilities: data movement pipeline activity execution SSIS package execution Self Hosted Integration Runtime Azure Integration Runtime On Prem Apps & Data Cloud Svcs, Apps & Data

Azure Data Factory v2 Service Command and Control Data Data Factory A data integration account. Location of orchestration, service metadata UX & SDK Authoring | Monitoring/Mgmt Azure Data Factory v2 Service Scheduling | Orchestration | Monitoring Integration Runtime (IR) ADF’s execution engine Three core capabilities: data movement pipeline activity execution SSIS package execution Pipeline SSIS Package Self Hosted Integration Runtime Azure Integration Runtime On Prem Apps & Data Cloud Svcs, Apps & Data

Building Data Pipelines 6/19/2018 10:44 AM Building Data Pipelines Modern DW Data Driven SaaS Apps © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

ADFv2 Pipelines Trigger Activity 1 Activity 2 “On Error” Activity 1 My Pipeline 1 My Pipeline 2 For Each… Trigger Event Wall Clock On Demand params params Activity 3 Activity 1 Activity 2 params Activity 4 “On Error” Activity 1 … params

ADFv2 Pipelines Trigger Activity 1 Data Flow “On Error” Activity 1 My Pipeline 1 My Pipeline 2 For Each… Trigger Event Wall Clock On Demand params params Activity 3 Activity 1 Data Flow params Activity 4 “On Error” Activity 1 … params

Gain insights from ADLA pipeline & recurring jobs New Pipeline Jobs View New Superset of original jobs view Adds grouping of jobs by pipelines & recurrences Jobs and consumption trends per pipeline Quickly identify pipelines and jobs to troubleshoot Quickly compare failed jobs with “last known good” instance Manage pipeline cost, improve efficiency and predict future cost How to use Create ADF v2 pipelines containing ADLA U-SQL activities Pipelines and Recurrences automatically appear in ADLA portal Submit and monitor pipeline/recurring jobs using Azure PowerShell, ADLA SDK and REST APIs

Data Movement Scalable Simple Access all your data per job elasticity Up to 1 GB/s Simple Visually author or via code (Python, .Net, etc) Serverless, no infrastructure to manage Access all your data 30+ connectors provided and growing (cloud, on premises, SaaS) Data Movement as a Service: 17 points of presence world wide Self-hostable Integration Runtime for hybrid movement Data Movement

Demo Copy data from Amazon S3 to Azure Blob Process/transform data using Spark, email on error Demo Ingest and Transform Pipeline Populate DW Pipeline Send Email error On Demand Trigger Copy Spark Copy S3 Blob Azure DW © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

  Azure Data Factory has simplified the integration of data from multiple hybrid sources at scale to drive meaningful insights for our customers

  “Azure Data Factory has enabled us to integrate heterogenous data from multiple hospitals allowing us to leverage big data and analytics offerings in Azure at scale to drive better health outcomes for our customers” - David B. McAuley, CTO, Lumedx

SSIS in ADFv2 Lift existing SSIS projects to the cloud 6/19/2018 10:44 AM SSIS in ADFv2 Lift existing SSIS projects to the cloud © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Integration Runtime for SSIS Managed Cloud Environment Pick # nodes & node size Resizable SQL Standard Edition, Enterprise coming soon Compatible Same SSIS runtime across Windows, Linux, Azure Cloud SSIS + SQL Server SQL Managed instance + SSIS (in ADFv2) Access on premises data via VNet Get Started Hourly pricing (no SQL Server license required) Use existing license (coming soon) Integration Runtime for SSIS Azure Integration Runtime SSIS Project

Demo Integration Runtime for SSIS in ADFv2 6/19/2018 10:44 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Looking Forward Visual Tools 6/19/2018 10:44 AM Looking Forward Visual Tools © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Roadmap 2017 2018 SDKs (Python, .Net, Powershell) New control-flow/data-flow based app model Familiar to existing SSIS, etc users. Serverless, pay per use SSIS runtime (in ADFv2) For lifting existing on prem SSIS solutions to cloud “Use existing license” (coming soon) Data movement More connectors, scale out, highly available self-hosted integration runtime Visual experience (coming soon) For control flow, data movement & monitoring 2018 Visual experiences (data transform) Data movement (connectivity, scale, … ) Further SSIS integration Data Discovery

Please evaluate this session Tech Ready 15 6/19/2018 Please evaluate this session From your Please expand notes window at bottom of slide and read. Then Delete this text box. PC or tablet: visit MyIgnite https://myignite.microsoft.com/evaluations Phone: download and use the Microsoft Ignite mobile app https://aka.ms/ignite.mobileapp Your input is important! © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Q & A mike.flasko@microsoft.com @mflasko

6/19/2018 10:44 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.