1. New Features and Enhancements in SSIS 2016
RISSUG
OCTOBER 5, 2016
Matt Batalon MCITP/MCSA

2. About Me
Presently a Senior SQL Server Developer for healthcare company in RI
Using SQL Server since SQL Server 2000
SQL Server Certified 2008, 2012, 2014 (MCTS, MCITP, MCSA)
President of the Rhode Island SQL Server Users Group (Official PASS Chapter)
sqlsaturday.com (Providence December 10, 2016) at Bryant University
Precons: December 9th, Grant Fritchey & Aarron Bertrand

3. Agenda: New SSIS Features in 2016
Too many to go over, some of the best ones (my opinion)
AlwaysOn Support for SSIS catalog
Automatically Tuning Buffer Size
Incremental Package Deployment
Always Encrypted
New Roles for SSIS
Custom Logging
Control Flow parts
Azure Support in SSIS
SQL Server 2016 is a major upgrade. Too many features to go over.

4. Demo Environment All free! SQL Server 2016 Developer Edition
SQL Server 2016 Management Studio
Visual Studio Community 2015
SQL Server Data Tools 2016
AdventureWorksCTP32016 db
WorldWideImporters db
All free!

5. SQL Server Integration Services
Started out as DTS in SQL Server 2000
Became SSIS in 2005
Largest upgrade in 2012
Project Deployment Model
SSIDB Catalog
More SSIS 2016 Features
Some build upon features made available in 2012 and 2014

6. 2016: SSISDB AlwaysOn Support
Used for High Availability of your SSISDB Catalog DB
“Technically” not supported in SQL Server versions 2012/2014
Could be implemented with some work arounds.
Stop and think about how critical your SSIS deployed packages are. SSIS Catalog is extremely popular right now.
Might be a useful feature if properly implemented. ensuring high availability for your projects, packages, logs and environments.
Backup SSISDB:
Backup Master Encryption Key to a file
Backup DB
Generate a Login Script for a specific user for a cleanup job, startup scripts and maintenance jobs.
Restore SSISDB
Enable Clr
Create and Assymetric Key and login with an UNSAFE permission, in order to run clr stored procs the ssisdb needs.
Past versions:
Connect to every node and create the SSISDB catalog. 
We need to create the catalog even on secondary nodes to create the other server-level objects (cleanup jobs, keys, accounts etc) that are used by SSIS.
Delete the SSISDB databases on secondary nodes.
Create an availability group, specifying SSISDB as the user database Specify secondary replicas.
dm_hadr_availability_replica_states ars inner join sys.availability_groups ag

7. SSISDB with AlwaysOn 2012/2014
When a failover occurred, SSISDB would failover to the secondary node, which is now the primary
When a package was executed on the new primary node, typically would generate and error:
Error 15581: Please create a master key in the database or open the master key in the session before performing this operation
This occurs because of the way SSISDB encrypts sensitive information (such as connection string passwords, and certain parameters.
To resolve this, the database master key needs to be re-encrypted by the service master key after failover occurs.
To resolve it, you’d need to re-encrypt the database master key by running this T-SQL script.

8. To enable AlwaysOn on SSIS 2016
Install a Windows Failover cluster, install SQL Server 2016 with Integration Services on each node and enable the Always On feature on each instance
Create the SSIS catalog on the primary node
Add the SSISDB to the Availability Group.
Right-click the SSIS catalogs node and choose Enable AlwaysOn Support...

9. Enable AlwaysOn Support
Once you do this, on the Primary Node, you can right click, and Enable AlwaysOnSupport.

10. Connect to the replicas

11. 2016: AutoAdjustBufferSize Property
New property added to the data flow task in 2016
Auto Adjusts the buffer size
Eliminates some trial and error with DefaultMaxBufferRows and DefaultBufferSize
Default Values are usually too low, forced intervention
The data flow uses buffers to transfer and transform the data. Buffers are chunks of memory.
The size of the buffer is determined by two properties: DefaultBufferMaxRows - with a default of 10,000 rows - andDefaultBufferSize - with a default of 10MB.
Imagine you have to fill buckets of water and pass them along a line to put out a fire. If you have bigger buckets, you can transfer more water and the fire will be put out more quickly. But if your buckets are too big, it takes too long to fill them with water while the fire is raging on. So you have to find the optimal size of the buckets to balance the amount of water being passed and the time to fill those buckets. The same is true with data flow buffers: you have to find the optimal size of the buffers.

12. SSIS Data Type Conversions
Example Source Row
SSIS Data Type Conversions
= DT_STR = 10 Bytes
= DT_Date = 3 Bytes
= DT_STR = 20 Bytes
= DT_UI4 = 4 Bytes
= DT_CY = 8 Bytes
Total: 79 Bytes

13. A look inside the buffer…
EstimatedRowSize * DefaultMaxBufferRows = EstimatedBufferSize
79Bytes * 10,000 = 790KB
790KB < 10MB – Up the default MaxRowSize
Smaller the row size, the more rows fit into a buffer.
The Goal is the have as many rows possible inside a single buffer

14. sp_spaceused <table name>
bytes = 42,112*1024 = 43,112,688
row = 43,112,688/ 500,000 = 86 bytes per row

15. DefaultMaxBufferRows
DefaultBufferSize / rowsize
10MB * 1,024 * 1,024 / 86 = 121,927

16. Automatic Performance Tuning
All the MATH goes away!
AutoAdjustBufferSize will adjust the buffer size accordingly to find the best pipeline performance
Ignores DefaultMaxBufferSize, DefaultMaxRowSize
Turned off by default

17. 2016: Incremental Package Deployment
SSIS 2012 introduced the SSIS catalog Project Deployment Model
1 or many SSIS packages could be created as a SSDT project, and deployed to the SSIS catalog
project parameters, project connection managers for all packages in the project
Easily switch environments, connections, parameters, etc..

18. One glaring omission….
You were forced to choose “project” or “package” model deployment.
If you chose project model, force to deploy the entire project, not just one package
Versioning : project has 30 packages, and I need to only change 1 package
Work In Progress project package, bug fix in current project
SSRS has this for a while. Based upon projects and solutions. Right click deploy.

19. 2012/2014 SSIS Previous Versions

20. 2016 SSIS Latest Version

21. 2016: Always Encrypted Data Support in SSIS
Very popular feature in SQL Server 2016
Enhanced data security for data not just at rest but also data in “motion”
SSIS 2016 support for reading and writing data in an Always Encrypted Db
Requires an additional connection property and Column Master Key, Column Encryption Key, local certificates(keys) and an ADO.NET connection
column encryption setting=enabled
First of all what is Always encrypted. Always Encrypted is a new feature in SQL Server 2016, which encrypts the data both at rest *and* in motion (and keeps it encrypted in memory).
So this protects the data from rogue administrators,
backup thieves, and man-in-the-middle attacks. Unlike TDE, as well, Always Encrypted allows you to encrypt only
certain columns, rather than the entire database.
The client library ensures that plaintext is only revealed within the application or middle tier, and nowhere in between the application and the database.
Always Encrypted is the ADO.NET 4.6, so you will need to ensure .NET Framework 4.6 is installed on any machine that will run a client application that interfaces with Always Encrypted data.
Demo Setup.
a connection has been made with the additional connection property column encryption setting=enabled
the column master key created in the database is imported into the local certificate store (more detail on this later on)

22. Always Encrypted cont. OLEDB Connection Managers do not support
ADO.Net Connection Managers only supported
When first trying to read/write data from and Always Encrypted Database, SSIS may throw an error
Normally in SSIS you would use OLE DB connection managers to move data between sql tables.

23. 2016: New SSIS roles
Old: ssis_admin This role provides full administrative access to the SSIS Catalog database.
New ssis_alwayson_monitor role to support AlwaysOn
ssis_logreader SSIDB View access, can now view the reports in the SSIS catalog
Previously, if you were not a ssis_admin, the reports would show no data when viewed
In the SSIS catalog, a lot of information about the execution of the packages is automatically logged.
There are a set of built-in reports helping you to
find detailed logging results about those executions.
However, in earlier versions you needed to be an SSIS admin in order to view the data inside the
reports. If not, the reports would remain empty.

24. No Results

25. 2016: Custom Logging Levels
Determines how much information in logged to the SSIS catalog
Something between None and Verbose
Can be set as a server wide default
Can minimize the data written to the SSISDB when packages execute
Not to be confused with log providers inside a ssis package, like text files, sql tables, windows events…

26. Previous Levels Included
None
Basic (default- everything)
Performance
Verbose
RuntimeLineage (new)
Custom (new)
None - logging is turned off.
Basic - all events are logged, except custom and diagnostic events. The name is quite misleading, as a lot of information is logged. This is the default logging level of the SSIS catalog.
Performance - only performance statistics and the OnWarning and OnEvent are logged. This logging level might log less information than the basic logging level, but it depends on the number of data flows and their complexity. Data flow performance information of this logging level is shown in the catalog report Execution Performance.
Verbose - all events are logged. Only use this logging level for advanced troubleshooting or fine tuning.
RuntimeLineage: new logging level in Collects the data required to track lineage in the data flow i.e. where the data came from and where did the data go.
Catalog .event_message_context.

27. 2016: Control Flow Parts Formerly name Control Flow Templates
Can be viewed in the SSIS solution explorer
Can contain an encapsulated Data flow task
Re-Usable component in packages
Can be changed once, and no need to edit each package
Can’t be edited inside a package
Once created, appear in SSIS toolbox

28. Moving Data to and from the Cloud
AzCopy
Azure Data Factory
Azure Export/Import service
Powershell
FTP
SSIS – New features in SSIS 2016
AzCopy: A popular command-line utility designed for high-performance uploading, downloading, and copying data to and from Microsoft Azure Blob Storage.
Azure Import/Export Service: Used to transfer large amounts of file data to Azure Blob storage in situations where uploading over the network is prohibitively expensive or not feasible by sending one or more hard drives containing that data to an Azure data center.
Azure Data Factory (ADF): With the latest ADF service update and Data Management Gateway release, you can copy from on-premises file system and SQL Server to Azure Blob.  See Azure Data Factory Update – New Data Stores and Move data to and from Azure Blob using Azure Data Factory and Move data to and from SQL Server on-premises or on IaaS (Azure VM) using Azure Data Factory.  
Powershell: Work with Azure storage programmatically.

29. 2016: Azure Support SSIS Azure Pack – Control Flow and Data Flow
Sources
Destinations
Connections
HDInsight
Import Export Wizard (2016)

30. 2016: Azure Pack Technically available in SSIS 2014
Still very new, separate download/install
Custom SSIS components to move data to and from Azure and do much more
DataFlow Azure Blob source is new in 2016
Along with HDInsight Pig task
Azure Blob Source may have been added to the 2014 azure pack.
Not highlighted in books online for 2014 azure pack.

31. Azure Blob Storage is cheap

32. Control Flow Data Flow

33. Azure Storage Connection Manager

34. Azure Storage Account Information

35. Import/Export Wizard
Always has been a very useful tool to quickly import or export data as one off task.
Now in 2016, has Azure blob source and destination support

38. Many More Updates
OData Source V3 or V4 OData connection manager for ATOM and JSON data
Excel 2013 and Excel 2016 connections
HDFS File Connection Support
Target Specific Server Versions
ATOM = Web Feeds ( open data source).

39. Sources and References for this Presentation - Links
What's New in Integration Services
MSSQL Tips