Overcoming SSIS Deployment and Configuration Challenges Erik Veerman Mentor, Solid Quality Mentors SQL Server MVP

//ATLANTAMDF 2 SQL Server MVP (Most Valuable Professional) Mentor with Solid Quality Mentors ~15 years in the industry, 10 years SQL Server BI experience (v1 SQL BI, beta SQL 7)… Telco, Manufacturing, Finance, Insurance, Retail, Real Estate… MS Worldwide BI Solution of the Year SQL Server Magazine Innovator Cup Microsoft Project REAL ETL Architect Author: SQL Server 2008 Integration Services Professional MS Press SQL Server BI Training Kit (70-445) ( in progress!) SQL Server 2005 Integration Services Expert SQL Server 2005 Integration Services Professional Speaker BIO – Erik Veerman

//ATLANTAMDF Industry-Leading Technical Experts, Authors, Speakers Solid Quality™ Mentors are more than 100 of the world’s top technical experts who specialize in assuring client success using integrated Microsoft technologies. “Catch the Solid Quality™ Advantage” 3

//ATLANTAMDF 4 Who has had challenges with SSIS deployment or configurations? Has anyone punched a wall or other visible frustrations? Has anyone actually had to go to anger management? Let’s be honest…

//ATLANTAMDF  Baby Steps: The Books-Online Basics – Security: Encryption and Connections – Configurations and Expressions – Deployment  The Problem: Bob  Anger Management: The Solutions Agenda

//ATLANTAMDF BOL Basics: SSIS Package Security  Security is comprised of several layers to support both SQL and File System based scenarios – Packages can be encrypted – Packages can be digitally signed – Packages can be stored in SQL DB and protected with roles – Packages can be stored in the File System and protected with File System Security Package Definition: Storage: SSIS Package File (.DTSX): Encrypted, Digitally Signed File System: File System Security MSDB: MSDB SQL Server Roles

//ATLANTAMDF BOL Basics: SSIS Package Security  Internal Package Security – Package encryption – Used to encrypting connection passwords or the entire package – Package Password – Optional password applied to the package to un-encrypt any encrypted contents  External Package Security – MSDB database roles for packages deployed to SQL Server. Reader and Writer roles included for execution and reading privileges – File system security for packages deployed to server file systems or file shares

//ATLANTAMDF BOL Basics: SSIS Package Encryption Levels  Don’t Save Sensitive  Encrypt Sensitive with User Key  Encrypt Sensitive with Password  Encrypt All with User Key  Encrypt All with Password  Server Storage

//ATLANTAMDF Package Encryption and MSDB demo … demo …

//ATLANTAMDF BOL Basics: Securing Data and Connections  Data Security – Data is not stored as part of a package  Connection Schema Security – Connection schema information will be stored in packages depending on the uses (Data Flow source/destinations and Execute SQL) – Lock down package access when appropriate  File system rights  Package encryption/password  Package roles and access

//ATLANTAMDF Package Connections and Encryption demo … demo … 12

//ATLANTAMDF SSIS Package Connection 1 = XYZ Variable 1 = XYZ SSIS Package Connection 1 = XYZ Variable 1 = XYZ SSIS Package Connection 1 = ABC Variable 1 = ABC BOL Basics: SSIS Configurations  Update package properties at execution time based on external setting – Variable properties – Connection Properties – Package properties – Task and Container Properties  Data Flow Transformation properties are not configurable except at the Task level Configuration Connection 1 = ABC Variable 1 = ABC

//ATLANTAMDF SSIS Configuration Types  XML File Configuration – Direct path or… – Environment Variable driven  SQL Server Configuration – Table Driven – Filter Column grouping configurations  Registry Configuration  Parent Package Variable Configuration – Inherit variables from master packages  Environment Variable Configuration

//ATLANTAMDF Package Configurations demo … demo …

//ATLANTAMDF BOL Basics: SSIS Property Expressions  Runtime updates of Control Flow components – Package level properties – Task and Container properties – Connections, too!  Expressions assignable through property windows and task editors  SSIS expression language based SSIS Package Variable1 = C:\FileA Connection1 = Step 1: Run Data Flow Access Connection1 Extract from C:\FileA Step 2: Update Variable Step 3: Run Data Flow Access Connection1 Extract from C:\FileB C:\FileA C:\FileB

//ATLANTAMDF SSIS Property Expressions  Common Property Expression usage – Dynamic SQL for non-parameterizable statements – Connections, Checkpoints, Disable flag – Loop Container enumeration functionality  Variable values can be updated by expressions through the EvaluateAsExpression and Expression property of the variable

//ATLANTAMDF Property Expressions demo … demo …

//ATLANTAMDF BOL Basics: Deployment  Deployment destinations – File System (file share or local drive) – MSDB Database (sysdtspackages90 or sysssispackages)  Deployment options – Manual windows command line script (Xcopy or Robocopy) – DTUtil command line script (file system or MSDB) – Deployment Wizard and Installer

//ATLANTAMDF BOL Basics: Deployment  File System  MSDB Development Test or Production SSIS Package BIDS Project Folder File System or Share File Copy, DTUtil, Installer Set Development Test or Production SSIS Package BIDS Project Folder MSDB Database SSMS, DTUtil, Installer Set

//ATLANTAMDF BOL Basics: Deployment Wizard  Designer can build a deployment set which includes a project's packages, configurations files and an installer file  You move the installer file set to another server/environment and execute to install packages to SQL or file system  Installer can "copy" misc files but will not install them, such as custom components  Deployment overwrites existing packages

//ATLANTAMDF Package Deployment demo … demo …

//ATLANTAMDF  Baby Steps: The Books-Online Basics – Security: Encryption and Connections – Configurations and Expressions – Deployment  The Problem: Bob  Anger Management: The Solutions Agenda

24 You want to live a normal IT life but the problem follows you around Just when you think you’ve solved the problem, there it is again The Problem: You’re the expert, but the problem will not go away Just knowing the baby steps doesn’t get you far enough

//ATLANTAMDF SSIS Deployment Challenges  Encryption causes your package to fail  Connections and other properties aren’t updated  Passwords are not available or too easily accessible  Shared configurations require properties and connections exist  SQL Configurations point to Dev database Encryption Failure: Server 1Server 2 SSIS Package Connection A SSIS Packages MSDB or File System SSIS Package Connection A Package Deployed Encryption With UserKey Encryption With UserKey Package Fails! Can not decrypt UserKey (New Machine) Package Fails! Can not decrypt UserKey (New Machine)

//ATLANTAMDF SSIS Deployment Challenges  Connections and other properties aren’t updated Server 1Server 2 SSIS Package Dev Connection A Dev Connection B Dev Connection A MSDB or File System Package Error! Connections Point to Wrong Databases/Files Package Error! Connections Point to Wrong Databases/Files SSIS Package Dev Connection A Dev Connection B Package Deployed Dev Connection B Prod Connection A Prod Connection B SSIS Packages

//ATLANTAMDF SSIS Deployment Challenges  SQL Configurations point to Dev database (or configuration database not accessible!) Server 1Server 2 SSIS Package Dev Connection A Dev Connection B Dev Config DB Dev Connection A MSDB or File System Config DB Connection Not Updated! SSIS Package Dev Connection A Dev Connection B Dev Config DB Package Deployed Dev Connection B Prod Connection A Prod Connection B Dev Config DB Prod Config DB 2 SSIS Packages

//ATLANTAMDF Pulling Your Hair Out! demo … demo …

//ATLANTAMDF  Baby Steps: The Books-Online Basics – Security: Encryption and Connections – Configurations and Expressions – Deployment  The Problem: Bob  Anger Management: The Solutions Agenda

Solution 1 Solution 1: Tie Bob to the Boat Pretend the problem is not there Try and man-handle the problem Make the problem worse

Solution 2 Solution 2: Blow up the house Give up Find a punching bag Declare to Bill Gates that you are not a PC anymore

//ATLANTAMDF Dealing with Encryption  Situation: All connections allow Windows Authentication  Use Windows Authentication only for all connections  Turn off encryption (Set encryption to DontSaveSensitive) Server 1Server 2 SSIS Package Connection A/B/C (Windows Auth) SSIS Packages MSDB or File System SSIS Package Connection A/B/C (Windows Auth) Package Deployed Encryption: DontSave- Sensitive Encryption: DontSave- Sensitive

//ATLANTAMDF Dealing with Encryption  Situation: One or more connections require password  Option 1: EncryptSensitiveWithPassword Pass in the package Password during execution  Option 2: Set encryption to DontSaveSensitive Use a Configuration to update the password  Option 3: Deploy to MSDB, set encryption to ServerStorage Option 1: Server 1Server 2 SSIS Package Connection A, password = ABC SSIS Packages MSDB or File System SSIS Package Connection A, password = ABC Package Deployed Encryption: EncryptSensitive- WithPassword Encryption: EncryptSensitive- WithPassword Package Executed with DTExec and the /P switch

//ATLANTAMDF Dealing with Encryption  Option 2: Set encryption to DontSaveSensitive Use a Configuration to update the password Server 1Server 2 SSIS Package Connection A, password = [Blank] SSIS Packages MSDB or File System SSIS Package Connection A, password = [Blank] Package Deployed Encryption: DontSave- Sensitive Encryption: DontSave- Sensitive Config File password = ABC Config File password = ABC

//ATLANTAMDF Solving the Encryption Problem demo … demo …

//ATLANTAMDF Dealing with Connections  Option 1: Use a File Configuration (direct or indirect path)  Option 2: Use a File and SQL Configuration  Option 3: Use Property Expressions and Variables Server 1Server 2 SSIS Package Dev Connection A Dev Connection B Dev Connection A MSDB or File System SSIS Package Dev Connection A Dev Connection B Package Deployed Dev Connection B Prod Connection A Prod Connection B Dev Config File Prod Config File 2 SSIS Packages Option 1

//ATLANTAMDF Dealing with Connections  Option 2: Use a File and SQL Configuration Server 1Server 2 SSIS Package Dev Connection A Dev Connection B Dev Config DB Dev Connection A MSDB or File System SSIS Package Dev Connection A Dev Connection B Dev Config DB Package Deployed Dev Connection B Prod Connection A Prod Connection B Dev Config File Prod Config File SSIS Packages SSIS Dev Config DB 2 SSIS Prod Config DB 2 3

//ATLANTAMDF Demo, Two-Pass Configuration Approach demo … demo …

//ATLANTAMDF Dealing with Connections  Option 3: Use a Package Variable and Expression Server 1Server 2 SSIS Package Connection A Variable Connection B Variable Connection A (Expression) Connection B (Expression) Dev Connection A MSDB or File System Dev Connection B Dev Config File 1 3 SSIS Packages 2 4 SSIS Package Connection A Variable Connection B Variable Connection A (Expression) Connection B (Expression) Dev Connection A Dev Connection B Dev Config File 1 2 Package Deployed Makes sharing configurations easier! 3 4

//ATLANTAMDF Demo, Variable Based Configuration Approach demo … demo …

//ATLANTAMDF BIDS Helper  CodePlex.com – BIDS Helper  Tools – Deployment SSIS Packages tool – dtsConfig File Formatter – Expression and Configuration Highlighter – Expression List – Non-default properties list – Variables Window Extensions – FixRelativePaths – …others… 41

//ATLANTAMDF 42 Consider storing your packages on SQL Server Consider storing your packages in the file system Will you have hundreds of SSIS packages Yes Do you need Pkg execution or read privileges for a team Package Storage Decision Flow Do you Need a centralized package backup and Security model Are you using SQL Server RDBMS? Are you the DBA or do you have DBA privileges? Can your Source Control tool handle deployment No Yes No Yes No Will you Have a lot of Master- Child packages? Yes No Other factors: cluster, multiple servers, etc No Yes No Yes

//ATLANTAMDF Use DTUtil and Command Line Scripting Do you have a set of packages to be deployed per deployment Package Deployment Decision Flow Do you Need to automate deployment Are you storing your packages in SQL Server? Can your Source Control handle deployment Do your Packages for deployment always come from the same BIDS project? No Yes No Yes Manually deploy package Files Use the Source Control Deployment Tools for file system deployment Use the SSIS Deployment Wizard Yes No

//ATLANTAMDF Package Encryption Decision Flow (version 1) Are you Breathing? Yes EncryptSensitive- WithUserKey EncryptAll- WithUserKey DontSave- Sensitive EncryptSensitive- WithPassword EncryptAll- WithPassword ServerStorage

//ATLANTAMDF Package Encryption Decision Flow (version 2) Are you Obsessive Compulsive (like Bob)? Yes EncryptSensitive- WithUserKey EncryptAll- WithUserKey DontSave- Sensitive EncryptSensitive- WithPassword EncryptAll- WithPassword No ServerStorage

//ATLANTAMDF Package Encryption Decision Flow (version 3) Are you the only person that will ever edit/design the package Yes EncryptSensitive- WithUserKey EncryptAll- WithUserKey DontSave- Sensitive EncryptSensitive- WithPassword EncryptAll- WithPassword No Will you always use the same Server? Do you need a class in anger management No Yes ServerStorage Yes

//ATLANTAMDF Package Encryption Decision Flow (version 4) Are you deploying to SQL Server… And are deploying using the SSIS Deployment Tool… And don’t use configurations… And your passwords don’t change… Yes No ServerStorage See other options…

//ATLANTAMDF Use an indirect (Env Var to reference) file configuration Do you or your org have a standard or pref. for storing settings in files or DBs? Package Configuration Decision Flow Do you have passwords or sensitive properties in the package? Will the file name or Path change between servers? Yes Tables No Use a direct file configuration Files Is your server locked down (file and DB)? Is it easier in your org to secure files or tables? No Yes Use SQL Config with a File Config as a pointer Tables Yes Files Other factor: Sharing configs

Thank you