Cool Tricks to Pull from Your SSIS Hat Julie Smith

Cool Tricks to Pull from Your SSIS Hat Julie Smith Business Intelligence Consultant at Key2 Consulting in Atlanta, GA With Audrey Hammonds the 11 Years Experience as ETL Developer

Digging into more advanced SSIS Techniques—Case Studies Step through some real life scenarios and learn some ways to use: Variables The SSIS Expression Language For Each Loops ADO Objects (Object Variables) Building Blocks for Frameworks!

In the Beginning… There was a connection manager There was a data flow task They were hardcoded (their value was Constant) It was good……

Until realized your hand was getting tired from all the right clicking as your requirements got more complicated. –“Tired of having to run your package then change all the configurations? –“Tired of doing it again next time?” –“Try Some Cool SSIS Tricks”

What is a Variable (programming)? From Wikipedia: In computer programming, a variable is a symbolic name given to some known or unknown quantity or information, for the purpose of allowing the name to be used independently of the information it represents. A variable name in computer source code is usually associated with a data storage location and thus also its contents, and these may change during the course of program execution.computer programmingsource codedata storage location

In SSIS, How do you change the value of a Variable? With an Execute SQL Task With an SSIS Expression Through configuration of For Each Loop With a Script Task or Component *not an exhaustive list.

Where we’ll be in next hour… Windows Explorer SQL Server Management Studio BIDS –What we’ll be doing (SSIS Functions include) : Data Flow (Table to Text File) Validate XML Task If we have time--File System Tasks

Before we Get to the Demos: Take a tour to familiarize ourselves with some key BIDS windows. Variables Window Properties and the Expression box. (Go to Tour)

Scenario 1 Business Problem: You have a regular job which exports rows from the GeospatialWidgets table into a text file. Often, the result set is empty, and you wish you knew how to prevent the creation of the empty file. We’re going to use a variable, the SSIS expression language, a precedence constraint, dynamically affecting properties.

Solution 1 Step 1: Create Variable called “rowcount” Step 2: Use Execute SQL task to capture the rowcount of table inside that variable. Do this 2 ways: 1. ResultSet method/ OLE DB connection. 2. Output Parameter of Sproc ADO.Net connection

Solution 1 continued Step 3--Use SSIS Expression Language to Either 1. Prevent the data flow’s execution using Precedence Constraints 2. Disable the data flow with the Disable Property Both ways mean --when “rowcount” = 0, then data flow does not execute.

Demo 1

Gotchas! Result Name has to be Changed (named). Ordinal number starting with 0 (zero)—or parameter name. This is not intuitive. Caution with Disable Property Once switched, it cannot be unswitched during execution of the package. Variable value of User:rowcount has to be non-zero value at start of execution, or it will disable itself.

Scenario 2 Business Problem: You have one file directory which will contain many XML files. Files must be validated and then processed. You will not process all the files in the directory with one run of the job. So you need to be able to “cherry pick” the files out of the directory based on other criteria.

Solution 2: Use a SQL table as a “Master List”. Table has “processed” flag (bit) for filtering. Use a Variable of ADO Object data type to hold the filtered “queue” of files to process from the Master List. Use For Each Loop in SSIS with the ADO Object as the Enumerator. With each loop, each row gets processed. Each row contains the fully qualified filename of one XML file

ADO Objects SSIS IDFileNameProcessed 1C:\file1.xml1 2C:\file2.xml0 3C:\File3.xml0 IDFileName 2C:\file2.xml 3C:\File3.xml The table: MyFileQueueTable The Query: Select ID, FileName from MyFileQueueTable where Processed = 0 (Creates a Runtime queue in SSIS) I’m a table. I live in SQL Server I’m an ADO Object, don’t I look like a table?

Demo 2

Gotchas! Value of file has to be real. It’s Confusing-- see next slide. Error Handling has to be implemented otherwise package fails on XML Failure. XML Failure in this case should not halt the package.

Journey of value of XML File. 1.XML validation task uses *”filename.xml” File Connection as Source. 2.*”filename.xml” has its Connection String dynamically populated by the value of the vFileName variable with an Expression in its Properties

Journey of value of XML File continued. 3.The vFileName has its value iteratively changed as the For Each Loop processes the ADO Object named “ADOValidate”. 4.The “ADOValidate” Object of the For Each loop gets its values from the resultset of the Execute SQL Task

Scenario 3: Business Problem A: In previous scenario, we have possibility of thousands of clients importing 2 types of files into the same database. Want to segregate their archived files by type of file and client. Don’t want to manually create these directories. Create task that will take the Client and filetype and create the properly named and located directory. **Credit to Wrox book, Veerman, Knight et al.

Scenario 3 continued Business Problem B: We also want to create auditing data. We write back to the Queue table with information on the execution Succeed? Fail? If it failed, why?

Solution 3 Use Expression Language to Create File Directories with File Task. (A) Move XML files from ToBeProcessed to Error/Client if necessary. (A) Use Execute SQL Task with variables to dynamically update the Queue table with relevant processing information. (B)

Demo 3

Gotchas! “\” is an Escape in SSIS Expression Language. In Production, Write Permissions.

Summary— Variables – give us a placeholder to work magic in. –For Each Loops and ADO Object Variables are VERY USEFUL Expression Language – is the magic

Cool Tricks to Pull from Your SSIS Hat Julie Smith Heartfelt Thanks to: –Audrey Hammonds –Kenneth Wright –SQL Rally Organizers –SQL Rally Sponsors –My Key2 Colleagues –Aaron Nelson and Rob Volk –Monty Python (Eric Idle, Graham Chapman, Terry Jones, Terry Gilliam, John Cleese, Michael Palin)

Cool Tricks to Pull from Your SSIS Hat Julie Smith Thanks for coming! Check out Also blog at:

Please Complete the Evaluation Form Pick up your evaluation form: In each presentation room Drop off your completed form Near the exit of each presentation room At the registration area Session Title 30 Sponsored by Dell

THANK YOU! For attending this session and PASS SQLRally Orlando, Florida Session Code | Session Title 31 Sponsored by Dell