1. SSIS Custom Pipeline Component A step-by-step guide
Arne Bartels
SSIS Custom Pipeline Component A step-by-step guide

2. BIG Thanks to SQLSat Denmark sponsors

3. Agenda
Who is Vestas Whoami Motivation Goal Loong Demo Outlook

4. Who is Vestas, what are we doing

5. Vestas is currently the world biggest wind turbine manufacturer
Surveillance of plants with turbines
Supercomputer Mindstorm
16848 cores 41600GB Memory TFlop/s
currently Rank 440
Datawarehouse/Data Mart server
Six servers Dev/Test/Prod ETL(”DSA”) DM(”DPA”)
SQL 2016
Prod and Test 40core 3.5GHz 1.5TB Memory
Dev 20core 3GHz 256GB Memory
Storage raw 160TB, all SSD, 800TB effective capacity
Hardware located in Copenhagen DK
Core Team of a handfull SQL/SSIS-Developers from five countries mainly in Aarhus DK
(No column name) (No column name)

6. Whoami
Arne Bartels
Working for Vestas Wind Systems A/S in Aarhus DK. Since approx as Lead Software Development Engineer
Software developer in wind industry since 2001
Physicist by education, programmer by profession
Worked in C/C++,Pascal, Linux scripts/PHP, MySQL…
Recent: T-SQL, SSIS, C#
LinkedIn

7. Motivation Inspired by real world challenge to import files into a DW
Make that automizable/Biml-able
Show you to overcome quite some of the tripwires
Motivate to play around
Show off what I can do

8. Goal
Build a reusable component to scan folders for file patterns within date ranges and batch sizes. Sort by write date and pathname. Optionally delete old data.

9. What is a Custom Pipeline Component?
ClassLibrary (DLL) implementing the necessary interfaces for SSIS.
To use it in development of SSIS, it has to be copied at the ”right” place (DEV).
To be executed, it has to be added to the assembly (DEV and PROD), which requires admin rights.
All the above has to be correct otherwise the SSIS-toolbox won’t show the component. There is no debug option; you have to try, and try and…

10. What is needed? Visual Studio e.g. 2015, Community edtion is enough
+SSDT for SSIS-package editing
Optionally Icon editor e.g. Gimp
Ideas and patience

11. Documentation Microsoft: Developing a Custom Data Flow Component
How to add an icon to your component
SQLSaturday DK 2015 Wolfgang Strasser

12. Demo: how to get started
Have a SSIS test project
Start new project of type ClassLibrary
Sign assembly
Add Reference to Microsoft.SQLServer.PipelineHost.dll
Inherit class from PipelineComponent
Add DtsPipelineComponentAttribute
For convenience setup build events and debug

13. The bare minimum

14. Add Icon
[DtsPipelineComponent(DisplayName = "Folder Source" , Description = "Display folder information as data stream" ,IconResource = "Folder_Source.Resources.FolderSource.ico" ,ComponentType = ComponentType.SourceAdapter )] To find the right string, ildasm is helpful

15. Override design time methods …
#region design time methods public override void ProvideComponentProperties() { base.ProvideComponentProperties(); RemoveAllInputsOutputsAndCustomProperties(); //todo: setup CustomProperties an Output } public override DTSValidationStatus Validate() return base.Validate(); #endregion design time methods

16. … and run time methods
#region run time methods public override void PreExecute() { //do work } public override void PrimeOutput(int outputs, int[] outputIDs, PipelineBuffer[] buffers) //do output //public override void ProcessInput(int inputID, PipelineBuffer buffer) //{ // no Input to process => nothing to do //} #endregion run time methods

17. Add columns…
#region Output Declaration IDTSOutput100 output = ComponentMetaData.OutputCollection.New(); output.Name = "Output"; output.SynchronousInputID = 0; output.DeleteOutputOnPathDetached = true; output.IsSorted = true; IDTSOutputColumn100 outputColumn = null; outputColumn = output.OutputColumnCollection.New(); outputColumn.Name = "FullName"; outputColumn.SetDataTypeProperties(DataType.DT_WSTR, 260, 0, 0, 0); outputColumn.SortKeyPosition = 2; // ... #endregion Output Declaration

18. …and custom properties
#region Properties Declaration IDTSCustomProperty100 property = null; property = ComponentMetaData.CustomPropertyCollection.New(); property.Name = "FolderPatterns"; property.Description = "Files matching will be displayed."; property.ExpressionType = DTSCustomPropertyExpressionType.CPET_NOTIFY; property.TypeConverter = typeof(String).AssemblyQualifiedName; property.Value // ... #endregion Properties Declaration

19. Debugging Design time methods: straightforward, hit F5
Run time methods: Attach to process
Hit F5, stop SSIS project at breakpoint, note processid
Go back ClassLibrary debug and ”Attach to process” DtsDebugHost.exe to the ”other” ProcessID. If needed use ”taskkill /f /im dtsdebughost.exe” in adm command prompt.

20. Or use ”inline debugging”
#region Members private bool fireagain = false; #endregion Members … public override void PreExecute() { ComponentMetaData.FireInformation(0, "PreExecute", "start", null, 0, ref fireagain); foreach (IDTSCustomProperty100 property in ComponentMetaData.CustomPropertyCollection) ComponentMetaData.FireInformation(0,"Property: "+ property.Name, property.Value, null, 0, ref fireagain); foreach (IDTSOutput100 output in ComponentMetaData.OutputCollection) ComponentMetaData.FireInformation(0, "Output: "+output.Name, "has:"+output.OutputColumnCollection.Count.ToString()+" columns", null, 0, ref fireagain); foreach(IDTSOutputColumn100 outputColumn in output.OutputColumnCollection) ComponentMetaData.FireInformation(0, "Output column: " + outputColumn.Name, outputColumn.DataType.ToString(), null, 0, ref fireagain); } ComponentMetaData.FireInformation(0, "PreExecute", "end", null, 0, ref fireagain);

21. Read and store column mapping
Store all column indices in Key=>Value Array:
private SortedList outputColumnIndices = null; …
#region Output column index mapping
outputColumnIndices = new SortedList();
try
{//remember the indices of the actual remaining columns for later use
IDTSOutput100 output = ComponentMetaData.OutputCollection["Output"];
if(output!=null)
foreach (IDTSOutputColumn100 outputColumn in output.OutputColumnCollection)
outputColumnIndices.Add(outputColumn.Name, BufferManager.FindColumnByLineageID(output.Buffer, outputColumn.LineageID)); }
catch (Exception e) {
#endregion Output column index mapping

22. Read and store custom properties
private List<String> folderPatterns = new List<String>(); … #region Properties mapping try { foreach (IDTSCustomProperty100 property in ComponentMetaData.CustomPropertyCollection) switch (property.Name) case "FolderPatterns": folderPatterns.AddRange(((String)property.Value).Split('|')); break; } catch (Exception e) #endregion Properties mapping

23. Parse folder structure recursively
Split folder strings of the form C:\folder1\folder2\*\* recursively and check for each recusing step matching subfolders until leading string empty.
E.g.
C:\folder1\folder2\*\*, \.
folder1\folder2\*\*, C:\
folder2\*\*, C:\folder1\
*\*, C:\folder1\folder2\
*, C:\folder1\folder2\folder3a
C:\folder1\folder2\folder3a\folder4aa
C:\folder1\folder2\folder3a\folder4ab
*, C:\folder1\folder2\folder3b
C:\folder1\folder2\folder3a\folder4ba
C:\folder1\folder2\folder3a\folder4bb
Then check for matching files after coming back from recursion.
Sort after date

24. Outlook Add customized userinterface (Windows Forms)
Add connection manager (again with Windows Forms)
Add other source systems ftp/sftp/…

25. Wanted!
Data Engineering & Analytics is looking for consultants in the greater DK area, preferably close to Aarhus DK. To work with SQL/SSIS Data warehousing and supercomputing. Contact:

26. BIG Thanks to SQLSat Denmark sponsors