Presentation is loading. Please wait.

Presentation is loading. Please wait.

2013-09-05 BIML & EzAPI.

Similar presentations


Presentation on theme: "2013-09-05 BIML & EzAPI."— Presentation transcript:

1 BIML & EzAPI

2 BIML and EzAPI Two approaches to creating SSIS packages programmatically Daniel Otykier //

3 Feel free to ask questions along the way!
Agenda About me Why SSIS programmatically? Prerequisites BIML Introduction Samples References EzAPI Summary and discussion Strength and weaknesses Comparing BIML and EzAPI Feel free to ask questions along the way!

4 About me Daniel Otykier do@kapacity.dk
28 years old, married, no kids (yet ) Cand. IT. 5 years of experience with SQL Server and Microsoft BI 8+ years of experience with C# and .NET BI consultant and developer at Jyllands-Posten since August 2008 Senior Business Intelligence Consultant at Kapacity since February 2013

5 Why work with SSIS programmatically?
Eliminate repetitive and time-consuming labor 1:1 data extracts Controller packages Dimension Load packages Create metadata-driven SSIS solutions Connections, source tables, columns, transformations = metadata No manual editing SSIS packages when things change Improve manageability of larger solutions Increase productivity Reuse existing work Use templates …and for the code-loving programmers among us: Work with structured code rather than a clumsy GUI SCREENSHOT på næste slide

6 Why work with SSIS programmatically?

7 Prerequisites? Developer Software Solid SSIS experience
SQL Server 2008R2 or 2012 BIDS or SSDT For BIML: BIDS Helper - For EzAPI: Visual Studio edition with C# or VB.NET Project Type EzAPI.dll -

8 Business Intelligence Markup Language
BIML

9 BIML introduction Developed and supported by Varigence XML-based
Abstract description of BI solution ASP.NET-style scripting (BIMLScript) Free version with BIDS Helper Generate SSIS packages Integrates nicely in SSIS projects Full version with Mist™ IDE (Varigence) Generate SQL tables, SSAS dimensions, cubes, etc.

10 Using BIML Make sure BIDS Helper is installed! Create .biml files
Check for errors Execute BIML Script = “Generate SSIS packages” DEMO Åbn 1.1-BIMLIntroduction.sln Opret BIML-fil. Åbn BIML-fil. Vis syntax highlighting / biml.xsd skema Check for fejl Kør pakke (ingenting sker) Indsæt følgende: <Packages><Package Name="PackageCreatedFromBIML" ConstraintMode="Linear"></Package></Packages> Kør pakke (opretter tom pakke)

11 Visual Studio hints biml.xsd for syntax highlighting / IntelliSense
Disable xml formatting on paste:

12 BIML syntax Creating a package that contains a single data flow task:
<Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="MyTestPackage" ConstraintMode="Linear"> <Tasks> <Dataflow Name="My Data Flow"> <!-- ...dataflow components here... --> </Dataflow> </Tasks> </Package> </Packages> </Biml>

13 BIML XML tags Many BIML XML tags, corresponding to the various SSIS tasks, components, etc. Examples: <Packages> <Package> <Tasks> <ExecuteSQL> <ExecutePackage> <FileSystem> <Ftp> <SendMail> <ForEachFileLoop> <Script> <Container> <DataFlow> <Transformations> <OleDbSource> <ExcelSource> <DerivedColumns> <Sort> <Merge> <DataConversion> <AdoNetDestination> <Connections> <Variables> <PackageConfigurations> …and many, MANY more!

14 BIMLScript basics Inline C# or VB.NET scripting with <# and #>
Same as T4 templates Example: C# variables template language="C#" hostspecific="true"#> <# var packageName = "TestPackage"; #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="<#=packageName#>" ConstraintMode="Linear"/> </Packages> </Biml> 1.2-BIMLScriptBasics.sln

15 BIMLScript basics Example: C# conditionals
template language="C#" hostspecific="true"#> <# var value = 1; #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# if (value == 1) { #> <Package Name="PackageA" ConstraintMode="Linear"/> <# } else { #> <Package Name="PackageB" ConstraintMode="Linear"/> <# } #> </Packages> </Biml> 1.2-BIMLScriptBasics.sln

16 BIMLScript basics Example: C# loops
template language="C#" hostspecific="true"#> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# for (var i = 1; i <= 5; i++) { #> <Package Name="Package <#=i#>" ConstraintMode="Linear"/> <# } #> </Packages> </Biml> When running this BIML, it is expanded into… 1.2-BIMLScriptBasics.sln

17 BIMLScript loop expansion
<Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <Package Name="Package 1" ConstraintMode="Linear"/> <Package Name="Package 2" ConstraintMode="Linear"/> <Package Name="Package 3" ConstraintMode="Linear"/> <Package Name="Package 4" ConstraintMode="Linear"/> <Package Name="Package 5" ConstraintMode="Linear"/> </Packages> </Biml> All inline C# code is gone! …and then SSIS packages are generated.

18 Using metadata Repeated for every row in table “Packages”
template language="C#" hostspecific="true" #> <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Packages> <# var metadataConnection = "Provider=SQLNCLI11;Server=localhost;Initial Catalog... var metadataSql = "SELECT PackageName FROM Packages"; var metadataTable = ExternalDataAccess.GetDataTable(metadataConnection, metadataSql); foreach (DataRow row in metadataTable.Rows) { #> <Package Name="<#=row["PackageName"]#>" ConstraintMode="Linear"> </Package> <# } #> </Packages> </Biml> 1.3-BIMLScriptMetadata.sln Repeated for every row in table “Packages”

19 Real-world sample Generate all extract packages from metadata!
Metadata table containing: Connection Manager name (also used as Schema name for extract tables) Source table name or SQL SELECT-statement Destination table name Optional pre-execute and post-execute SQL statements (for dropping and creating indices, etc.) BIML script generates: Extract package for each row in metadata table Controller package, that executes every extract package 1.4-BIMLRealWorldExtractSample - Bemærk muligheden for at droppe og oprette index: Prod.BillOfMaterials - Demonstrér tilføjelse af en tabel, Production.TransactionHistory: Script tabellen, tilføj DW_ExtractDate kolonne, og tilføj til ExtractTables tabel. Benyt script: Create table Prod.TransactionHistory.sql

20 General usage of metadata with BIML
Metadata source options: INFORMATION_SCHEMA.TABLES Custom metadata table (previous slide) Custom XML or text file metadata BIML automatically maps dataflow columns if Column names match in source and destination Column mappings are explicitly specified in BIML What happens when metadata is changed? Just execute BIML script again Don’t edit generated SSIS-packages manually

21 More BIML BIML supports most SSIS features: Variables Expressions
Configurations Events

22 More BIML BIML supports new SSIS 2012 features:
Project Connection Managers: <OleDbConnection Name="OLTP" ConnectionString=”...” CreateInProject="true"/> Package Parameters: <Parameter DataType="String" Name=”MyParam">MyValue</Parameters> Use project/package parameters just like variables: <Variable Name=”MyVariable" DataType="String” </Variable>

23 BIML references BIML has an active user community:
Varigence.com BIMLScript.com bidshelper.codeplex.com:

24 SSIS API wrapper written in C#
EzAPI

25 EzAPI introduction Developed by the Microsoft SSIS team
Written in C#, source code available Wraps low-level DTS libraries Use with any .NET compatible language (C#, VB.NET, Visual C++, etc.) Maximum flexibility Everything that can be done in BIDS, can be done with EzAPI “Scripttime” metadata access

26 Terminology “Designtime” Designtime “Scripttime” Runtime SSDT / BIDS
BIML / EzAPI “Designtime” Script / code writing Package creation (metadata available) Designtime “Scripttime” Runtime Package execution

27 “Scripttime” in EzAPI When EzAPI code is executed
Equivalent to designtime in SSDT / BIDS Complete access to SSIS metadata Utilization: Create SQL tables on the fly Perform operations depending on column data types, f.x: Character conversion String trimming Inspect metadata while debugging code

28 EzAPI classes Out-of-the-box Ez*-classes, corresponding to SSIS objects: Top level: Control flow: Components: EzProject EzSequence Ez***Source EzPackage EzForLoop Ez***Destination EzForEachLoop EzSqlDestination (fast load) Connection mgrs.: EzDerivedColumn Ez***CM Tasks: EzDataConvert EzSqlOleDbCM EzExecSqlTask EzMulticast EzOracleOleDbCM EzExecPackage EzUnionAll EzDb2OleDbCM EzFileSystemTask EzMerge EzFlatFileCM EzDataFlow EzLookup EzExcelCM EzScript …but feel free to create your own! *** = OleDb / AdoNet

29 EzAPI syntax (C# edition)
var myContainer = new EzSequence(myPackage) { Name = “Step 1" }; Parent object reference Object reference Object properties EzAPI class (Sequence Container) Read as: “Create a new Sequence Container inside myPackage. Set the containers name to ‘Step 1’. Use the myContainer variable as a reference for the new Sequence Container.

30 EzAPI syntax (c# edition)
// Create package: var myPackage = new EzPackage() { Name = "BasicEzApiPackage" }; // Create OLE DB connection manager inside package: var myConMgr = new EzOleDbConnectionManager(myPackage) { Name = "DW Meta", ConnectionString = "Provider=sqlncli11;Data Source=localhost;Initial Catalog=MSBIP_EzAP... }; // Create Execute SQL task: var estTest = new EzExecSqlTask(myPackage) Name = "EST Test", Connection = myConMgr, SqlStatementSource = "SELECT 'test' AS [Dummy]" // Save package: myPackage.SaveToFile("BasicEzApiPackage.dtsx"); Vis demo 2.1-EzAPI-Introduction

31 Structuring EzAPI code
Use C# language features to structure code: // Set up project: var project = new EzProject() { Name = "SSIS_EzAPI" }; // Set up project connection managers: var axCM = Generator.GetAXConnectionManager(project); var dwCM = Generator.GetDWConnectionManager(project); // Add Extract packages for AX: project.AddPackage(Generator.GenerateFullExtractPackage(axCM, "CUSTTABLE", dwCM, "AX_CUSTOMER")); project.AddPackage(Generator.GenerateFullExtractPackage(axCM, "PRODUCT", dwCM, "AX_PRODUCT")); project.AddPackage(Generator.GenerateFullExtractPackage(axCM, "SALESORDER", dwCM, "AX_SALESORDER")); project.AddPackage(Generator.GenerateIncrementalExtractPackage(axCM, "INVENTTRANS", dwCM, "AX_INVENT... // Add Load Dimension packages: project.AddPackage(Generator.GenerateLoadDimensionPackage(dwCM, "Customer")); project.AddPackage(Generator.GenerateLoadDimensionPackage(dwCM, "Product")); project.AddPackage(Generator.GenerateLoadDimensionPackage(dwCM, "Calendar")); project.SaveAs(OutputPath + "CodeStructureProject.ispac"); Note the .ispac project file type (not .dtproj) 2.2-EzAPI-Code-Structure

32 Accessing “scripttime” metadata
After calling .AttachTo() on a Data Flow component, column metadata is available: var derCmp = new EzDerivedColumn(dataFlowTask) { Name = "DER Transform" }; derCmp.AttachTo(srcCmp); var inputColumns = derCmp.Meta.InputCollection[0].InputColumnCollection; foreach (var col in inputColumns) { if(col.DataType == DataType.DT_WSTR) derCmp.Expression[col.Name] = string.Format("RTRIM({0})", col.Name); } 2.3-EzAPI-AccessMetadata

33 Extending EzAPI Create task and component C# classes
Wrap existing SSIS objects (FtpTask, WebTask, etc.) Wrap 3rd party SSIS objects (KimballSCD, etc.) Extend EzAPI with helper classes EzAPIExtensions (available for download) Create SQL tables on the fly Various EzAPI improvements More to come… Create reusable templates

34 EzAPI metadata demo Demo time!

35 More EzAPI Load existing SSIS projects and packages Batch changes
Reverse engineering

36 EzAPI references Get EzAPI from Codeplex: SSIS-team blog: Other blogs:

37 Summary and discussion
BIML and EzAPI Summary and discussion

38 BIML vs. EzAPI BIML EzAPI Pros Pros
No licensing (included in BIDShelper) Intuitive for experienced SSIS users Many tutorials and samples online Integrates nicely in SSIS projects Goes beyond SSIS Cons Limited to standard functionality / not extensible BIMLScript XML may grow cumbersome and difficult to read and maintain Pros Very flexible and extensible Access metadata at scripttime Make changes to existing packages C# and object-oriented programming Cons Requires Visual Studio license Few tutorials online

39 Learning curves Næste slide: THANKS FOR LISTENING

40 Thanks for listening! Slides, samples and tutorials coming soon on our blog. Stay tuned! Feedback and questions:


Download ppt "2013-09-05 BIML & EzAPI."

Similar presentations


Ads by Google