Presentation is loading. Please wait.

Presentation is loading. Please wait.

BIML & EzAPI 2013-09-05. BIML and EzAPI Two approaches to creating SSIS packages programmatically Daniel Otykier //

Similar presentations

Presentation on theme: "BIML & EzAPI 2013-09-05. BIML and EzAPI Two approaches to creating SSIS packages programmatically Daniel Otykier //"— Presentation transcript:


2 BIML and EzAPI Two approaches to creating SSIS packages programmatically Daniel Otykier //

3 Agenda About me Why SSIS programmatically? Prerequisites BIML –Introduction –Samples –References EzAPI –Introduction –Samples –References Summary and discussion –Strength and weaknesses –Comparing BIML and EzAPI Feel free to ask questions along the way!

4 About me Daniel Otykier 28 years old, married, no kids (yet ) Cand. IT. 5 years of experience with SQL Server and Microsoft BI 8+ years of experience with C# and.NET BI consultant and developer at Jyllands-Posten since August 2008 Senior Business Intelligence Consultant at Kapacity since February 2013

5 Why work with SSIS programmatically? Eliminate repetitive and time-consuming labor –1:1 data extracts –Controller packages –Dimension Load packages Create metadata-driven SSIS solutions –Connections, source tables, columns, transformations = metadata –No manual editing SSIS packages when things change –Improve manageability of larger solutions Increase productivity –Reuse existing work –Use templates …and for the code-loving programmers among us: Work with structured code rather than a clumsy GUI

6 Why work with SSIS programmatically?

7 Prerequisites? Developer –Solid SSIS experience Software –SQL Server 2008R2 or 2012 –BIDS or SSDT –For BIML: BIDS Helper - –For EzAPI: Visual Studio edition with C# or VB.NET Project Type EzAPI.dll -

8 BIML Business Intelligence Markup Language

9 BIML introduction Developed and supported by Varigence XML-based Abstract description of BI solution ASP.NET-style scripting (BIMLScript) Free version with BIDS Helper –Generate SSIS packages –Integrates nicely in SSIS projects Full version with Mist™ IDE (Varigence) –Generate SQL tables, SSAS dimensions, cubes, etc.

10 Using BIML Make sure BIDS Helper is installed! –Create.biml files –Check for errors –Execute BIML Script = “Generate SSIS packages”

11 Visual Studio hints biml.xsd for syntax highlighting / IntelliSense Disable xml formatting on paste:

12 BIML syntax Creating a package that contains a single data flow task:

13 BIML XML tags Many BIML XML tags, corresponding to the various SSIS tasks, components, etc. Examples: …and many, MANY more!

14 Inline C# or VB.NET scripting with –Same as T4 templates Example: C# variables " ConstraintMode="Linear"/> BIMLScript basics

15 Example: C# conditionals BIMLScript basics

16 Example: C# loops " ConstraintMode="Linear"/> When running this BIML, it is expanded into… BIMLScript basics

17 All inline C# code is gone! …and then SSIS packages are generated. BIMLScript loop expansion

18 <# var metadataConnection = "Provider=SQLNCLI11;Server=localhost;Initial Catalog... var metadataSql = "SELECT PackageName FROM Packages"; var metadataTable = ExternalDataAccess.GetDataTable(metadataConnection, metadataSql); foreach (DataRow row in metadataTable.Rows) { #> " ConstraintMode="Linear"> Using metadata Repeated for every row in table “Packages”

19 Real-world sample Generate all extract packages from metadata! Metadata table containing: –Connection Manager name (also used as Schema name for extract tables) –Source table name or SQL SELECT-statement –Destination table name –Optional pre-execute and post-execute SQL statements (for dropping and creating indices, etc.) BIML script generates: –Extract package for each row in metadata table –Controller package, that executes every extract package

20 General usage of metadata with BIML Metadata source options: –INFORMATION_SCHEMA.TABLES –Custom metadata table (previous slide) –Custom XML or text file metadata BIML automatically maps dataflow columns if –Column names match in source and destination –Column mappings are explicitly specified in BIML What happens when metadata is changed? –Just execute BIML script again –Don’t edit generated SSIS-packages manually

21 More BIML BIML supports most SSIS features: –Variables –Expressions –Configurations –Events

22 More BIML BIML supports new SSIS 2012 features: –Project Connection Managers: –Package Parameters: MyValue –Use project/package parameters just like

23 BIML references BIML has an active user community:

24 EzAPI SSIS API wrapper written in C#

25 EzAPI introduction Developed by the Microsoft SSIS team Written in C#, source code available Wraps low-level DTS libraries Use with any.NET compatible language (C#, VB.NET, Visual C++, etc.) Maximum flexibility Everything that can be done in BIDS, can be done with EzAPI “Scripttime” metadata access

26 Terminology Designtime “Scripttime” Runtime SSDT / BIDSBIML / EzAPI Package creation (metadata available) Package execution “Designtime” Script / code writing

27 “Scripttime” in EzAPI When EzAPI code is executed Equivalent to designtime in SSDT / BIDS Complete access to SSIS metadata Utilization: –Create SQL tables on the fly –Perform operations depending on column data types, f.x: Character conversion String trimming –Inspect metadata while debugging code

28 EzAPI classes Out-of-the-box Ez*-classes, corresponding to SSIS objects: Top level: EzProject EzPackage Connection mgrs.: Ez***CM EzSqlOleDbCM EzOracleOleDbCM EzDb2OleDbCM EzFlatFileCM EzExcelCM Control flow: EzSequence EzForLoop EzForEachLoop Tasks: EzExecSqlTask EzExecPackage EzFileSystemTask EzDataFlow Components: Ez***Source Ez***Destination EzSqlDestination (fast load) EzDerivedColumn EzDataConvert EzMulticast EzUnionAll EzMerge EzLookup EzScript *** = OleDb / AdoNet …but feel free to create your own!

29 EzAPI syntax (C# edition) var myContainer = new EzSequence(myPackage) { Name = “Step 1" }; Object reference EzAPI class (Sequence Container) Parent object reference Object properties Read as: “Create a new Sequence Container inside myPackage. Set the containers name to ‘Step 1’. Use the myContainer variable as a reference for the new Sequence Container.

30 EzAPI syntax (c# edition) // Create package: var myPackage = new EzPackage() { Name = "BasicEzApiPackage" }; // Create OLE DB connection manager inside package: var myConMgr = new EzOleDbConnectionManager(myPackage) { Name = "DW Meta", ConnectionString = "Provider=sqlncli11;Data Source=localhost;Initial Catalog=MSBIP_EzAP... }; // Create Execute SQL task: var estTest = new EzExecSqlTask(myPackage) { Name = "EST Test", Connection = myConMgr, SqlStatementSource = "SELECT 'test' AS [Dummy]" }; // Save package: myPackage.SaveToFile("BasicEzApiPackage.dtsx");

31 Structuring EzAPI code Use C# language features to structure code: // Set up project: var project = new EzProject() { Name = "SSIS_EzAPI" }; // Set up project connection managers: var axCM = Generator.GetAXConnectionManager(project); var dwCM = Generator.GetDWConnectionManager(project); // Add Extract packages for AX: project.AddPackage(Generator.GenerateFullExtractPackage(axCM, "CUSTTABLE", dwCM, "AX_CUSTOMER")); project.AddPackage(Generator.GenerateFullExtractPackage(axCM, "PRODUCT", dwCM, "AX_PRODUCT")); project.AddPackage(Generator.GenerateFullExtractPackage(axCM, "SALESORDER", dwCM, "AX_SALESORDER")); project.AddPackage(Generator.GenerateIncrementalExtractPackage(axCM, "INVENTTRANS", dwCM, "AX_INVENT... // Add Load Dimension packages: project.AddPackage(Generator.GenerateLoadDimensionPackage(dwCM, "Customer")); project.AddPackage(Generator.GenerateLoadDimensionPackage(dwCM, "Product")); project.AddPackage(Generator.GenerateLoadDimensionPackage(dwCM, "Calendar")); project.SaveAs(OutputPath + "CodeStructureProject.ispac"); Note the.ispac project file type (not.dtproj)

32 Accessing “scripttime” metadata After calling.AttachTo() on a Data Flow component, column metadata is available: var derCmp = new EzDerivedColumn(dataFlowTask) { Name = "DER Transform" }; derCmp.AttachTo(srcCmp); var inputColumns = derCmp.Meta.InputCollection[0].InputColumnCollection; foreach (var col in inputColumns) { if(col.DataType == DataType.DT_WSTR) derCmp.Expression[col.Name] = string.Format("RTRIM({0})", col.Name); }

33 Extending EzAPI Create task and component C# classes –Wrap existing SSIS objects (FtpTask, WebTask, etc.) –Wrap 3rd party SSIS objects (KimballSCD, etc.) Extend EzAPI with helper classes –EzAPIExtensions (available for download) Create SQL tables on the fly Various EzAPI improvements More to come… Create reusable templates

34 EzAPI metadata demo Demo time!

35 More EzAPI Load existing SSIS projects and packages Batch changes Reverse engineering

36 EzAPI references Get EzAPI from Codeplex: SSIS-team blog: ezapi-alternative-package-creation-api.aspx ezapi-alternative-package-creation-api.aspx Other blogs:

37 Summary and discussion BIML and EzAPI

38 BIML vs. EzAPI BIML EzAPI Pros No licensing (included in BIDShelper) Intuitive for experienced SSIS users Many tutorials and samples online Integrates nicely in SSIS projects Goes beyond SSIS Cons Limited to standard functionality / not extensible BIMLScript XML may grow cumbersome and difficult to read and maintain Pros Very flexible and extensible Access metadata at scripttime Make changes to existing packages C# and object-oriented programming Cons Requires Visual Studio license C# and object-oriented programming Few tutorials online

39 Learning curves

40 Thanks for listening! Slides, samples and tutorials coming soon on our blog. Stay tuned! Feedback and questions:

Download ppt "BIML & EzAPI 2013-09-05. BIML and EzAPI Two approaches to creating SSIS packages programmatically Daniel Otykier //"

Similar presentations

Ads by Google