BIML: Step by Step Julie Smith

Julie Smith SQL Server MVP and Consultant at Innovative Architects in Atlanta, GA ½ of the DataChix at Datachix.com Twitter: @JulieChix Contributing Author:

About Julie SSIS Lover Not a developer No, you hang up

BIML: Intro and History Domain specific language. Specific to BIDS –Microsoft—codeplex. Once compiled, generates BI objects. Scott Currie. When he was at Msft. Started Varigence Began as Project VULCAN

Scott Currie and Varigence Steve and Scott

What is it? XML like, domain specific language. Emits Microsoft BI Objects Tables SSIS Packages Cubes

Quick Review of XML Other (child) elements eXtensible Markup Language Hallmark of XML is tags: <tag>stuff</tag> The content between and including a start and end tag is an element. Elements can contain any of the following: Other (child) elements Text ~ non quoted contents of an element Attributes ~  the value of a single property for an element, using a name/value pair with quotes around the value. *Self closing element syntax: <tag stuff/>

Here is a package with a dataflow: <Packages> <Package Name="MyPackage" ConstraintMode="Linear"> <Tasks> <Dataflow Name="My Data Flow"> </Dataflow> </Tasks> </Package> </Packages>

Here is what it emits:

How do I get it? Free version –comes with BIDS Helper from Codeplex Buy MIST from Varigence--Enterprise

Why Do I Need It? BIML Scripting – not such a win. BIML+ BIMLScript + System Info from SQL Server about your schema = shortcuts! = BAM! = #SQLWINNING

Examples My last project—400 source tables or files. Generally accepted best practice one package each. We used about four different patterns and same framework for each package. Guess how long it took and how many folks?

Estimating ETL My wiki wiki ways: estimating ETL development time http://tinyurl.com/EstimateETL Vincent McBurney Aug 7, 2006 Each source base 1 day. Then add for complexity Lookups, error handling, source complexity Multiply by experience level of your developers http://it.toolbox.com/wiki/index.php/DataStage:_estimate_how_long_it_will_take_to_build_a_DataStage_job

Good Fits for BIML Already Whiz Bang SSIS Developer Helps to know C# Have too much to do with resources available. Invest the time—it will pay off.

How to Make it work for you: Identify. Your patterns BIML-ize. Write in BIML Meta-tate. Choose your metadata Script-a-loop. Script and loop

How to Make it work for you: Identify your patterns in SSIS you need to automate For instance, you want to stage all tables from NW db to a staging db, with the addition of a date loaded column. Step One!

How to Make it work for you: Step Two! BIML-ize. Write one copy of the package with BIML using resources such as Varigence documentation.

How to Make it work for you: Identify what metadata to loop through. Choices : Native metadata (such as SQL System views, which have built in BIML Script methods) Custom metadata stores. Manual –in SQL, in Excel Koen Verbeeck  David Stein –Kimball spreadsheet Step Three!

How to Make it work for you: Apply BIML Scripting to your Biml prototype. Loop through your metadata of choice using Bimlscript code Step Four!

How to Make it work for you: Step Four! How to Make it work for you:

How to Make it work for you: Step Four! How to Make it work for you: <#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #> <# foreach (var table in tables) { #> <#=table.SchemaQualifiedName#>  -- Directive --Scripting --literals

Common syntax concepts: XML has elements and attributes An element is enclosed in a tag Let’s say we have element Packages <Packages> Content for packages </Packages> If it comes from a toolbox, or a toolbar it’s an element. If it’s a “property” of something from above, then it’s probably an attribute. http://it.toolbox.com/wiki/index.php/DataStage:_estimate_how_long_it_will_take_to_build_a_DataStage_job

Draw or Type? If I can DRAW it (it’s a shape in SSIS) then it’s going to be an Element. An element is enclosed in a tag <tag>Stuff</tag> Stuff can be more elements and or it can be attributes. Case sensitive. Attributes in quotes. <tag>Stuff</tag> We have element Packages <Packages> Content for packages </Packages> If it comes from a toolbox, or a toolbar it’s an element. If it’s a property of something from above, then it’s probably an attribute. http://it.toolbox.com/wiki/index.php/DataStage:_estimate_how_long_it_will_take_to_build_a_DataStage_job

In BIML <Biml xmlns="http://schemas.varigence.com/biml.xsd"> <Connections> <Connection Name="julieBiml" ConnectionString="Data Source=6wk9cs1-pc;Initial Catalog=julieBiml;Provider=SQLNCLI10.1;Integrated Security=SSPI;" /> <Connection Name="TheBimlMist" ConnectionString="Data Source=6wk9cs1-pc;Initial Catalog=TheBimlMist;Provider=SQLNCLI10.1;Integrated Security=SSPI;" /> </Connections> <Packages> <Package Name="TableCopy" ConstraintMode="Parallel" ProtectionLevel="DontSaveSensitive" > <Tasks> <Dataflow Name="DFT Table1"> <Transformations> <OleDbSource Name="table1" DefaultCodePage="1252" ConnectionName="TheBimlMist"> <ExternalTableInput Table="[dbo].[Name]" /> </OleDbSource> <OleDbDestination Name="Dest1" ConnectionName="julieBiml" > <InputPath OutputPathName="table1.Output" /> <ExternalTableOutput Table="[Dest1]" /> </OleDbDestination> </Transformations> </Dataflow> </Tasks> </Package> </Packages> </Biml> In BIML

Use the BimlScript Editor at BimlScript.com/develop

Patterns Out There Incremental Load. Uses Defined parts of schema. So FK is used bc it’s defined as BK Stairway to BIML Other option is define as Annotation in the table. Create and Load a staging environment. Scott Currie

References http://linchpinpeople.com/2013/06/using-biml-as-an-ssis-design-patterns-engine/ Video with Andy Leonard and Paul Waters http://varigence.com/Documentation/ http://varigence.com/Products/Biml/FAQ http://www.sqlservercentral.com/articles/BIML/101330/ Stairway to BIML BIMLScript.com http://bimlscript.com/Develop This is the editor, so you can see your script without VS messing it up.

http://www.varigence.com/documentation/biml/biml_Varigence.Languages.Biml.AstRootNode.html