Presentation on theme: "Deep Dive into ETL Implementation with SQL Server Integration Services"— Presentation transcript:
1 Deep Dive into ETL Implementation with SQL Server Integration Services Anton Rozenson
2 About this training event First event with core focus on Microsoft BI technologyHelp the community by sharing our learning and experience based on real world scenariosNetwork with peers and learn from you!These training events will be held every 2 months
3 Agenda Importance and complexity of ETL process ETL Architecture Changed Data Capture challenge and optionsData Flow design and performance considerationsSSIS project deploymentPackage execution optionsPerformance monitoring in SSIS catalog
4 Business Intelligence GNet Group OfferingsSharePointBusiness IntelligenceFrameworksCloud
5 Moving data“Data Warehouse is a system that extracts, cleans, conforms, and delivers source data into a dimensional data store and then supports and implements querying and analysis for the purpose of decision making” – Ralph Kimball, Joe Casertam. (2004) “The Data Warehouse ETL ToolkitEstimated 80% of work in building Data warehouse solution is related to ETL design and implementationData Warehouse is only as good as the data that it contains
6 Common ETL Architecture SourcesStagingEDWFlat FileData warehouseRDBMSChanged dataReference dataArtifacts and Error tablesConsumption readyDe-normalizedClean dataCloud
7 ELT Architecture Sources EDW Flat File Data warehouse RDBMS Changed dataReference dataArtifacts and Error tablesDe-normalizedRetains traceable business keyCloud
8 CDC Challenge What Changed? When changed ? Unable to modify source systems to include CDC attribution?Reliable timestamps?Need to reliably determine modified data for incremental loadingNew dataUpdated dataDeleted data
9 CDC options Source system can provide time stamps Reliability of process and completeness of information?Did all yesterday’s transactions committed?Does Source system include work flows that can cause late arrival of data records?How to determine deleted records?Comparing EDW data to Source system to determine differencesVery expensive query affecting Source systemDoes Source System incorporates archival process?Operational Data Store (ODS) can helpKeeping change historyClearly define state of data recordsStores metadata
10 CDC in SQL 2014In SQL Server, change data capture offers an effective solution to the challenge of efficiently performing incremental loads from source tables to data marts and data warehouses.Change Data Capture process stores transaction information from SQL log into system tables in CDC Schema.Data from CDC tables can be extracted by using table valued functions generated when CDC is enabled on the table.Key concepts:LSN – a binary timestamp representation used to restrict changed setAll Changes – changed set includes all DML transactionsNet Changes – changed set includes last DML transactions based on unique index.
12 Data Flow design challenges How should Data Flow perform?Consider following factors when designing an ETL SolutionSource System structure and ability to execute logic such as sorting and filtering.Parallel processing should be used with caution. Fastest way to load a table is Fast Load with table lock. This prevents loading data in parallel. Partition switching can be an answer.Requirements for data availability in EDWService Level Agreement (SLA)
13 Data Flow blocking tasks Data flow transformations in SSIS use memory/buffers in different ways. The way a transformation uses memory can dramatically impact the performance of your package. Transformation buffer usage can be classified into 3 categories: Non Blocking, Partially Blocking, and (Full) Blocking.Non Blocking transformations: Audit, Character Map, Conditional Split, Copy Column, Data Conversion, Derived Column, Import Column, Lookup, Multicast, Percentage sampling, Row count, Row sampling, Script componentPartially Blocking transformations: Data mining, Merge, Merge Join, Pivot/Unpivot, Term Extraction, Term Lookup, Union AllBlocking transformations: Aggregate, Fuzzy Grouping, Fuzzy Lookup, Sort
14 Work around blocking transformations in Data Flow It is not always possible to avoid using blocking or partially blocking transformations, but in some cases it is possible.For example Merge transformation requires data set to be sorted. While Sort transformation is expensive, in some cases sorting can be handled in the source query. Make sure to set IsSorted property of Data Source output to true and assign proper SortKeyPosition to output columns.Another example is Aggregate Transformation. Use Script transformation to perform aggregation of data and return result to a variable.
16 Project Deployment model Project deployment model allows following features:Parameters can be used in expressions or tasks. Parameters can reference an environment variable. Environment variable values are resolved at the time of package execution.An environment is a container of variables that can be referenced by Integration Services projects. Environments allow you to organize the values that you assign to a package. For example, you might have environments named "Dev", "test", and "Production".SSISDB catalog allows you to use folders to organize your projects and environments.Catalog stored procedures and views can be used to manage Integration Services objects in the catalog.
17 Package Execution An execution is an instance of a package execution. Package execution can be scheduled via SQL Agent job. SQL Agent provides an easy to use interface for mapping of Project parameters to environment variables.Packages can also be executed via Execute package tasks from another SSIS package. This allows creation of robust workflow incorporated into the Master Package.
18 Package ExecutionSSIS catalog allows package execution to be controlled programmatically from within T-SQL. A Number of stored procedures are provided to manage Package Execution.catalog.create_execution creates an instance of package execution and assigns Execution_ID.catalog.set_execution_parameter_value assigns parameters to the instance of package execution. Execution parameters control Logging Level, Dump settings, Synchronized execution option as well as ability to assign values to Project or Package scoped parameters.catalog.start_execution starts an instance of execution.
19 Execution MonitoringCatalog provides a set of standard reports allowing administrators easy access to execution performance and statistics.For details about executions, validations, messages that are logged during operations, and contextual information related to errors, query these views.executions list of Executions includes environmental dataexecution_data_statistics data flow performance informationexecution_parameter_values list of run time parametersevent_messages messages that were logged during executions
22 Additional ResourcesWhat's New in SQL Server SSIS Catalog Deployment of Projects and Packages Change Data Capture Change Data Capture (SSIS) CDC Flow Components Enable and Disable Change Data Capture (SQL Server) SQL Server OLE DB Deprecation and Integration Services oData Data source setup oData samples