1. SQL Server DML Change Capture An overview of several useful SQL Server data change capture technologies Matt Smith Software Architect, Enterprise Data Warehouse Otter Products, LLC Denver, CO SQL Server User Group. 20160317

2. Today’s Agenda Review three core SQL Server DML change capture technologies Custom T-SQL DML triggers SQL Server Change Tracking SQL Server Change Data Capture (CDC) Quick review of code exercises (time permitting)

3. Why capture data changes? Capturing changes is useful for solving common business problems Application auditing  “The sales team is deleting sales order lines instead of canceling them.“  “I think someone is deleting purchase order line information.” Business KPI’s & Customer metrics  Order Status change tracking (need to know the first date/time an order line went to status 77)  Did we hit our customer shipment targets? (monitor variance between promised and actual first shipment date) Data Warehousing/ETL Loading  Incremental data staging/data warehouse loading, intra-day reporting table updates  Infrastructure constraints (disk, network) Continuous Improvement Initiatives: cycle time  Monitoring process performance (ex. Key accounts return process, Customer warranty process)

4. Customer requirements: relevant to reporting & BI Our Goal: Clarify the problem(s) you are solving  Can you explain why we need capture data changes?  What decisions will you be making with this data?  Use Case: Can you explain how you are going to be using this data?  What are the metrics or KPI’s you are measuring?  How long do you want to retain this data? (negotiate)

5. Method #1: Custom DML Triggers Pros  Roll your own. Totally customizable and lots of options. It’s your logic  Compatible with any version of SQL Server (Express to Enterprise)  Track history of all or some changes (you choose)  No external processes/dependencies to worry about (SQL Agent Jobs, Capture Intervals, etc.)  Quick to implement. Code some triggers and go to lunch  Control sits firmly in dev team’s hands Cons  You must customize the source schema. You might not be able customize your databases as you may void your support contract  Trigger amnesia: Uh…oh yeah, I forgot about that trigger…mass data update took forever to run  High-transaction OLTP and triggers don’t get along: overhead, locking  Trigger proliferation and technical debt: Triggers are fun, we like to write triggers, now triggers are everywhere and it’s harder to change things Why DML Triggers? It’s because we are developers and that’s what we do! After Insert/Update/Delete triggers.

6. Method #2: SQL Server Change Tracking Pros  Definitively tells you that a column has changed over a specific range of DML statements  No customization of the source schema required (other than a PK).  Easy to determine rows that have changed, a little harder to determine cols that have changed  Included in SQL Server Standard edition (2008+)  A dev and ops collaboration for configuration and maintenance Cons  DML Change history not tracked. Change Tracking does not provide a sequential history of the changed values, just tells you that the value changed  Does not provide you with the time that the change occurred  Requires a Primary Key on the table for tracking changes. No heaps allowed  Requires significant coding and job scheduling to extract your changed data “Change tracking is a lightweight solution…”- msft

7. Method #3: SQL Server Change Data Capture (CDC) Pros  Does NOT require a primary key (supports heaps). exception: Net Changes TVF  No customization of the source schema required. CDC reads from the SQL Server transaction log and writes to CDC tables (uses sql server replication – run profiler and watch!)  Enable for all columns or for only a subset of columns  Useful functions and tracking tables based on time and Log Sequence number (LSN) ranges are built-in to help you extract change data  Basic SQL Server Data Tools SSIS Integration components for ETL processing are included (Attunity)  A dev and ops collaboration for configuration and maintenance Cons  SQL Server ENT edition ONLY (2008+)  You must be aware of transaction log management and HA/DR dependencies  SQL Agent required for capture and cleanup jobs, dbowner must be sa  CDC must be torn down and rebuilt in the event of transaction log database maintenance such as fixing VLF fragmentation or a Log Ship failover  Catch: DDL! For table DDL changes involving PK’s or unique indexes you may need to disable and re-enable CDC on the table. For Truncate (alter table DDL) - must disable CDC on the table first  Watch out for transaction log growth due to daily cleanup job (take smaller bites - schedule to run multiple times per day or limit with the threshold param)

8. Choosing a solution – quick review

9. Now that you have selected a technology, the real work begins. Working with the changed data is the majority of the effort and requires creative solutions.  Revisit your ETL pattern(s): Stage all changed data, apply changes as required to reporting tables and data warehouse staging tables  Change history tables grow quickly. Think about partitioning and trimming (deleting)  Consider data compression (PAGE) for change data destination tables, indexing and statistics updates for reporting tables (trace flag 2371)  Maintenance window changes: For Change Tracking and Change Data Capture, transaction log maintenance may lead to gaps in your change data  Recommendation: Avoid the Attunity SSIS CDC components. Use them to gain an understanding of how CDC works with SSIS for ETL, then create your own ETL solution Image: https://msdn.microsoft.com/en-us/library/cc645937.aspxhttps://msdn.microsoft.com/en-us/library/cc645937.aspx

10. Links and Contact This prezo along with demo code for Custom DML Triggers, Change Tracking and Change Data Capture is posted at www.mattfsmith.com |questions? matt@mattfsmith.comwww.mattfsmith.com matt@mattfsmith.com Scripts: Useful Resources & References:  Change Tracking (Mike Byrd - Solarwinds) : http://logicalread.solarwinds.com/sql-server-change- tracking-bulletproof-etl-p1-mb01http://logicalread.solarwinds.com/sql-server-change- tracking-bulletproof-etl-p1-mb01  Change Data Capture: https://msdn.microsoft.com/en-us/library/bb510744.aspxhttps://msdn.microsoft.com/en-us/library/bb510744.aspx  SSIS CDC Components (Matt Masson – SSIS Team): http://www.mattmasson.com/2011/12/cdc-in-ssis- for-sql-server-2012-2/http://www.mattmasson.com/2011/12/cdc-in-ssis- for-sql-server-2012-2/  Special thanks for the peer review from @CyberSnark@CyberSnark