I know what you did last transaction eric@berkflow.com (561) 914-2097 Eric Berkowitz SQL Saturday Orlando 10/6/18

World’s Greatest Credentials – Eric Berkowitz First developed with SQL Server on OS/2 LAN Manager and Sybase on Netware Worked for, and certified, by Sybase, the original developer of SQL Server, primarily in client- server systems Have used SQL Server in every consulting and permanent position since 1989 – over 35 shops Have been VP level corporate IT, owned firm in NYC, served as manager, mentor, architect, developer – sometimes all at once! Now doing independent consulting primarily in South Florida and also for clients in Massachusetts and California Using SharePoint, SSIS, SSRS and workflow and forms projects Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Eric Berkowitz - eric@berkflow.com - (561) 914-2017 What we’ll be covering Methods you can use as an administrator or developer to enable SQL querying of previous values of columns in rows that have been changed, with zero to minimal developer involvement Emphasis on specific tables, with changes retained for usually some known period of time Uses include tracking tables similar to “slowly changing dimensions” and data repairs made in transactional systems Other approaches Security-centric: see SQL Server Audit (doesn’t track before and after on Update) Service Broker: cannot directly query the changes Comprehensive third party products see Apex SQL Audit Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Eric Berkowitz - eric@berkflow.com - (561) 914-2017 Methods we’ll review Trigger with history table Change Data Capture Not Change Tracking, it is lightweight and can be useful but doesn’t track changed data Temporal (System-versioned) tables Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Trigger with History Table Demonstration Review the trigger Note how the Operation is detected and the Before and After are noted You can choose which columns to include initially and as schema changes, and which users’ changes to record, which both help minimize inefficiencies in performance and storage The history table can be stored anywhere and horizontally partitioned Manage retention with a SQL Agent job that deletes rows from history table with an operation date more than X days ago No latency on queries The trigger is included in the transaction so there is a performance impact If you insert into an audit table with an identity column upon insert this will change @@Identity Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Trigger with History Table Query like any table Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Eric Berkowitz - eric@berkflow.com - (561) 914-2017 Change Data Capture Available in Standard Edition of SQL Server as of 2016 SP1; previously Enterprise Edition Enable CDC in database and then use System Stored Procedures sp_cdc… to create: CDC tables and functions in System Tables A log-reading agent (similar to replication) so there is no latency or direct performance impact Avoids performance impact during writes Introduces small latency – cannot query the changes immediately An agent to manage retention of data You can specify which columns to capture changes for (using @captured_column_list argument of sys.sp_cdc_enable_table) Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Change Data Capture (cont’d) Considerations If you need to introduce a column in the underlying table or a separate one to record the user changing the data this will require a trigger anyway The capture table retains its shape during a schema change Dropped columns info replaced with null New columns are not added – this is possibly most significant weakness Workaround: create a second capture with the new data structure and drop the old capture when the previous data is no longer needed (there is a max of two capture instances per table) Database owner must be sysadmin SQL Server Agent must be running Incompatible with databases supporting memory-optimized tables (like the demo database) Failover considerations in High Availability When replication is not used the sp_cdc_add_job must be run on the (now) primary to capture changes When replication is used this isn’t necessary because the log reader handles the change captures Either way the cleanup job needs to be set up on the (new) primary Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Change Data Capture Configuration Using System Stored Procedures -- Enable CDC in the database EXEC sys.sp_cdc_enable_db GO -- Enable the table for CDC EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'People', @role_name = NULL, @supports_net_changes = 1 EXEC [sys].[sp_cdc_change_job] @job_type = N'cleanup', @retention = 259200 -- 180 days Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Change Data Capture (cont’d) Functions are created when you enable a table for data capture The query functions that are created are determined by supports_net_changes 0: functions to query for all changes. Returns multiple rows, i.e. one net change row for each source row changed within the LSN range. 1: additionally creates functions to query for net changes. Returns one row for the content at end of LSN range. For Net changes, must have a primary key or you must specify the index_name parameter Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Eric Berkowitz - eric@berkflow.com - (561) 914-2017 Change Data Capture Query using functions created when the table was enabled Typically specify the begin and end dates to search, ‘all update old’ (to include previous values) or ‘all’ Operation: 1=Delete 2=Insert 3=Update (before) 4=Update (after) Eric Berkowitz - eric@berkflow.com - (561) 914-2017

“Siri, what is a temporal table?” “I found this information about Tempura on the Web” Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Temporal (System Versioned) Tables More native implementation of the Trigger/History table technique Two Datetime2 columns added to the table track the “currency” (my term) of the row over time – bounds of from date it was current (row start) and through date (row end), one for Period Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Temporal (System Versioned) Tables The terminology for the tables is the “CURRENT” and “HISTORY” tables The history table is transparent with a preassigned name unless you want to specify its storage or create indexes. Limitations – these are only a few! No truncating No INSTEAD OF triggers on table History table must be in same database Limited use of replication Current table cannot include FILETABLE or FILESTREAM Same schema as current table Recommendations Traditional trigger method allows you to omit varchar(max), varbinary(max) text, image so if you don’t anticipate needing to track changes to values in these columns you can save a ton of storage by using trigger. In a new database, consider vertically partitioning tables that you are planning to make system-versioned so these columns aren’t written to history table Existing INSERTS, SELECT INTO, and even SELECTS that don’t specify column names may have trouble with the generated columns, but you can hide them to avoid this. If they’re not hidden, use DEFAULT as the inserted value System versioning must be off to drop a temporal table; use SSMS to generate the drop script since it will do this and also drop the transparent history table Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Querying Temporal Table Using System Stored Procedures Add FOR SYSTEM_TIME clause to SELECT statement Use one of these sub-clauses in your query AS OF datetime FROM datetime TO datetime (Row Start < TO value) and (Row End > FROM value) BETWEEN datetime AND datetime (Row Start <= second value) and (Row End > First value) CONTAINED IN (datetime, datetime) (Row Start >= first value) and (Row End <= Second Value) LIVE DEMOS! Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Make Existing table a Temporal Table Use ALTER to add the ROW START, ROW END and PERIOD columns Make them hidden to avoid problems with existing application code In either case, use DEFAULT as the inserted value in an INSERT ALTER table to set SYSTEM_VERSIONING ON and optionally speciafy the history table Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Other Temporal Table features Built-in support for stretching and partitioning to Azure Retention: You can create a custom stored procedure to manage retention but this also involves turning off SYSTEM_VERSIONING on the table (do it inside a transaction) You can set the retention period to be managed automatically at time of creation or through ALTER HISTORY_RETENTION_PERIOD = 6 MONTHS Takes effect immediately in queries but data is not immediately deleted, and you can change the setting back to a longer duration If the primary use case is data audit (i.e. searching for historical changes for a single row from the current table), then a good choice is to create rowstore history table with a clustered index (direct from Microsoft) Eric Berkowitz - eric@berkflow.com - (561) 914-2017

Eric Berkowitz - eric@berkflow.com - (561) 914-2017 Recommendations Temporal tables is great choice for new development When database is being designed from ground-up with it in mind; and, When it is intended to be used as a permanent feature But you have the greatest chance of unintentionally breaking production You can replicate changes from older databases to a SQL 2017 server Triggers are a good choice when you are very confident they won’t break production or cause performance issues, or need specific columns, or want to omit users, column changes or applications CDC is good as a low-risk transparent solution for short- and long-term auditing and change tracking Smallest chance of harming production except in replicated environments In all cases, be wary of security issues on the history tables Eric Berkowitz - eric@berkflow.com - (561) 914-2017