1. I know what you did last transaction eric@berkflow.com (561) 914-2097
Eric Berkowitz
SQL Saturday Orlando
10/6/18

2. World’s Greatest Credentials – Eric Berkowitz
First developed with SQL Server on OS/2 LAN Manager and Sybase on Netware
Worked for, and certified, by Sybase, the original developer of SQL Server, primarily in client- server systems
Have used SQL Server in every consulting and permanent position since 1989 – over 35 shops
Have been VP level corporate IT, owned firm in NYC, served as manager, mentor, architect, developer – sometimes all at once!
Now doing independent consulting primarily in South Florida and also for clients in Massachusetts and California
Using SharePoint, SSIS, SSRS and workflow and forms projects
Eric Berkowitz - - (561)

3. Eric Berkowitz - eric@berkflow.com - (561) 914-2017
What we’ll be covering
Methods you can use as an administrator or developer to enable SQL querying of previous values of columns in rows that have been changed, with zero to minimal developer involvement
Emphasis on specific tables, with changes retained for usually some known period of time
Uses include tracking tables similar to “slowly changing dimensions” and data repairs made in transactional systems
Other approaches
Security-centric: see SQL Server Audit (doesn’t track before and after on Update)
Service Broker: cannot directly query the changes
Comprehensive third party products see Apex SQL Audit
Eric Berkowitz - - (561)

4. Eric Berkowitz - eric@berkflow.com - (561) 914-2017
Methods we’ll review
Trigger with history table
Change Data Capture
Not Change Tracking, it is lightweight and can be useful but doesn’t track changed data
Temporal (System-versioned) tables
Eric Berkowitz - - (561)

5. Trigger with History Table
Demonstration
Review the trigger
Note how the Operation is detected and the Before and After are noted
You can choose which columns to include initially and as schema changes, and which users’ changes to record, which both help minimize inefficiencies in performance and storage
The history table can be stored anywhere and horizontally partitioned
Manage retention with a SQL Agent job that deletes rows from history table with an operation date more than X days ago
No latency on queries
The trigger is included in the transaction so there is a performance impact
If you insert into an audit table with an identity column upon insert this will change
Eric Berkowitz - - (561)

6. Trigger with History Table
Query like any table
Eric Berkowitz - - (561)

7. Eric Berkowitz - eric@berkflow.com - (561) 914-2017
Change Data Capture
Available in Standard Edition of SQL Server as of 2016 SP1; previously Enterprise Edition
Enable CDC in database and then use System Stored Procedures sp_cdc… to create:
CDC tables and functions in System Tables
A log-reading agent (similar to replication) so there is no latency or direct performance impact
Avoids performance impact during writes
Introduces small latency – cannot query the changes immediately
An agent to manage retention of data
You can specify which columns to capture changes for (using @captured_column_list argument of sys.sp_cdc_enable_table)
Eric Berkowitz - - (561)

8. Change Data Capture (cont’d)
Considerations
If you need to introduce a column in the underlying table or a separate one to record the user changing the data this will require a trigger anyway
The capture table retains its shape during a schema change
Dropped columns info replaced with null
New columns are not added – this is possibly most significant weakness
Workaround: create a second capture with the new data structure and drop the old capture when the previous data is no longer needed (there is a max of two capture instances per table)
Database owner must be sysadmin
SQL Server Agent must be running
Incompatible with databases supporting memory-optimized tables (like the demo database)
Failover considerations in High Availability
When replication is not used the sp_cdc_add_job must be run on the (now) primary to capture changes
When replication is used this isn’t necessary because the log reader handles the change captures
Either way the cleanup job needs to be set up on the (new) primary
Eric Berkowitz - - (561)

9. Change Data Capture Configuration
Using System Stored Procedures
-- Enable CDC in the database
EXEC sys.sp_cdc_enable_db GO
-- Enable the table for CDC
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'People',
@role_name = NULL,
@supports_net_changes = 1
EXEC [sys].[sp_cdc_change_job]
@job_type = N'cleanup',
@retention = days
Eric Berkowitz - - (561)

10. Change Data Capture (cont’d)
Functions are created when you enable a table for data capture
The query functions that are created are determined by supports_net_changes
0: functions to query for all changes. Returns multiple rows, i.e. one net change row for each source row changed within the LSN range.
1: additionally creates functions to query for net changes. Returns one row for the content at end of LSN range.
For Net changes, must have a primary key or you must specify the index_name parameter
Eric Berkowitz - - (561)

11. Eric Berkowitz - eric@berkflow.com - (561) 914-2017
Change Data Capture
Query using functions created when the table was enabled
Typically specify the begin and end dates to search, ‘all update old’ (to include previous values) or ‘all’
Operation:
1=Delete
2=Insert
3=Update (before)
4=Update (after)
Eric Berkowitz - - (561)

12. “Siri, what is a temporal table?”
“I found this information about Tempura on the Web”
Eric Berkowitz - - (561)

13. Temporal (System Versioned) Tables
More native implementation of the Trigger/History table technique
Two Datetime2 columns added to the table track the “currency” (my term) of the row over time – bounds of from date it was current (row start) and through date (row end), one for Period
Eric Berkowitz - - (561)

14. Temporal (System Versioned) Tables
The terminology for the tables is the “CURRENT” and “HISTORY” tables
The history table is transparent with a preassigned name unless you want to specify its storage or create indexes.
Limitations – these are only a few!
No truncating
No INSTEAD OF triggers on table
History table must be in same database
Limited use of replication
Current table cannot include FILETABLE or FILESTREAM
Same schema as current table
Recommendations
Traditional trigger method allows you to omit varchar(max), varbinary(max) text, image so if you don’t anticipate needing to track changes to values in these columns you can save a ton of storage by using trigger.
In a new database, consider vertically partitioning tables that you are planning to make system-versioned so these columns aren’t written to history table
Existing INSERTS, SELECT INTO, and even SELECTS that don’t specify column names may have trouble with the generated columns, but you can hide them to avoid this. If they’re not hidden, use DEFAULT as the inserted value
System versioning must be off to drop a temporal table; use SSMS to generate the drop script since it will do this and also drop the transparent history table
Eric Berkowitz - - (561)

15. Querying Temporal Table
Using System Stored Procedures
Add FOR SYSTEM_TIME clause to SELECT statement
Use one of these sub-clauses in your query
AS OF datetime
FROM datetime TO datetime
(Row Start < TO value) and (Row End > FROM value)
BETWEEN datetime AND datetime
(Row Start <= second value) and (Row End > First value)
CONTAINED IN (datetime, datetime)
(Row Start >= first value) and (Row End <= Second Value)
LIVE DEMOS!
Eric Berkowitz - - (561)

16. Make Existing table a Temporal Table
Use ALTER to add the ROW START, ROW END and PERIOD columns
Make them hidden to avoid problems with existing application code
In either case, use DEFAULT as the inserted value in an INSERT
ALTER table to set SYSTEM_VERSIONING ON and optionally speciafy the history table
Eric Berkowitz - - (561)

17. Other Temporal Table features
Built-in support for stretching and partitioning to Azure
Retention:
You can create a custom stored procedure to manage retention but this also involves turning off SYSTEM_VERSIONING on the table (do it inside a transaction)
You can set the retention period to be managed automatically at time of creation or through ALTER
HISTORY_RETENTION_PERIOD = 6 MONTHS
Takes effect immediately in queries but data is not immediately deleted, and you can change the setting back to a longer duration
If the primary use case is data audit (i.e. searching for historical changes for a single row from the current table), then a good choice is to create rowstore history table with a clustered index (direct from Microsoft)
Eric Berkowitz - - (561)

18. Eric Berkowitz - eric@berkflow.com - (561) 914-2017
Recommendations
Temporal tables is great choice for new development
When database is being designed from ground-up with it in mind; and,
When it is intended to be used as a permanent feature
But you have the greatest chance of unintentionally breaking production
You can replicate changes from older databases to a SQL 2017 server
Triggers are a good choice when you are very confident they won’t break production or cause performance issues, or need specific columns, or want to omit users, column changes or applications
CDC is good as a low-risk transparent solution for short- and long-term auditing and change tracking
Smallest chance of harming production except in replicated environments
In all cases, be wary of security issues on the history tables
Eric Berkowitz - - (561)