SQL Server 2016 Database Engine – New Features -.

SQL Server 2016 Database Engine – New Features -

383 Companies in 12 countries
$4 million in the average total cost of data breach 29% increase in total cost of data breach since 2013 $158 is the average cost per lost or stolen record 15% increase in per capita cost since 2013.

Mohit K. Gupta Microsoft SQL Server PFE Twitter: @SQLCAN
Blog:

Mission-critical performance
Security Availability Scalability Operational analytics Insights on operational data; works with in-memory OLTP and disk-based OLTP In-memory OLTP enhancements Greater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs Query data store Monitor and optimize query plans Native JSON Expanded support for JSON data Temporal database support Query data as points in time Always encrypted Sensitive data remains encrypted at all times with ability to query Row-level security Apply fine-grained access control to table rows Dynamic data masking Real-time obfuscation of data to prevent unauthorized access Other enhancements Audit success/failure of database operations TDE support for storage of In- Memory OLTP tables Enhanced auditing for OLTP with ability to track history of record changes Enhanced AlwaysOn Three synchronous replicas for automatic failover across domains Round-robin load balancing of replicas Automatic failover based on database health DTC for transactional integrity across database instances with AlwaysOn Support for SSIS with AlwaysOn Enhanced database caching Cache data with automatic, multiple TempDB files per instance in multicore environments Performance Enhanced in-memory performance with up to 30x faster transactions, more than 100x faster queries than disk-based relational databases and real-time operational analytics. Security Upgrades Always Encrypted technology helps protect your data at rest and in motion, on-premises and in the cloud, with master keys sitting with the application, without any application changes. High Availability Even higher availability and performance than SQL Server 2014 of your AlwaysOn secondaries with the ability to have up to 3 synchronous replicas, DTC support, and round-robin load balancing of the secondaries. Scalability Enhanced database caching across multiple cores and support for Windows Server 2016 that efficiently scale compute, networking, and storage in both physical and virtual environments.

Operational analytics: disk-based and in-memory tables

Traditional operational/analytics architecture
Microsoft Ignite 2015 11/13/2018 8:57 PM Traditional operational/analytics architecture BI analysts BI and analytics Dashboards Reporting Key issues Complex implementation Requires two servers (capital expenditures and operational expenditures) Data latency in analytics High demand; requires real-time analytics Presentation layer IIS Server SQL Server Analysis Services Application tier ETL SQL Server Relational DW SQL Server Hourly, daily, weekly Database Database Performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Minimizing data latency for analytics
Microsoft Ignite 2015 11/13/2018 8:57 PM Minimizing data latency for analytics Challenges Analytics queries are resource intensive and can cause blocking Minimizing impact on operational workloads Sub-optimal execution of analytics on relational schema Benefits No data latency No ETL No separate data warehouse BI analysts BI and analytics Dashboards Reporting SQL Server Database Application tier Presentation layer IIS Server SQL Server Analysis Services This is OPERATIONAL ANALYTICS The ability to run analytics queries concurrently with operational workloads using the same schema is called operational analytics. Not a replacement for: Extreme analytics queries performance possible using schemas customized (Star/Snowflake) and pre-aggregated cubes Data coming from non-relational sources Data coming from multiple relational sources requiring integrated analytics Add analytics-specific indexes Performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Operational analytics
11/13/2018 Operational analytics The ability to run analytics queries concurrently with operational workloads using the same schema Not a replacement for: Extreme analytics performance queries possible only using customized schemas (e.g. Star/Snowflake) and pre-aggregated cubes Data coming from non-relational sources Data coming from multiple relational sources requiring integrated analytics Performance © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SQL Server 2016: operational analytics
Goals: Minimal impact on operational workloads with concurrent analytics Performance analytics for operational schema Achieved using columnstore indexes Performance

Operational analytics: disk-based tables

Operational analytics with columnstore index
Key points Create an updateable NCCI for analytics queries Drop all other indexes that were created for analytics No application changes Columnstore index is maintained just like any other index Query optimizer will choose columnstore index where needed Relational table (Clustered index/heap) B-tree index Delete bitmap Delta row groups Nonclustered columnstore index (NCCI) Performance

Operational analytics: Columnstore index overhead
Operational analytics with columnstore index DML operations on OLTP workload Operation B-tree (NCI) Non-clustered columnstore index (NCCI) Insert Insert row into B-tree. Insert row into B-tree (delta store). Delete Seek row(s) to be deleted. Delete the row. Seek row in delta stores. (There can be multiple rows.) If found, delete row. If not found, insert key into delete row buffer. Update Seek the row(s). Update. Delete row (steps same as above). Insert updated row into delta store. Operational analytics: Columnstore index overhead Performance

Operational analytics: minimizing columnstore overhead
Relational table (clustered index/heap) B-tree index Key points Create columnstore only on cold data by using filtered predicate to minimize maintenance Analytics query accesses both columnstore and ‘hot’ data transparently Example: Order Management Application: create nonclustered columnstore index where order_status = ‘SHIPPED’ HOT Delete bitmap Delta row groups Nonclustered columnstore index (NCCI): filtered index Performance

Using Availability Groups instead of data warehouses
AlwaysOn Availability Group Key points Mission-critical operational workloads typically configured for high availability using AlwaysOn Availability Groups You can offload analytics to readable secondary replica Secondary replica Primary replica Source: To configure an AlwaysOn availability group to support read-only routing in SQL Server 2016, you can use either Transact-SQL or PowerShell. Read-only routing refers to the ability of SQL Server to route qualifying read-only connection requests to an available AlwaysOn readable secondary replica (that is, a replica that is configured to allow read-only workloads when running under the secondary role). To support read-only routing, the availability group must possess an availability group listener. Read-only clients must direct their connection requests to this listener, and the client's connection strings must specify the application intent as "read-only." That is, they must be read-intent connection requests. Performance

Operational analytics: in-memory tables

Operational analytics: columnstore on in-memory tables
11/13/2018 Operational analytics: columnstore on in-memory tables Hash index No explicit delta row group Rows (tail) not in columnstore stay in In-Memory OLTP table No columnstore index overhead when operating on tail Background task migrates rows from tail to columnstore in chunks of 1 million rows not changed in last 1 hour Deleted Rows Table (DRT) – Tracks deleted rows Columnstore data fully resident in memory Persisted together with operational data No application changes required Range index Deleted Rows Table Tail In-Memory OLTP table Hot Source: For an overview of columnstore indexes, see Columnstore Indexes Described. Create a Clustered Columnstore Index To create a clustered columnstore index, first create a rowstore table as a heap or clustered index, and then use the CREATE CLUSTERED COLUMNSTORE INDEX (Transact-SQL) statement to convert the table to a clustered columnstore index. If you want the clustered columnstore index to have the same name as the clustered index, use the DROP_EXISTING option. Load Data into a Clustered Columnstore Index You can add data to an existing clustered columnstore index by using any of the standard loading methods. For example, the bcp bulk loading tool, Integration Services, and INSERT … SELECT can all load data into a clustered columnstore index. Clustered columnstore indexes leverage the deltastore in order to prevent fragmentation of column segments in the columnstore. Deltastore loading scenarios Rows accumulate in the deltastore until the number of rows is the maximum number of rows allowed for a rowgroup. When the deltastore contains the maximum number of rows per rowgroup, SQL Server marks the rowgroup as “CLOSED”. A background process, called the “tuple-mover”, finds the CLOSED rowgroup and moves into the columnstore, where the rowgroup is compressed into column segments and the column segments are stored in the columnstore. For each clustered columnstore index there can be multiple deltastores. If a deltastore is locked, SQL Server will try to obtain a lock on a different deltastore. If there are no deltastores available, SQL Server will create a new deltastore. For a partitioned table, there can be one or more deltastores for each partition. Change Data in a Clustered Columnstore Index Clustered columnstore indexes support insert, update, and delete DML operations. Use INSERT (Transact-SQL) to insert a row. The row will be added to the deltastore. Use DELETE (Transact-SQL) to delete a row. If the row is in the columnstore, SQL Server marks the row as logically deleted but does not reclaim the physical storage for the row until the index is rebuilt. If the row is in the deltastore, SQL Server logically and physically deletes the row. Use UPDATE (Transact-SQL) to update a row. If the row is in the columnstore, SQL Server marks the row as logically deleted, and then inserts the updated row into the deltastore. If the row is in the deltastore, SQL Server updates the row in the deltastore. Performance © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Operational analytics: columnstore overhead
DML operations on In-Memory OLTP Operation Hash or range index HK-CCI Insert Insert row into HK. Delete Seek row(s) to be deleted. Delete the row. Delete the row in HK. If row in TAIL, then return. If not, insert <colstore-RID> into DRT. Update Seek the row(s) to be updated. Update (delete/insert). Seek the row(s) to be updated. Update (delete/insert) in HK. If row in TAIL, then return Source: DML Triggers: SQL Server now support AFTER triggers for INSERT, UPDATE, and DELETE operations on memory-optimized tables. Triggers on memory-optimized tables must be natively-compiled, and natively-compiled triggers are only supported for memory-optimized tables. Note that, like all native modules, native triggers must be schemabound and must consist of an ATOMIC block. To create a trigger, right-click on the Triggers node for the memory-optimized table in Management Studio and select "New Natively Compiled Trigger". Performance

Operational analytics: minimizing columnstore overhead
Microsoft Ignite 2015 11/13/2018 8:57 PM Operational analytics: minimizing columnstore overhead In-Memory OLTP table Updateable CCI Tail DRT Range index Hash index Hot Like delta row group DML operations Keep hot data only in in-memory tables Example: data stays hot for 1 day, 1 week… Workaround: Use TF – 9975 to disable auto-compression Force compression using a spec-proc “sp_memory_optimized_cs_migration” Analytics queries Offload analytics to AlwaysOn readable secondary Source: No MSDN topic for this yet. Alternative source: Niko Neugebauer – New stored procedure sys.sp_memory_optimized_cs_migration which should migrate data from the Tail Row Group into the compressed Row Groups, make sure that you are invoking the version from the local database and not from the master DB © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary of improvements
SQL Server 2014 SQL Server 2016 clustered columnstore index Master copy of the data (10x compression) Only index supported; simplified analytics No PK/FK constraints Uniqueness can be enforced through materialized views Locking granularity for UPDATE/DELETE at row group level DDL: ALTER, REBUILD, REORGANIZE Additional B-tree indexes for efficient equality, short-range searches, and PK/FK constraints Locking granularity at row level using NCI index path updateable non-clustered index Introduced in SQL Server 2012 NCCI is read-only: no delete bitmap or delta store Optimizer will choose between NCCI and NCI(s)/CI or heap-based on the cost-based model Partitioning supported Updateable Ability to mix OLTP and analytics workload Ability to create filtered NCCI equality and short-range queries Row group elimination (when possible) Partition-level scan (somewhat expensive) Full index scan (expensive) Optimizer can choose NCI on column C1; index points directly to row group No full index scan Covering NCI index string predicate pushdown Retrieve 10 million rows by converting dictionary encoded value to string Apply string predicate on 10 million rows Apply filter on dictionary entries Find rows that refer to dictionary entries that qualify (R1) Find rows not eligible for this optimization (R2) Scan returns (R1 + R2) rows Filter node applies string predicate on (R2) Row returned by Filter node = (R1 + R2’) Source: Performance

Support for index maintenance
Operation SQL Server 2014 SQL Server 2016 Removing deleted rows Requires index REBUILD Index REORGANIZE Remove deleted rows from single compressed RG Merge one or more compressed RGs with deleted rows Done ONLINE Smaller RG size resulting from: Smaller BATCHSIZE Memory pressure Index build residual Index REBUILD Ordering rows Create clustered index Create columnstore index by dropping clustered index No changes Query Row group granularity No support for RCSI or SI Recommendation: use read uncommitted Support of SI and RCSI (non-blocking) Insert Lock at row level (trickle insert) Row group level for set of rows Delete Lock at row group level Row-level lock in conjunction with NCI Update Implemented as Delete/Insert AlwaysOn Failover Clustering (FCI) Fully supported AlwaysON Availability Groups Fully supported except readable secondary Fully supported with readable secondary Index create/rebuild Offline Source: Task #1: Defragment the columnstore index Rebuilding a columnstore index removes fragmentation, and moves all rows into the columnstore. Use CREATE COLUMNSTORE INDEX (Transact-SQL) or ALTER INDEX (Transact-SQL) to perform a full rebuild of an existing clustered columnstore index. Additionally, you can use ALTER INDEX … REBUILD to rebuild a specific partition. Recommendations for Rebuilding a Columnstore Index Rebuilding a columnstore index is useful for removing fragmentation, and for moving all rows into the columnstore. We have the following recommendations: Rebuild a partition instead of the entire table. Rebuilding the entire table takes a long time if the index is large, and it requires enough disk space to store an additional copy of the index during the rebuild. Usually it is only necessary to rebuild the most recently used partition. For partitioned tables, you do not need to rebuild the entire columnstore index because fragmentation is likely to occur in only the partitions that have been modified recently. Fact tables and large dimension tables are usually partitioned in order to perform backup and management operations on chunks of the table. Rebuild a partition after heavy DML operations. Rebuilding a partition will defragment the partition and reduce disk storage. Rebuilding will delete all rows from the columnstore that are marked for deletion, and it will move all rowgroups from the deltastore into the columnstore. Note, there can be multiple rowgroups in the deltastore that each have less than one million rows. Rebuild a partition after loading data. This ensures all data is stored in the columnstore. When concurrent processes each load less than 100K rows into the same partition at the same time, the partition can end up with multiple deltastores. Rebuilding will move all deltastore rows into the columnstore. Task #2: Force compression of deltastore rowgroups and remove deleted rows Reorganizing a columnstore index moves the closed rowgroups into the columnstore and removes deleted rows. To perform a reorganize, use ALTER INDEX (Transact-SQL) with the REORGANIZE option. Recommendations for Reorganizing When to reorganize a columnstore index: Reorganize a columnstore index after one or more data loads to achieve query performance benefits as quickly as possible. Reorganizing will initially require additional CPU resources to compress the data, which could slow overall system performance. However, as soon as the data is compressed, query performance can improve. Performance

Summary: operational analytics
Capability Ability to run analytics queries concurrently with operational workloads using the same schema Data Warehouse queries can be run on In-Memory OLTP workloads with no application changes Benefits Minimal impact on OLTP workloads Best performance and scalability available Offloading analytics workload to readable secondary . BI analysts BI and analytics Dashboards Reporting SQL Server Database Application tier Presentation layer IIS Server SQL Server Analysis Services Source: Add analytics-specific indexes Performance

In-Memory OLTP enhancements

In-Memory OLTP enhancements
ALTER support Full schema change support: add/alter/drop column/constraint Add/drop index supported Surface area improvements Almost full T-SQL coverage including scaler user-defined functions Improved scaling Increased size allowed for durable tables; more sockets Other improvements MARS support Lightweight migration reports ALTER TABLE Sales.SalesOrderDetail ALTER INDEX PK_SalesOrderID REBUILD WITH (BUCKET_COUNT= ) T-SQL surface area: New {LEFT|RIGHT} OUTER JOIN Disjunction (OR, NOT) UNION [ALL] SELECT DISTINCT Subqueries (EXISTS, IN, scalar) Source: In SQL Server 2016, improvements to In-Memory OLTP enable scaling to larger databases and higher throughput in order to support bigger workloads. In addition, a number of limitations concerning tables and stored procedures have been removed to make it easier to migrate your applications to and leverage the benefits of In-Memory OLTP. Performance

Altering memory-optimized tables
| ADD { <column_definition> | <table_constraint> | <table_index> } [ ,...n ] | DROP [ CONSTRAINT ] constraint_name | COLUMN column_name | INDEX index_name | ALTER INDEX index_name REBUILD (WITH <rebuild_index_option>) } The ALTER TABLE syntax is used for making changes to the table schema, as well as for adding, deleting, and rebuilding indexes Indexes are considered part of the table definition Key advantage is the ability to change the BUCKET_COUNT with an ALTER INDEX statement Source: In SQL Server 2016 you can perform ALTER operations on memory-optimized tables by using the ALTER TABLE statement. The database application can continue to run, and any operation that is accessing the table is blocked until the alteration process is completed. In the previous release of SQL Server, you had to manually complete several steps to update memory-optimized tables. Before starting the ALTER TABLE operation, the workload needs to be halted. Any user transaction that started before the ALTER TABLE started, and that accesses the table, can cause the ALTER TABLE to fail with a serializable validation failure (error code 41325). The ALTER TABLE syntax is used for making changes to the table schema, as well as for adding, deleting, and rebuilding indexes. Indexes are considered part of the table definition. The syntax ALTER TABLE … ADD/DROP/ALTER INDEX is supported only for memory-optimized tables. The syntax CREATE INDEX, DROP INDEX, and ALTER INDEX (without specifying the ALTER TABLE) is not supported for indexes on memory-optimized tables. Performance

Altering natively compiled stored procedures
CREATE PROCEDURE [dbo].[usp_1] WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER AS BEGIN ATOMIC WITH ( TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N'us_english' ) SELECT c1, c2 from dbo.T1 END GO ALTER PROCEDURE [dbo].[usp_1] SELECT c1 from dbo.T1 You can now perform ALTER operations on natively compiled stored procedures using the ALTER PROCEDURE statement Use sp_recompile to recompile stored procedures on the next execution Source: In SQL Server 2016 , you can perform ALTER operations on natively compiled stored procedures using the ALTER PROCEDURE statement. In the previous release of SQL Server, you had to complete manual steps to modify natively compiled stored procedures. When executing ALTER PROCEDURE on a natively compiled stored procedure, the procedure is recompiled using a new definition. While recompilation is in progress, the old version of the procedure continues to be available for execution. Once compilation completes, procedure executions are drained, and the new version of the procedure is installed.When you alter a natively compiled stored procedure, you can modify the following options. Parameter EXECUTE AS TRANSACTION ISOLATION LEVEL LANGUAGE DATEFIRST DATEFORMAT DELAYED_DURABILITY For more information on ALTER PROCEDURE functionality and syntax, see ALTER PROCEDURE (Transact-SQL) You can execute sp_recompile on a natively compiled stored procedure, which causes the stored procedure to recompile on the next execution. Performance

Greater Transact-SQL (T-SQL) coverage
CREATE PROCEDURE (Transact-SQL) DROP PROCEDURE (Transact-SQL) ALTER PROCEDURE (Transact-SQL) SELECT (Transact-SQL) and INSERT SELECT statements SCHEMABINDING and BEGIN ATOMIC (required for natively compiled stored procedures) NATIVE_COMPILATION Parameters and variables can be declared as NOT NULL Table-valued parameters. EXECUTE AS OWNER, SELF, and user. GRANT and DENY permissions on tables and procedures. Nesting natively compiled stored procedures RIGHT OUTER JOIN, LEFT OUTER JOIN, INNER JOIN, and CROSS JOIN in SELECT statements NOT, OR, and IN operators in SELECT, UPDATE and DELETE statement UNION ALL and UNION SELECT DISTINCT GROUP BY clause with no aggregate functions in the SELECT clause (<select> list). COLUMNSTORE COLLATE Source: For information about unsupported constructs, see Transact-SQL Constructs Not Supported by In-Memory OLTP. The following are supported: CREATE PROCEDURE (Transact-SQL) DROP PROCEDURE (Transact-SQL) ALTER PROCEDURE (Transact-SQL) SELECT (Transact-SQL) and INSERT SELECT statements SCHEMABINDING and BEGIN ATOMIC (required for natively compiled stored procedures) For more information, see Creating Natively Compiled Stored Procedures. NATIVE_COMPILATION For more information, see Native Compilation of Tables and Stored Procedures. Parameters and variables can be declared as NOT NULL (available only for natively compiled modules: natively compiled stored procedures and natively compiled, scalar user-defined functions). Table-valued parameters. For more information, see Use Table-Valued Parameters (Database Engine). EXECUTE AS OWNER, SELF, and user. This is required for natively compiled modules: natively compiled stored procedures and natively compiled, scalar user-defined functions. GRANT and DENY permissions on tables and procedures. For more information, see GRANT Object Permissions (Transact-SQL) and DENY Object Permissions (Transact-SQL). Nesting natively compiled stored procedures You can use the EXECUTE syntax inside the stored procedures, as long as the referenced procedure is also natively compiled. RIGHT OUTER JOIN, LEFT OUTER JOIN, INNER JOIN, and CROSS JOIN in SELECT statements For more information, see FROM (Transact-SQL). NOT, OR, and IN operators in SELECT, UPDATE and DELETE statements, in the WHERE clause, HAVING clause, and JOIN conditions. This support extends to NOT IN and NOT BETWEEN. For more information, see Logical Operators (Transact-SQL). UNION ALL and UNION You can now combine SELECT statements in natively compiled stored procedures. The UNION syntax by itself is also supported. For more information, see UNION (Transact-SQL). SELECT DISTINCT GROUP BY clause with no aggregate functions in the SELECT clause (<select> list). COLUMNSTORE Columnstore index on memory-optimized tables is supported. For more information, see Columnstore Indexes Described. COLLATE In-Memory OLTP now fully supports collations. For more information, see Collations and Code Pages. Performance

Improved scaling Other enhancements include:
Multiple threads to persist memory- optimized tables Multi-threaded recovery MERGE operation Dynamic management view improvements to sys.dm_db_xtp_checkpoint_stats and sys.dm_db_xtp_checkpoint_files DBCC CHECKDB performance changed to support 1 TB database check improvement by 7x In-Memory OLTP engine has been enhanced to scale linearly on servers up to 4 sockets Source: In-memory OLTP engine has been enhanced to scale linearly on servers up to 4 sockets. Other enhancements include: Multiple threads to persist memory-optimized tables. In the previous release of SQL Server, there was a single offline checkpoint thread that scanned the transaction log for changes to memory-optimized tables and persisted them in checkpoint files (such as data and delta files). With larger number of COREs, the single offline checkpoint thread could fall behind. In SQL Server 2016, there are multiple concurrent threads responsible to persist changes to memory-optimized tables. Multi-threaded recovery. In the previous release of SQL Server, the log apply as part of recovery operation was single threaded. In SQL Server 2016, the log apply is multi-threaded. MERGE Operation. The MERGE operation is now multi-threaded. Dynamic management views. sys.dm_db_xtp_checkpoint_stats (Transact-SQL) and sys.dm_db_xtp_checkpoint_files (Transact-SQL)have been changed significantly. Manual Merge has been disabled as multi-threaded merge is expected to keep up with the load. The In-memory OLTP engine continues to use memory-optimized filegroup based on FILESTREAM, but the individual files in the filegroup are decoupled from FILESTREAM. These files are fully managed (such as for create, drop, and garbage collection) by the In-Memory OLTP engine. DBCC SHRINKFILE (Transact-SQL) is not supported. Performance

Improvements in Management Studio
Lightweight performance analysis Transaction Performance Analysis report pinpoints hotspots in the application Generating migration checklists Migration checklists show unsupported features used in current disk-based tables and interpreted T-SQL stored procedures Generated checklists for all or some tables and procedures Use GUI or PowerShell Performance

New Transaction Performance Analysis Overview report
New report replaces the need to use the Management Data Warehouse to analyze which tables and stored procedures are candidates for in-memory optimization Note that in SQL Server 2014 you needed to set up a Management Data Warehouse (MDW) in order to run the In-Memory OLTP reports. SQL Server 2016 now includes lightweight reports available in SSMS without setting up MDW. Open SSMS and right click on the AdventureWorks2016 database and select Reports | Standard Reports | Transaction Performance Analysis Overview. Select the Tables Analysis report link. This presents you with a prioritized plot chart that graphs the amounts of potential gain against the potential amount of work to migrate the top tables to In-Memory OLTP. This analysis is based upon the workload that we just ran with the batch file. This helps you see the highest potential gain with the lowest potential amount of migration work required. Select the SalesOrderDetail_ondisk to explore further. Note that this report contains information on latch statistics, lock statistics, recommended In-Memory index type, and more. Click the Navigate Backwards button twice in the upper left of the report bar (see illustration above) to go back to the main Report and click on Stored Procedure Analysis. Performance

Using multiple active result sets (MARS)
Data Source=MSSQL; Initial Catalog=AdventureWorks; Integrated Security=SSPI; MultipleActiveResultSets=True Set up MARS connection for memory-optimized tables using the MultipleActiveResultsSets =True in your connection string Source: SQL Server 2005 introduced support for multiple active result sets (MARS) in applications accessing the Database Engine. In earlier versions of SQL Server, database applications could not maintain multiple active statements on a connection. When using SQL Server default result sets, the application had to process or cancel all result sets from one batch before it could execute any other batch on that connection. SQL Server 2005 introduced a new connection attribute that allows applications to have more than one pending request per connection, and in particular, to have more than one active default result set per connection. MARS simplifies application design with the following new capabilities: Applications can have multiple default result sets open and can interleave reading from them. Applications can execute other statements (for example, INSERT, UPDATE, DELETE, and stored procedure calls) while default result sets are open. Applications using MARS will find the following guidelines beneficial: Default results sets should be used for short lived or short result sets generated by single SQL statements (SELECT, DML with OUTPUT, RECEIVE, READ TEXT, and so on). Server cursors should be used for longer lived or large result sets generated by single SQL statements. Always read to the end of results for procedural requests regardless of whether they return results or not, and for batches that return multiple results. Wherever possible, use API calls to change connection properties and manage transactions in preference to Transact-SQL statements. In MARS, session-scoped impersonation is prohibited while concurrent batches are running. In-Memory OLTP In-memory OLTP supports MARS using queries and natively compiled stored procedures. MARS enables requesting data from multiple queries without the need to completely retrieve each result set before sending a request to fetch rows from a new result set. To successfully read from multiple open result sets you must use a MARS enabled connection. MARS is disabled by default so you must explicitly enable it by adding MultipleActiveResultSets=True to a connection string. MARS with In-Memory OLTP is essentially the same as MARS in the rest of the SQL engine. The following lists the differences when using MARS in memory-optimized tables and natively compiled stored procedures. MARS and memory-optimized tables The following are the differences between disk-based and memory-optimized tables when using a MARS enabled connection: Two statements can modify data in the same target object but if they both attempt to modify the same record a write-write conflict will cause the new operation to fail. However, if both operations modify different records, the operations will succeed. Each statement runs under SNAPSHOT isolation so new operations cannot see changes made by the existing statements. Even if the concurrent statements are executed under the same transaction the SQL engine creates batch-scoped transactions for each statement that are isolated from each other. However, batch-scoped transactions are still bound together so rollback of one batch-scoped transaction affects other ones in the same batch. DDL operations are not allowed in user transactions so they will immediately fail. MARS and natively compiled stored procedures Natively compiled stored procedures can run in MARS enabled connections and can yield execution to another statement only when a yield point is encountered. A yield point requires a SELECT statement, which is the only statement within a natively compiled stored procedure that can yield execution to another statement. If a SELECT statement is not present in the procedure it will not yield, it will run to completion before other statements begin. MARS and In-memory OLTP transactions Changes made by statements and atomic blocks that are interleaved are isolated from each other. For example, if one statement or atomic block makes some changes, and then yields execution to another statement, the new statement will not see changes made by the first statement. In addition, when first statement resumes execution, it will not see any changes made by any other statements. Statements will only see changes that are finished and committed before the statement starts. A new user transaction can be started within the current user transaction using the BEGIN TRANSACTION statement – this is supported only in interop mode so the BEGIN TRANSACTION can only be called from a T-SQL statement, and not from within a natively compiled stored procedure.You can create a save point in a transaction using SAVE TRANSACTION or an API call to transaction.Save(save_point_name) to rollback to the savepoint. This feature is also enabled only from T-SQL statements, and not from within natively compiled stored procedures. Performance

Support for Transparent Data Encryption (TDE)
Transparent Database Encryption architecture In SQL Server 2016, the storage for memory- optimized tables will be encrypted as part of enabling TDE on the database Simply follow the same steps as you would for a disk-based database Windows Operating System Level Data Protection DPAPI encrypts the Service Master Key SQL Server Instance Level Created at a time of SQL Server setup Service Master Key Service Master Key Encrypts the Database master Key for the master Database Master Database Level Database Encryption Key Statement: CREAT MASTER KEY… Database Master Key of the master Database creates a certificate in the master database Statement: CREATE CERTIFICATE… Source: Encryption In SQL Server 2016, the storage for memory-optimized tables will be encrypted as part of enabling TDE on the database. For more information, see Transparent Data Encryption (TDE). The certificate encrypts the database Encryption Key in the user database User Database Level Statement: CREATE DATABASE ENCRYPTION KEY… Database Encryption Key The entire user database is secured by the Datbase Encryption Key (DEK) of the user database by using transparent database encryption Statement: ALTER DATABSE… SET ENCRYPTION Performance

Summary: In-Memory OLTP enhancements
Capability ALTER support for memory-optimized tables Greater Transact-SQL coverage Benefits Improved scaling: In-Memory OLTP engine has been enhanced to scale linearly on servers up to 4 sockets Tooling improvements in Management Studio MARS (multiple active result sets) support TDE (Transparent Data Encryption)-enabled: all on-disk data files are now encrypted once TDE is enabled Source: Performance

Query Store Your flight data recorder for your database

Problems with query performance
Website Is down Fixing query plan choice regressions is difficult Query plan cache is not well-suited for performance troubleshooting Long time to detect the issue (TTD) Which query is slow? Why is it slow? What was the previous plan? Long time to mitigate (TTM) Can I modify the query? How to use plan guide? Database is not working Temporary perf issues Impossible to predict / root cause DB upgraded Regression caused by new bits Plan choice change can cause these problems Performance

The solution: Query Store
11/13/2018 8:57 PM The solution: Query Store Dedicated store for query workload performance data Captures the history of plans for each query Captures the performance of each plan over time Persists the data to disk (works across restarts, upgrades, and recompiles) Significantly reduces TTD/TTM Find regressions and other issues in seconds Allows you to force previous plans from history DBA is now in control Monitoring Performance By Using the Query Store, August 14, 2015, Performance © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Query data store Query Store SQL
Collects query texts (plus all relevant properties) Stores all plan choices and performance metrics Works across restarts / upgrades / recompiles Dramatically lowers the bar for performance troubleshooting New Views Intuitive and easy plan forcing Compile MSG Execute MSG Query Store Durability latency controlled by DB option DATA_FLUSH_INTERNAL_SECONDS Async Write-Back Compile Execute SQL Plan store Runtime stats Query Store schema Query store is a new feature that provides DBAs with insight on query plan choice and performance. It simplifies performance troubleshooting by enabling you to quickly find performance differences caused by changes in query plans. The feature automatically captures a history of queries, plans, and runtime statistics, and retains these for your review. It separates data by time windows, allowing you to see database usage patterns and understand when query plan changes happened on the server. The query store presents information by using a Management Studio dialog box, and lets you force the query to one of the selected query plans. For more information, see Monitoring Performance By Using the Query Store. Performance

Query Store write architecture
11/13/2018 Query Store write architecture Query Store captures data in-memory to minimize I/O overhead Data is persisted to disk asynchronously in the background Query Store Query Execution Internal tables Query and Plan Store Runtime stats store Query exec. stats Compile Execute async Query text and plan Performance © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Query Store read architecture
11/13/2018 Query Store read architecture Views merge in-memory and on-disk content Users always see ‘latest’ data Query Store views Query Store Query Execution Internal tables Query and Plan Store Runtime stats store Query exec. stats Compile Execute async Query text and plan Performance © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Runtime stats interval
Query Store schema explained Internal tables Exposed views Query text Query 1 - n Plan 1 - n Runtime stats sys. Compile stats: query_store_query_text query_context_settings query_store_query query_store_plan Context settings Runtime stats interval Runtime stats: query_store_runtime_stats_interval query_store_runtime_stats One row per query text per plan affecting option (example: ANSI NULLS on/off) One row per plan (for each query) One row per plan per time interval (example: 5 min) Performance

Keeping stability while upgrading to SQL Sever 2016
SQL Server 2016 QO enhancements tied to database compatibility level Install bits Keep existing compat. level Run Query Store (create a baseline) Move to vNext CompatLevel Fix regressions with plan forcing Source: Maintaining Query Performance Stability For queries that are executed multiple times you may notice that SQL Server used different plans which resulted in different resource utilization and duration. With Query Store you can easily detect when the query performance regressed and determine the optimal plan within a period of interest. Then you can force that optimal plan for future query execution. You can also identify inconsistent query performance for a query with parameters (either auto- parameterized or manually parameterized). Among different plans you can identify plan which is fast and optimal enough for all or most of the parameter values and force that plan; keeping predictable performance for the wider set of user scenarios. Force or a plan for a query (apply forcing policy). When a plan is forced for a certain query, every time a query comes to execution it will be executed with the plan that is forced. EXEC = = 49; When using sp_query_store_force_plan you can only force plans that were recorded by Query Store as a plan for that query. In other words, the only plans available for a query are those that were already used to execute Q1 while Query Store was active. Remove plan forcing for a query. To rely again on the SQL Server query optimizer to calculate the optimal query plan, use sp_query_store_unforce_plan to unforce the plan that was selected for the query. EXEC = = 49; Other notes: De-risk upgrades: QDS PF and COMPAT LEVEL All plans are freezed in pre-upgrade state We changed QP to keep CE changes under compat level Questions: Would described scenario work in your environment? plan freezing will keep your current forcing decisions and apply the LAST plan for other queries. Is that what you would expect? Performance

Monitoring performance by using the Query Store
The Query Store feature provides DBAs with insight on query plan choice and performance Source: The point of this slide is to emphasize that Query Store comes with extraordinary UI that will help broader set of users to benefit from collected perf data immediately. SSMS is focused around handful really the most important scenarios making feature instantly useful in everyday activities of typical DBA. We want to encourage people to try out new UI and learn from it: it is a great knowledge source because people can easily learn first steps of using Query Store DMVs by analyzing queries generated by SSMS. Performance

Working with Query Store
/* (6) Performance analysis using Query Store views*/ SELECT q.query_id, qt.query_text_id, qt.query_sql_text, SUM(rs.count_executions) AS total_execution_count FROM sys.query_store_query_text qt JOIN sys.query_store_query q ON qt.query_text_id = q.query_text_id JOIN sys.query_store_plan p ON q.query_id = p.query_id JOIN sys.query_store_runtime_stats rs ON p.plan_id = rs.plan_id GROUP BY q.query_id, qt.query_text_id, qt.query_sql_text ORDER BY total_execution_count DESC /* (7) Force plan for a given query */ exec sp_query_store_force_plan 12 14 /* (1) Turn ON Query Store */ ALTER DATABASE MyDB SET QUERY_STORE = ON; /* (2) Review current Query Store parameters */ SELECT * FROM sys.database_query_store_options /* (3) Set new parameter values */ ALTER DATABASE MyDB SET QUERY_STORE ( OPERATION_MODE = READ_WRITE, CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ), DATA_FLUSH_INTERVAL_SECONDS = 3000, MAX_SIZE_MB = 500, INTERVAL_LENGTH_MINUTES = 15 ); /* (4) Clear all Query Store data */ ALTER DATABASE MyDB SET QUERY_STORE CLEAR; /* (5) Turn OFF Query Store */ ALTER DATABASE MyDB SET QUERY_STORE = OFF; DB-level feature exposed through T-SQL extensions ALTER DATABASE Catalog views (settings, compile, and runtime stats) Stored Procs (plan forcing, query/plan/stats cleanup) Source: Enabling the Query Store Query Store is not active for new databases by default. By Using the Query Store Page in Management Studio In Object Explorer, right-click a database, and then click Properties. Note Requires at least SQL Server 2016 Community Technology Preview 2 (CTP2) version of Management Studio. In the Database Properties dialog box, select the Query Store page. In the Enable box, select True. By Using Transact-SQL Statements Use the ALTER DATABASE statement to enable the query store. For example: ALTER DATABASE AdventureWorks2012 SET QUERY_STORE = ON; For more syntax options related to the query store, see ALTER DATABASE SET Options (Transact-SQL). See Key Usage Scenarios topic - Performance

Live query statistics View CPU/memory usage, execution time, query progress, and more Enables rapid identification of potential bottlenecks for troubleshooting query performance issues Allows drill down to live operator level statistics: Number of generated rows Elapsed time Operator progress Live warnings Source: SQL Server Management Studio provides the ability to view the live execution plan of an active query. This live query plan provides real-time insights into the query execution process as the controls flow from one query plan operator to another. The live query plan displays the overall query progress and operator-level run-time execution statistics such as the number of rows produced, elapsed time, operator progress, etc. Because this data is available in real time without needing to wait for the query to complete, these execution statistics are extremely useful for debugging query performance issues. Remarks The statistics profile infrastructure must be enabled before live query statistics can capture information about the progress of queries. Specifying Include Live Query Statistics in Management Studio enables the statistics infrastructure for the current query session. There are two other ways to enable the statistics infrastructure which can be used to view the live query statistics from other sessions (such as from Activity Monitor). Execute SET STATISTICS XML ON; or SET STATISTICS PROFILE ON; in the target session. Enable the query_post_execution_showplan extended event. This is a server wide setting that enable live query statistics on all sessions. To enable extended events, see Monitor System Activity Using Extended Events. Limitations Queries using columnstore indexes are not supported. Queries with memory optimized tables are not supported. Natively compiled stored procedures are not supported. Performance

Summary: Query Store Capability Benefits
Query Store helps customers quickly find and fix query performance issues Query Store is a ‘flight data recorder’ for database workloads Benefits Greatly simplifies query performance troubleshooting Provides performance stability across SQL Server upgrades Allows deeper insight into workload performance Source: Performance

JavaScript Object Notation (JSON)
11/13/2018 8:57 PM JavaScript Object Notation (JSON) © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

JSON and Microsoft A top feature request on MS Connect (1050 votes)
11/13/2018 8:57 PM JSON and Microsoft OneDrive Office Dynamics Bing Yammer TFS A top feature request on MS Connect (1050 votes) Performance © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Use cases Generate web service content Flexible database schema
Generate JSON that will be returned to clients Flexible database schema Make reasonable trade-offs in database schema design Analyze JSON documents Parse, query, and analyze JSON documents Service integration Exchange information with various services Performance

JSON in SQL Server Number Date Customer Price Quantity
11/13/2018 JSON in SQL Server Built-in functions ISJSON JSON_VALUE JSON_MODIFY Table 2 JSON Formats result set as JSON text. [ { "Number":"SO43659", "Date":" T00:00:00" "AccountNumber":"AW29825", "Price":59.99, "Quantity": }, { "Number":"SO43661", "Date":" T00:00:00“ "AccountNumber":"AW73565“, "Price":24.99, "Quantity": } ] Number Date Customer Price Quantity SO43659 T00:00:00 MSFT 59.99 1 SO43661 T00:00:00 Nokia 24.99 3 JSON support in a nutshell Easily format result sets as JSON Use built-in functions to handle JSON data (check validity, extract scalar values or objects from it) Transform JSON into a relational table JSON 2 table Migrates JSON text to table Performance © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Main facts No custom type or index Does JSON work with X?
11/13/2018 Main facts No custom type or index Store as NVARCHAR Does JSON work with X? Does NVARCHAR work with X? Yes, with in-memory, row-level security, stretch, compression, encryption, and more Yes, with all client drivers Different from DocumentDB SQL DB: Relational, easily exchanges data with modern services, unified insights DocumentDB: Transactional schema-less document store © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Built-in functions CREATE TABLE SalesOrderRecord ( Id int PRIMARY KEY IDENTITY, OrderNumber NVARCHAR(25) NOT NULL, OrderDate DATETIME NOT NULL, JSalesOrderDetails NVARCHAR(4000) CONSTRAINT SalesOrderDetails_IS_JSON CHECK ( ISJSON(JSalesOrderDetails)>0 ), Quantity AS CAST(JSON_VALUE(JSalesOrderDetails, '$.Order.Qty') AS int) ) GO CREATE INDEX idxJson ON SalesOrderRecord(Quantity) INCLUDE (Price); JSON is plain text ISJSON guarantees consistency Optimize further with computed column and INDEX Source:

SQL Server and Azure DocumentDB
Microsoft Ignite 2015 11/13/2018 8:57 PM SQL Server and Azure DocumentDB Premium relational DB capable to exchange data with modern apps and services Derives unified insights from structured/unstructured data Schema-free NoSQL document store Scalable transactional processing for rapidly changing apps JSON JS JSON See TechNet Virtual Lab “Exploring SQL Server 2016 support for JSON data” - SQL Server 2016 Performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary: JSON Capability Benefits
Greatly enhances developer productivity Benefits Added native JSON support in the core database engine supports schema-free data. Tackle more diverse data types right in SQL Server Support in DocumentDB Performance

Temporal Query back in time

Why temporal Data changes over time Temporal in DB
11/13/2018 8:57 PM Why temporal Data changes over time Tracking and analyzing changes is often important Temporal in DB Automatically tracks history of data changes Enables easy querying of historical data states Advantages over workarounds Simplifies app development and maintenance Efficiently handles complex logic in DB engine Time travel Data audit Slowly changing dimensions Repair record-level corruptions Source: A temporal table is a new type of table that provides correct information about stored facts at any point in time. Each temporal table consists of two tables actually, one for the current data and one for the historical data. The system automatically ensures that when the data changes in the table with the current data, the previous values are stored in the historical table. Querying constructs are provided to hide this complexity from users. For more information, see Temporal Tables. Introduction to Key Components and Concepts What is a Temporal Table? A temporal table is a table for which a PERIOD definition exists and which contains system columns with a datatype of datetime2 into which the period of validity is recorded by the system, and which has an associated history table into which the system records all prior versions of each record with their period of validity. With a temporal table, the value of each record at any point in time can be determined, rather than just the current value of each record. A temporal table is also referred to as a system-versioned table. Why Temporal? Real data sources are dynamic and more often than not business decisions rely on insights that analysts can get from data evolution. Use cases for temporal tables include: Understanding business trends over time Tracking data changes over time Auditing all changes to data Maintaining a slowly changing dimension for decision support applications Recovering from accidental data changes and application errors Performance © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

No change in programming model
Microsoft Ignite 2015 11/13/2018 8:57 PM How to start with temporal ANSI compliant No change in programming model New Insights FOR SYSTEM_TIME AS OF FROM..TO BETWEEN..AND CONTAINED IN Temporal Querying CREATE temporal TABLE PERIOD FOR SYSTEM_TIME… ALTER regular_table TABLE ADD PERIOD… DDL INSERT / BULK INSERT UPDATE DELETE MERGE DML SELECT * FROM temporal Querying Source: How Does Temporal Work? System-versioning for a table is implemented as a pair of tables, a current table and a history table. Within each of these tables, two additional datetime (datetime2 datatype) columns are used to define the period of validity for each record – a system start time (SysStartTime) column and a system end time (SysEndTime) column. The current table contains the current value for each record. The history table contains the each previous value for each record, if any, and the start time and end time for the period for which it was valid. INSERTS: On an INSERT, the system sets the value for the SysStartTime column to the UTC time of the current transaction based on the system clock and assigns the value for the SysEndTime column to the maximum value of – this marks the record as open. UPDATES: On an UPDATE, the system stores the previous value of the record in the history table and sets the value for the SysEndTime column to the UTC time of the current transaction based on the system clock. This marks the record as closed, with a period recorded for which the record was valid. In the current table, the record is updated with its new value and the system sets the value for the SysStartTime column to the UTC time for the transaction based on the system clock. The value for the updated record in the current table for the SysEndTime column remains the maximum value of DELETES: On a DELETE, the system stores the previous value of the record in the history table and sets the value for the SysEndTime column to the UTC time of the current transaction based on the system clock. This marks this record as closed, with a period recorded for which the previous record was valid. In the current table, the record is removed. Queries of the current table will not return this value. Only queries that deal with history data return data for which a record is closed. MERGE: On a MERGE, MERGE behaves as an INSERT, an UPDATE, or a DELETE based on the condition for each record. Performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Temporal database support: BETWEEN
Provides correct information about stored facts at any point in time, or between two points in time There are two orthogonal sets of scenarios with regards to temporal data: System (transaction)-time Application-time SELECT * FROM Person.BusinessEntityContact FOR SYSTEM_TIME WHERE ContactTypeID = 17 Source: Returns a table with the values for all record versions that were active within the specified time range, regardless of whether they started being active before the <start_date_time> parameter value for the FROM argument or ceased being active after the <end_date_time> parameter value for the TO argument. Internally, a union is performed between the temporal table and its history table and the results are filtered to return the values for all row versions that were active at any time during the time range specified. Records that became active exactly on the lower boundary defined by the FROM endpoint are included and records that became active exactly on the upper boundary defined by the TO endpoint are also included. Performance

How does system-time work?
Microsoft Ignite 2015 11/13/2018 8:57 PM How does system-time work? Temporal table (actual data) History table * Old versions Source: The SYSTEM_TIME period columns used to record the SysStartTime and SysEndTime values must be defined with a datatype of datetime2. Update */ Delete * Insert / Bulk Insert Performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

* Include historical version
Microsoft Ignite 2015 11/13/2018 8:57 PM How does system-time work? Temporal table (actual data) History table * Include historical version Source: The SYSTEM_TIME period columns used to record the SysStartTime and SysEndTime values must be defined with a datatype of datetime2. Regular queries (current data) Temporal queries * (Time travel, etc.) Performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Easy time travel querying
Application-time temporal ALTER TABLE Employee ADD CONSTRAINT FK_Employee_Department FOREIGN KEY (LocationId, PERIOD VALID_TIME) REFERENCES Location (LocationId, PERIOD VALID_TIME); SELECT * FROM Employee WHERE VALID_TIME CONTAINS ' ' WHERE EmployeeNumber = 1 AND VALID_TIME OVERLAPS PERIOD (' ', ' ') /* Temporal join */ SELECT * FROM Employee E JOIN Position D ON E.Position = D.Position AND D.VALID_TIME CONTAINS PERIOD E.VALID_TIME UPDATE Employee FOR PORTION OF VALID_TIME FROM ' ' TO ' ' SET [Position] = 'CEO' WHERE EmployeeNumber = 1 DELETE FROM Employee FROM ' ' TO ' ' CREATE TABLE Employee ( [EmployeeNumber] int NOT NULL, [Name] nvarchar(100) NOT NULL, [LocationId] int NOT NULL, [Position] varchar(50) NOT NULL, [AnnualSalary] decimal (10,2) NOT NULL, ValidFrom datetime2 NOT NULL, ValidTo datetime2 NOT NULL, PERIOD FOR VALID_TIME (ValidFrom,ValidTo), CONSTRAINT PK_Employee PRIMARY KEY CLUSTERED (EmployeeNumber, VALID_TIME WITHOUT OVERLAPS) ) Limits of system-time Time flows ‘forward only’ System-time ≠ business-time (sometimes) Immutable history, future does not exist App-time = new scenarios Correct past records as new info is available (HR, CRM, insurance, banking) Project future events (budgeting, what-if, loan repayment schedule) Batch DW loading (with delay) Go through limits of system time… Examples of use cases for app time… Main value prop for app-time: Consistency (non-overlapping constraints and such) Temporal edits (preserving the logical sequence of validity periods) Easy querying Consistency Easy time travel querying Temporal edits Performance

Temporal data continuum
Microsoft Ignite 2015 11/13/2018 8:57 PM Temporal data continuum SQL Database Temporal queries Temporal and Stretch are a powerful combination! Scenario: Historical data much larger than the current data Retained between 3 and 10 years “Warm”: up to a few weeks/months “Cold”: rarely queried Solution: Stretch the history table Temporal querying still works Performance © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

In-Memory OLTP and temporal
11/13/2018 In-Memory OLTP and temporal Current data (in-memory) Extreme OLTP with cost-effective data history Disk-based history table Super-fast DML and current data querying Temporal querying in interop mode Fast DML In-memory buffer (internal) Could you turn your in-memory table into a temporal table? Yes! Extreme OLTP comes from in-memory tables, cost-effective data changes come from Temporal Go through items and basically read them How does this work? Animate the diagram: 1) Actual table (in-mem), in-mem buffer (that enables super-fast DML) and disk-based history table Internal data retention Historical data (disk-based) Performance © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary: temporal Capability Benefits
Greatly enhances developer productivity Benefits Added temporal database support so you can record, audit, and query data changes over time

Always Encrypted

The need for Always Encrypted
11/13/2018 The need for Always Encrypted Prevents data disclosure Client-side encryption of sensitive data using keys that are never given to the database system Queries on encrypted data Support for equality comparison, including join, group by, and distinct operators Application transparency Minimal application changes via server and client library enhancements Allows customers to securely store sensitive data outside of their trust boundary. Data remains protected from high-privileged, yet unauthorized users. Security © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SQL Server or SQL Database
How it works Help protect data at rest and in motion, on-premises and in the cloud Encrypted sensitive data and corresponding keys are never seen in plaintext in SQL Server SQL Server or SQL Database trust boundary Client "SELECT Name FROM Customers WHERE SSN " " "SELECT Name FROM Customers WHERE SSN 0x7ff654ae6d ciphertext ADO .NET Result Set Result Set Name Wayne Jefferson Name 0x19ca706fbd9a dbo.Customers Name SSN Country 0x19ca706fbd9a 0x7ff654ae6d USA ciphertext Security

Types of encryption for Always Encrypted
Randomized encryption Encrypt(' ') = 0x17cfd50a Repeat: Encrypt(' ') = 0x9b1fcf32 Allows for transparent retrieval of encrypted data but NO operations More secure Deterministic encryption Encrypt(' ') = 0x85a55d3f Repeat: Encrypt(' ') = 0x85a55d3f Allows for transparent retrieval of encrypted data AND equality comparison E.g. in WHERE clauses and joins, distinct, group by Two types of encryption available Randomized encryption uses a method that encrypts data in a less predictable manner Deterministic encryption uses a method which always generates the same encrypted value for any given plaintext value Source: Selecting Deterministic or Randomized Encryption Always Encrypted supports two types of encryption: randomized encryption and deterministic encryption. Deterministic encryption uses a method which always generates the same encrypted value for any given plain text value. Using deterministic encryption allows grouping, filtering by equality, and joining tables based on encrypted values, but can also allow unauthorized users to guess information about encrypted values by examining patterns in the encrypted column. This weakness is increased when there is a small set of possible encrypted values, such as True/False, or North/South/East/West region. Deterministic encryption must use a column collation with a binary2 sort order for character columns. Randomized encryption uses a method that encrypts data in a less predictable manner. Randomized encryption is more secure, but prevents equality searches, grouping, indexing, and joining on encrypted columns. Use deterministic encryption for columns that will be used as search or grouping parameters, for example a government ID number. Use randomized encryption, for data such as confidential investigation comments, which are not grouped with other records, or used to join tables, and which are lthe row which contains the encrypted column of interest. Security

Key provisioning 1. Generate CEKs and master key 2. Encrypt CEK
Column encryption key (CEK) Column master key (CMK) 2. Encrypt CEK Encrypted CEK CMK store: Certificate store HSM Azure Key Vault … CMK 3. Store master key securely Source: Creating and registering a custom Column Master Key Store Provider Information the driver receives from SQL Server for query parameters which need to be encrypted, and for query results which need to be decrypted, includes: An encrypted value of a column encryption key, which should be used to encrypt or decrypt a parameter or a result. The name of a key store provider that encapsulates a key store containing the column master key which was used to encrypt the column encryption key. A key path that specifies the location of the column master key in the key store. The name of the algorithm that was used to encrypt the column encryption key. The driver uses the above information to use the key store provider implementation to decrypt the retrieved encrypted value of the column encryption key, which is subsequently used to either encrypt a query parameter or to decrypt a query result. The driver comes with an implementation for one system provider: SqlColumnEncryptionCertificateStoreProvider which can be used to store column master keys in Windows Certificate Store. You can use a custom key store provider by extending the SqlColumnEncryptionKeyStoreProvider class and registering it using the SqlConnection.RegisterColumnEncryptionKeyStoreProviders() method. Security Officer 4. Upload encrypted CEK to DB Encrypted CEK Database Security

Example Client (trusted) CMK store SQL Server (untrusted)
using (SqlCommand cmd = new SqlCommand( "SELECT Name FROM Customers WHERE SSN , conn)) { cmd.Parameters.Add(new SqlParameter( SqlDbType.VarChar, 11).Value = " "); SqlDataReader reader = cmd.ExecuteReader(); Enhanced ADO.NET exec sp_describe_parameter_encryption @params = VARCHAR(11)' = N'SELECT * FROM Customers WHERE SSN Param Encryption Type/ Algorithm Encrypted CEK Value CMK Store Provider Name CMK Path @SSN DET/ AES 256 CERTIFICATE_STORE Current User/ My/f2260… Plaintext CEK Cache Encryption metadata EXEC sp_execute_sql N'SELECT * FROM Customers WHERE SSN = Source: Result set (plaintext) Param Encryption Type/ Algorithm Encrypted CEK Value CMK Store Provider Name CMK Path @Name Non-DET/ AES 256 CERTIFICATE_STORE Current User/ My/f2260… Name Jim Gray Encryption metadata Name 0x19ca706fbd9 Result set (ciphertext) Security

Existing application setup User experience: SSMS or SSDT (Visual Studio)
UI for selecting columns (no automated data classification) Select candidate columns to be encrypted Analyze schema and application queries to detect conflicts and identify optimal encryption settings Set up the keys Encrypt selected columns while migrating the database to a target server (e.g. in SQL Database Key Setup tool to streamline selecting CMK, generating and encrypting CEK, and uploading key metadata to the database Encryption tool creating new (encrypted) columns, copying data from old (plaintext) columns, swapping columns and re-creating dependencies Select desired encryption settings for selected columns UI for configuring encryption settings on selected columns (accepting/editing recommendations from the analysis tool) Schema/workload analysis tool analyzing the schema and profiler logs Source: Security

Setup (SSMS or SSDT) User experience: SSMS or SSDT (Visual Studio)
Select columns to be encrypted Analyze schema and application queries to detect conflicts (build time) Set up the keys: master & CEK Static schema analysis tool (SSDT only) UI for selecting columns (no automated data classification) Key setup tool to automate selecting CMK, generating and encrypting CEK, and uploading key metadata to the database Source: Security

Server & Tools Business
11/13/2018 Summary: Always Encrypted Protect data at rest and in motion, on-premises and in the cloud Capability ADO.Net client library provides transparent client-side encryption, while SQL Server executes T-SQL queries on encrypted data Benefits Sensitive data remains encrypted and queryable at all times on-premises and in the cloud Unauthorized users never have access to data or keys No application changes Data remains encrypted during query TCE-enabled ADO .NET library Apps SQL Server Encrypted query No app changes Master key Source: Source: When it comes to mission critical security we are introducing a unique encryption technology that protects data at rest and in motion and can be full queried while encrypted. The new ADO .NET library provide transparent client-side ecryption, while SQL Server executes T-SQL queries on encrypted data. The master keys stay with the application and not with SQL Server. This can work on-premises or SQL Server in Azure VM. So think about the hybrid scenarios where you wanted to take advantage of Azure cloud computing, but for certain data could not take advantage of cloud scale due to data security requirements. This technology ensures your data is always encrypted. Best of all no application changes are required. Columnar key Security © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Row-level security SQL Server 2016 SQL Database

The need for row-level security
Protect data privacy by ensuring the right access across rows Customer 1 Customer 2 Customer 3 Fine-grained access control over specific rows in a database table Help prevent unauthorized access when multiple users share the same tables, or to implement connection filtering in multitenant applications Administer via SQL Server Management Studio or SQL Server Data Tools Enforcement logic inside the database and schema is bound to the table SQL Database Row-Level Security (RLS) simplifies the design and coding of security in your application. RLS enables you to implement restrictions on data row access. For example ensuring that workers can access only those data rows that are pertinent to their department, or restricting a customer's data access to only the data relevant to their company. The access restriction logic is located in the database tier rather than away from the data in another application tier. The database system applies the access restrictions every time that data access is attempted from any tier. This makes your security system more reliable and robust by reducing the surface area of your security system. Implement RLS by using the CREATE SECURITY POLICY Transact-SQL statement, and predicates created as inline table valued functions. Limitations during the preview: RLS is incompatible with database export using Data Tier Application Framework (DACFx). You must drop all RLS policies before exporting. Security policies cannot target views. Certain query patterns using OR logic can trigger un-optimized table scans, decreasing query performance. No syntax highlighting in SQL Server tools. Status: public preview Security

Benefits of row-level security (RLS)
11/13/2018 Benefits of row-level security (RLS) Fine-grained access control Keeping multitenant databases secure by limiting access by other users who share the same tables Application transparency RLS works transparently at query time, no app changes needed Compatible with RLS in other leading products Centralized security logic Enforcement logic resides inside database and is schema-bound to the table it protects providing greater security. Reduced application maintenance and complexity Source: Row level security introduces predicate based access control. It features a flexible, centralized, predicate-based evaluation that can take into consideration metadata (such as labels) or any other criteria the administrator determines as appropriate. The predicate is used as a criterion to determine whether or not the user has the appropriate access to the data based on user attributes. Label based access control can be implemented by using predicate based access control. For more information, see Row-Level Security. Store data intended for many consumers in a single database/table while at the same time restricting row-level read and write access based on users’ execution context. Security © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

RLS concepts Predicate function Security predicate Security policy
Performance? Inline functions get optimized to provide comparable performance to views, as if the logic were directly embedded in the original query statement Predicate function User-defined inline table-valued function (iTVF) implementing security logic Can be arbitrarily complicated, containing joins with other tables Security predicate Binds a predicate function to a particular table, applying it for all queries Two types: filter predicates and blocking predicates (coming soon) Security policy Collection of security predicates for managing security across multiple tables Source: Row-Level Security enables customers to control access to rows in a database table based on the characteristics of the user executing a query (e.g., group membership or execution context). Row-level filtering of data selected from a table is enacted through a security predicate filter defined as an inline table valued function. The function is then invoked and enforced by a security policy. The policy can restrict the rows that may be viewed (a filter predicate), but does not restrict the rows that can be inserted or updated from a table (a blocking predicate). There is no indication to the application that rows have been filtered from the result set; if all rows are filtered, then a null set will be returned. Filter predicates are applied while reading data from the base table, and it affects all get operations: SELECT, DELETE (i.e. user cannot delete rows that are filtered), and UPDATE (i.e. user cannot update rows that are filtered, although it is possible to update rows in such way that they will be subsequently filtered). Blocking predicates are not available in this version of RLS, but equivalent functionality (i.e. user cannot INSERT or UPDATE rows such that they will subsequently be filtered) can be implemented using check constraints or triggers. Filter predicates and security policies have the following behavior: Define a security policy that filters the rows of a table. The application is unaware that any rows have been filtered for SELECT, UPDATE, and DELETE operations, including situations where all the rows have been filtered out. The application can INSERT any rows, regardless of whether or not they will be filtered during any other operation. Define a predicate function that joins with another table and/or invokes a function. The join/function is accessible from the query and works as expected without any additional permission checks. Issue a query against a table that has a security predicate defined but disabled. Any rows that would have been filtered or restricted are not affected. The dbo user, a member of the db_owner role, or the table owner queries against a table that has a security policy defined and enabled. Rows are filtered/restricted as defined by the security policy. Attempts to alter the schema of a table bound by a security policy will result in an error. However, columns not referenced by the filter predicate can be altered. Attempts to add a predicate on a table that already has one defined (regardless of whether it is enabled or disabled) results in an error. Attempts to modify a function used as a predicate on a table within a security policy results in an error. Defining multiple active security policies that contain non-overlapping predicates, succeeds. CREATE SECURITY POLICY mySecurityPolicy ADD FILTER PREDICATE dbo.fn_securitypredicate(wing, startTime, endTime) ON dbo.patients Security

Example CREATE FUNCTION int) RETURNS TABLE WITH SCHEMABINDING AS return SELECT 1 as [fn_securitypredicate_result] FROM StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId) WHERE e.UserSID = SUSER_SID() = d.Wing; CREATE SECURITY POLICY dbo.SecPol ADD FILTER PREDICATE dbo.fn_securitypredicate(Wing) ON Patients WITH (STATE = ON) Fine-grained access control over rows in a table based on one or more pre-defined filtering criteria, such as user’s role or clearance level in organization Concepts: Predicate function Security policy Source: Security

RLS in three steps Database Security Policy Application Patients
Microsoft Research 2013 11/13/2018 8:57 PM RLS in three steps Two App user (e.g., nurse) selects from Patients table Three Security Policy transparently rewrites query to apply filter predicate Database Nurse Policy Manager Filter Predicate: INNER JOIN… Security Policy Application Patients CREATE FUNCTION int) RETURNS TABLE WITH SCHEMABINDING AS return SELECT 1 as [fn_securitypredicate_result] FROM StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId) WHERE e.UserSID = SUSER_SID() = d.Wing; CREATE SECURITY POLICY dbo.SecPol ADD FILTER PREDICATE dbo.fn_securitypredicate(Wing) ON Patients WITH (STATE = ON) Policy manager creates a filter predicate and security policy App user (e.g., a nurse) selects from the Patients table The query is transparently rewritten to apply the filter predicate Notice: No app changes! SELECT * FROM Patients SEMIJOIN APPLY dbo.fn_securitypredicate(patients.Wing); SELECT * FROM Patients SELECT Patients.* FROM Patients, StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId) WHERE e.UserSID = SUSER_SID() AND Patients.wing = d.Wing; Security © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Create a security policy
-- The following syntax creates a security policy with a filter predicate for the Customer table, and leaves the security policy disabled CREATE SECURITY POLICY [FederatedSecurityPolicy] ADD FILTER PREDICATE [rls].[fn_securitypredicate]([CustomerId]) ON [dbo].[Customer]; -- Create a new schema and predicate function, which will use the application user ID stored in CONTEXT_INFO to filter rows. CREATE FUNCTION rls.fn_securitypredicate int) RETURNS TABLE WITH SCHEMABINDING AS RETURN ( SELECT 1 AS fn_securitypredicate_result WHERE DATABASE_PRINCIPAL_ID() = DATABASE_PRINCIPAL_ID('dbo') -- application context AND CONTEXT_INFO() = GO Creates a security policy for row-level security The following examples demonstrate the use of the CREATE SECURITY POLICY syntax For an example of a complete security policy scenario, see Row-Level Security Source: Row-level filtering of data selected from a table is enacted through a security predicate filter defined as an inline table valued function. The function is then invoked and enforced by a security policy. The policy can restrict the rows that may be viewed (a filter predicate), but does not restrict the rows that can be inserted or updated from a table (a blocking predicate). There is no indication to the application that rows have been filtered from the result set; if all rows are filtered, then a null set will be returned. Filter predicates are applied while reading data from the base table, and it affects all get operations: SELECT, DELETE (i.e. user cannot delete rows that are filtered), and UPDATE (i.e. user cannot update rows that are filtered, although it is possible to update rows in such way that they will be subsequently filtered). Blocking predicates are not available in this version of RLS, but equivalent functionality (i.e. user cannot INSERT or UPDATE rows such that they will subsequently be filtered) can be implemented using check constraints or triggers. Filter predicates and security policies have the following behavior: Define a security policy that filters the rows of a table. The application is unaware that any rows have been filtered for SELECT, UPDATE, and DELETE operations, including situations where all the rows have been filtered out. The application can INSERT any rows, regardless of whether or not they will be filtered during any other operation. Define a predicate function that joins with another table and/or invokes a function. The join/function is accessible from the query and works as expected without any additional permission checks. Issue a query against a table that has a security predicate defined but disabled. Any rows that would have been filtered or restricted are not affected. The dbo user, a member of the db_owner role, or the table owner queries against a table that has a security policy defined and enabled. Rows are filtered/restricted as defined by the security policy. Attempts to alter the schema of a table bound by a security policy will result in an error. However, columns not referenced by the filter predicate can be altered. Attempts to add a predicate on a table that already has one defined (regardless of whether it is enabled or disabled) results in an error. Attempts to modify a function used as a predicate on a table within a security policy results in an error. Defining multiple active security policies that contain non-overlapping predicates, succeeds. Source: Security

Common RLS use cases Traditional RLS workloads Multitenant databases
Custom business logic to determine which rows each user can SELECT, INSERT, UPDATE, and DELETE based on their role, department, and security level Target sectors: Finance, insurance, healthcare, energy, and government Multitenant databases Ensuring tenants can only access their own rows of data in a shared database, with enforcement logic in the database rather than in the app tier For example: multitenant shards with elastic database tools in SQL Database Reporting, analytics, and data warehousing Different users access same database through various reporting tools, and work with different subsets of data based on their identity/role Security

Summary: RLS Capability Benefits
Row-level security provides fine-grained access control over rows in a table based on conditions you set up Benefits Store data for many users in the same databases and tables while limiting access by other users who share the same tables Security

Dynamic data masking SQL Server 2016 SQL Database

Dynamic data masking Prevent the abuse of sensitive data by hiding it from users Table.CreditCardNo Configuration made easy in the new Azure portal Policy-driven at the table and column level, for a defined set of users Data masking applied in real-time to query results based on policy Multiple masking functions available (e.g. full, partial) for various sensitive data categories (credit card numbers, SSN, etc.) Real-time data masking; partial masking SQL Database SQL Server 2016 Source: Dynamic data masking limits sensitive data exposure by masking it to non-privileged users. Dynamic data masking helps prevent unauthorized access to sensitive data by enabling customers to designate how much of the sensitive data to reveal with minimal impact on the application layer. It’s a policy-based security feature that hides the sensitive data in the result set of a query over designated database fields, while the data in the database is not changed. Dynamic data masking is easy to use with existing applications, since masking rules are applied in the query results, and there is no need to modify existing queries. For example, a call center support person may identify callers by several digits of their social security number or credit card number, but those data items should not be fully exposed to the support person. A developer can define a masking rule to be applied to each query result that masks all but the last four digits of any social security number or credit card number in the result set. For another example, by using the appropriate data mask to protect personally identifiable information (PII) data, a developer can query production environments for troubleshooting purposes without violating compliance regulations. Dynamic data masking limits the exposure of sensitive data and prevents accidental viewing by engineers that access directly databases for troubleshooting purposes or non-privileged application users. Dynamic data masking doesn’t aim to prevent privileged database users from connecting directly to the database and running exhaustive queries that expose pieces of the sensitive data. Dynamic data masking is complimentary to other SQL Server security features (auditing, encryption, row level security…) and it is highly recommended to enable them in addition in order to protect better the sensitive data in the database. Since data is masked just before being returned to the user, changing the data type to an unmasked type will return unmasked data. Dynamic data masking is available in SQL Server However, to enable dynamic data masking, you must use trace flags 209 and 219. For SQL Database, see Get started with SQL Database Dynamic Data Masking (Azure Preview portal). Security

Benefits of dynamic data masking
11/13/2018 Benefits of dynamic data masking Regulatory compliance A strong demand for applications to meet privacy standards recommended by regulating authorities Sensitive data protection Protects against unauthorized access to sensitive data in the application, and against exposure to developers or DBAs who need access to the production database Agility and transparency Data is masked on the fly, with underlying data in the database remaining intact. Transparent to the application and applied according to user privilege Source: Row-level security introduces predicate-based access control. It features a flexible, centralized, predicate-based evaluation that can take into consideration metadata (such as labels) or any other criteria the administrator determines as appropriate. The predicate is used to determine whether or not the user has the appropriate access to the data based on user attributes. Label-based access control can be implemented by using predicate-based access control. For more information, see Row-Level Security. Limit access to sensitive data by defining policies to obfuscate specific database fields, without affecting the integrity of the database. Security © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11/13/2018 How it works Table.CreditCardNo Limit sensitive data exposure by obfuscating it to non-privileged users On-the-fly masking of sensitive data in query results On-the-fly obfuscation of data in query results Policy-driven on the table and column Multiple masking functions available for various sensitive data categories Flexibility to define a set of privileged logins for un-masked data access By default, database owner is unmasked See: Dynamic masking Azure DB Configurable service, which limit sensitive data exposure by obfuscation in query results Ability to mask data based on its table and column or alias Ability to select masking functions Ability to define a list of privileged logins that will be able to the data in its CLER format Security © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Dynamic data masking walkthrough
Microsoft Research 2013 11/13/2018 8:57 PM Dynamic data masking walkthrough 2) Application user selects from Employee table 3) Dynamic data masking policy obfuscates the sensitive data in the query results Security officer defines dynamic data masking policy in T-SQL over sensitive data in Employee table Security Officer ALTER TABLE [Employee] ALTER COLUMN [SocialSecurityNumber] ADD MASKED WITH (FUNCTION = ‘SSN()’) ALTER TABLE [Employee] ALTER COLUMN [ ] ADD MASKED WITH (FUNCTION = ‘ ()’) ALTER TABLE [Employee] ALTER COLUMN [Salary] ADD MASKED WITH (FUNCTION = ‘RANDOM(1,20000)’) GRANT UNMASK to admin1 admin1 login other login SELECT [Name], [SocialSecurityNumber], [ ], [Salary] FROM [Employee] Security © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary: dynamic data masking
Capability Protects against unauthorized disclosure of sensitive data in the application Benefits Enables you to set up policies at the table and column level that provide multiple masking functions Allows certain privileged logins to see the data unmasked Security

Enhanced AlwaysOn

The need for mission-critical availability
Reliable Detects failures reliably Able to handle multiple failures Unified, simplified solution Easy to deploy, manage, and monitor Integrated Flexible Reuse existing investments SAN/DAS environments Able to use HA hardware resources Fast, transparent failover Efficient Availability

Enhanced AlwaysOn Availability Groups
Greater scalability Load balancing readable secondaries Increased number of automatic failover targets Log transport performance Improved manageability DTC support Database-level health monitoring Group Managed Service Account Domain-independent Availability Groups Unified HA solution Hong Kong (Secondary) AG AG_Listener New Jersey (Secondary) AG AG Asynchronous data Movement Source: Load-balancing of read-intent connection requests is now supported across a set of read-only replicas. The previous behavior always directed connections to the first available read-only replica in the routing list. For more information, see Configure load-balancing across read-only replicas. The number of replicas that support automatic failover has been increased from two to three. Group Managed Service Accounts are now supported for AlwaysOn Failover Clusters. For more information, see Group Managed Service Accounts. For Windows Server 2012 R2, an update is required to avoid temporary downtime after a password change. To obtain the update, see gMSA-based services can't log on after a password change in a Windows Server 2012 R2 domain. AlwaysOn Availability Groups supports distributed transactions and the DTC on Windows Server For more information, see SQL Server 2016 Support for DTC and AlwaysOn Availablity Groups. You can now configure AlwaysOn Availability Groups to failover when a database goes offline. This change requires setting the DB_FAILOVER option to ON in the CREATE AVAILABILITY GROUP (Transact-SQL) or ALTER AVAILABILITY GROUP (Transact-SQL) statements. Synchronous data Movement New York (Primary) Availability

Load balancing in readable secondaries
Computer5 DR site Computer2 Computer3 Computer4 Primary site Computer1 (Primary) READ_ONLY_ROUTING_LIST= (('COMPUTER2', 'COMPUTER3', 'COMPUTER4'), 'COMPUTER5') In SQL 2014, read-only transactions routed by the Listener went to the first secondary that was available Read-only Routing (ROR) lists Now you can configure the ROR lists to round-robin among a specific set of secondaries (for each primary) Availability

More than two automatic failover targets
11/13/2018 More than two automatic failover targets Increased solution scale Increased resiliency Now any sync secondary can be a target for automatic failover Total of three automatic failover targets Availability © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Log transport performance
New hardware pushes boundaries Very high transaction rates have caused problems when coupled with fast hardware (i.e. PCIe flash storage) The entire pipeline has been reworked end to end Result is much lower CPU consumption, and much better performance Performance bar: comparison to the performance of a standalone server Availability

Distributed Transaction Coordinator (DTC) support
Currently, any distributed transactions touching a database in an availability group are not allowed Many customers run unsupported, at risk to their data and reputation Many enterprise applications need cross-database transactions Fully supported in SQL Server 2016 Joint effort with Windows Requires specific patch in order to work cleanly Other requirements: Availability groups must be running on Windows Server 2016 Technical Preview 2 Availability groups must be created with the CREATE AVAILABILITY GROUP command and the WITH DTC_SUPPORT = PER_DB clause. You cannot currently alter an existing availability group Learn more: Availability

Database-level failover trigger
Currently, Availability Groups only monitor the health of the instance A database can be offline or corrupt, but will not trigger a failover as long as the instance itself is healthy SQL Server 2016: option to also monitor the health of the databases in the Availability Group Databases going offline trigger a change in the health status Availability

gMSA support Group Managed Service Accounts (gMSA)
Automatically set domain scope for Managed Service Accounts Automatic password rotation Much more secure than regular domain accounts Enables cross-system security context Why would I want a gMSA? No need to manually change passwords on all AlwaysOn instances How does it work? Passwords are managed by the domain What versions will it be supported in? Supported in SQL Server 2014 and SQL Server 2016 Availability

Domain-independent Availability Groups
11/13/2018 Domain-independent Availability Groups New feature in Windows Server 2016 Environments supported: Cross domains (with trust) Cross domains (no trust) No domain at all Cluster management via PowerShell only SQL management as normal Uses certificate-secured endpoints like DBM Availability © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary: enhanced AlwaysOn
Capability For scalability, SQL Server 2016 adds in load balancing of readable secondaries Increases the number of auto-failover targets from two to three Benefits Log transport performance has been improved Support for Distributed Transaction Coordinator (DTC) — enrolled transactions for Availability Group databases Database-level health monitoring gMSA: domain-level accounts that are automatically managed Availability

Scalability improvements

Enhanced database caching
Supports caching data with automatic, multiple TempDB files per instance in multicore environments Reduces metadata and allocation contention for TempDB workloads, improving performance and scalability Source: Multiple TempDB Databases Setup adds multiple tempdb data files during the installation of a new instance. Below are the pertinent details to consider: By default, setup adds as many tempdb files as the CPU count or 8, whichever is lower. You can configure the number of tempdb database files using the new command line parameter- /SQLTEMPDBFILECOUNT. It can be used for unattended and interactive installations. setup.exe /Q /ACTION="INSTALL" /IACCEPTSQLSERVERLICENSETERMS /FEATURES="SqlEngine" /INSTANCENAME="SQL15" .. /SQLTEMPDBDIR="D:\tempdb" /SQLTEMPDBFILECOUNT="4" SQLTEMPDBFILECOUNT only accepts an integer value. If the parameter is not given or the value <= 0, setup will use the default value that is the number of (logical) cores on the machine or 8, whichever is lower. If the value is greater than the maximum allowed (cpu count), the installation will fail with an error. Important The SQLTEMPDBFILECOUNT parameter is supported only in the following scenarios or actions: Install, InstallFailoverCluster, CompleteImage (sysprep), CompleteFailoverCluster (sysprep), and RebuildDatabase. You can configure the number of tempdb database files using the new UI input control on the Database Engine Configuration section. The primary database file for tempdb will still be tempdb.mdf. The additional tempdb files are named as tempdb_mssql_#.ndf where # where # represents a unique number for each additional tempdb database file created during setup. The purpose of this naming convention is to make them unique. Uninstalling an instance of SQL Server deletes the files with naming convention tempdb_mssql_#.ndf. Do not use tempdb_mssql_*.ndf naming convention for user database files. . RebuildDatabase scenario deletes system databases and installs them again in clean state. Because the setting of tempdb file count does not persist, the value of number of tempdb files is not known during setup. Therefore, RebuildDatabase scenario does not know the count of tempdb files to be re-added. You can provide the value of the number of tempdb files again with the SQLTEMPDBFILECOUNT parameter. If the parameter is not provided, RebuildDatabase will add a default number of tempdb files, which is as many tempdb files as the CPU count or 8, whichever is lower. Scalability

Enhanced support for Windows Server
Hardware acceleration for TDE encryption/decryption Implements the next generation of Microsoft cryptography Takes advantage of specialized microprocessor instructions Can improve performance 3x to 10x Parallelizing the built-in decryption function to improve read performance Decryption now supported as parallelizable (used to be sequential only) Results in dramatically better response times for queries with encrypted data columns Source: Transparent Data Encryption Transparent Data Encryption has been enhanced with support for Intel AES-NI hardware acceleration of encryption. This will reduce the CPU overhead of turning on Transparent Data Encryption. Scalability

Cloud Platform System advantages
Partnership with Dell Combination of optimized hardware/software Designed specifically to reduce the complexity and risk of implementing a self-service cloud CPS can go from delivery to live within days—not months—and lets service providers and enterprises move up the stack to focus on delivering services to users Switches Networking Compute Rack 2 Edge Storage Rack 1 Management Rack 3 Rack 4 Scalability

Deeper insights across data
Access any data Scale and manage Powerful insights Advanced analytics PolyBase Insights from data across SQL Server and Hadoop with the simplicity of T-SQL Enhanced SSIS Designer support for previous SSIS versions Enterprise-grade Analysis Services Enhanced performance and scalability for Analysis Services Single SSDT in Visual Studio 2015 Build richer analytics solutions as part of your development projects in Visual Studio Enhanced MDS Excel add-in 15x faster; more granular security roles; archival options for transaction logs; and reuse entities across models Mobile BI Business insights for your on- premises data through rich visualization on mobile devices with native apps for Windows, iOS, and Android Enhanced Reporting Services New modern reports with rich visualizations R integration Bringing predictive analytic capabilities to your relational database Expand your “R” script library with Microsoft Azure Marketplace Access Any Data Query relational and non-relational data with the simplicity of T-SQL with PolyBase. Manage document data with native JSON support. Scale and Manage Enhanced performance, scalability and usability across SQL Server Enterprise Information Management tools and Analysis Services. Enhanced MDS. For domain based attribute, optionally, user can select a parent attribute whose value will constrain the allowed values for this attribute. For example 1. Model has State, City, Account entity. 2. Account has a City DBA to City entity and State DBA to State entity. 3. City has a State DBA to State entity and a derived hierarchy from State to City. 4. A constraint can be added on Account.City attribute which parent is Account.State. So the City attribute dropdown list is constrained by State value. Powerful Insights on any device Business insights through rich visualizations on mobile devices. Native apps for Windows, iOS and Android. New modern reports for all browsers. Advanced Analytics at massive scale Built-in advanced analytics provide the scalability and performance benefits of running your “R” algorithms directly in SQL Server. Expand your analytics library with Microsoft Azure Marketplace.

Hyperscale cloud Hybrid solutions Simplicity Consistency
Stretch Database Stretch operational tables in a secure manner into Azure for cost-effective historic data availability. Works with Always Encrypted and row-level security Power BI with on-premises data New interactive query with Analysis Services. Customer data stays behind your firewall Enhanced backup to Azure Faster restore times and 50% reduction in storage. Supports larger DBs with block blobs and custom backup schedule with local staging Easy migration of on-premises SQL Server Simple point-and-click migration to Azure Suite of advisors for upgrading to SQL Server 2016 SQL Server 2016 Upgrade Advisor in the adoption of new SQL Server features Simplified Add Azure Replica Wizard Automatic listener configuration for AlwaysOn in Azure VMs Common development, management, and identity tools Including Active Directory, Visual Studio, Hyper- V, and System Center Consistent experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS Breakthrough hybrid scenarios Stretch Database technology keeps more of your customers’ historical data at your fingertips by transparently stretching your warm and cold OLTP data to Microsoft Azure on-demand, without application changes. Simplicity New tools that make SQL Server migration to Microsoft Azure and hybrid scenarios even easier. Consistency Consistent experience from on-premises to Microsoft Azure IaaS and PaaS.

SQL Server 2016 Database Engine – New Features -.

Similar presentations

Presentation on theme: "SQL Server 2016 Database Engine – New Features -."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

SQL Server 2016 Database Engine – New Features -.

Similar presentations

Presentation on theme: "SQL Server 2016 Database Engine – New Features -."— Presentation transcript:

Similar presentations

About project

Feedback