Ascending Key Problem in SQL Server Large Tables Milos Radivojevic, Principal Database Consultant, bwin.party

Our Main Sponsors:

Say Thank you to Volunteers: They spend their FREE time to give you this event. Because they are crazy.  Because they want YOU to learn from the BEST IN THE WORLD.

Paulo Matos:

Pedro Simões:

André Batista:

Paulo Borges:

André Melancia:

Murilo Miranda:

Quilson Antunes:

5 Sponsor Sessions at 15:10 Don’t miss them, they might be getting distributing some awesome prizes! Rumos BI4ALL Bold Int CozyRoc Pythian

Important Activities: WIT – Women in Technology 15:10 at BizSpark Room (Ground Floor) SQLClinic Challenges 10:00 DEV (Neil Hambly) 11:50 DBA (Uwe Ricken) 17:00 BI (Steph Locke)

About Me Milos Radivojevic SQL Server MVP Principal Database Consultant, bwin.party Vienna, Austria MCT, MCITP, MCTS, MCP Co-Founder: SQL Pass Austria Conference Speaker: SQLBits, SQL Saturday, SQL Pass Austria Contact: E Milos.Radivojevic@bwinparty.com W www.bwinparty.com @MilosSQL http://milossql.wordpress.com

Agenda SQL Server and Statistics Key Ascending Problem Solutions in SQL Server 2012 Solutions in SQL Server 2014 14 | 12/31/201812/31/2018 | Footer Goes Here

Sample Table Sample rows... Size: 1M rows CREATE TABLE dbo.Orders( id INT IDENTITY(1,1) NOT NULL, custid INT NOT NULL, orderdate DATETIME NOT NULL, amount MONEY NOT NULL, CONSTRAINT PK_Orders PRIMARY KEY CLUSTERED (id ASC) ); Sample rows... CREATE INDEX IXDate ON dbo.Orders(orderdate); Size: 1M rows Order dates from 2000 to 2013

SQL Server and Statistics DBCC SHOW_STATISTICS ('dbo.Orders','IXDate'); STATS HEADER DENSITY VECTOR STATS HISTOGRAM

SQL Server and Statistics SELECT * FROM dbo.Orders WHERE orderdate = '20141230'; Estimation = EQ_ROWS

Statistics SELECT * FROM dbo.Orders WHERE orderdate = '20141229'; Estimation = AVG_RANGE_ROWS

Statistics DECLARE @d AS DATETIME = '20141230' SELECT * FROM dbo.Orders WHERE orderdate = @d; Estimation = All density * Rows

Statistics SELECT * FROM dbo.Orders WHERE orderdate = '20150101'; Estimation = 1

Statistics SELECT * FROM dbo.Orders WHERE orderdate > '20150101'; Estimation = 1

Est.Number of Rows - Calculation Literal equal to an entry from Statistics Histogram => EQ_ROWS Literal not equal to an entry from Statistics Histogram, but within a histogram => AVG_RANGE_ROWS Literal wrapped in a variable => All density * Rows Beyond Statistics Histogram => 1

Key Ascending Problem - Demo Create the same table from previous example with indexes on custid and orderdate Populate it with 10M rows with orderdate in the range 2000- 2013 (statistics are up-to-date) Add additionl 1,5M rows with orderdate from 2014 (statistics are out-of-date, 15% of newly added rows is not enough for automatically stats update) D E M O

Simple Query Get all orders from 2014 for a given customer SELECT * FROM dbo.Orders WHERE custid = 160 AND orderdate >= '20140101' AND orderdate < '20150101';

SQL Server 2012

SQL Server 2014

SQL Server 2012 – Estimation Details Estimation for custid = 160 Estimation for orderdate in year 2014

SQL Server 2014 – Estimation Details Estimation for custid = 160 SQL 2012: 98 SQL 2014: 113 SQL 2012: 1 SQL 2014: 135.000 Estimation for orderdate in year 2014

Key Ascending Problem Monotonically increasing values in columns newly added rows have value greater than values in all existing rows newly added rows are beyond the range covered by the statistics histogram SQL Server does not know for recent rows This can result in bad cardinality estimates and slow query performance Most recent rows are most frequently accessed

Key Ascending Problem – Solutions in SQL Server 2012 Update Statistics Use TF 2371 to change the default threshold for automatically update Manually and frequently statistics update (not waiting for 20% of changes) Trace Flags 2390 (for Leading Column Type: Unknown and Ascending) 2389 (for Leading Column Type: Ascending) 4139 (for all three Leading Column Types)

Ascending Columns – Leading Column Type DBCC TRACEON(2388); DBCC SHOW_STATISTICS(Orders, IXDate); DBCC TRACEOFF(2388); Leading Column Type: Unknown Ascending Stationary

Ascending Columns – Leading Column Type Simple Distribution in one of my databases: Unknown Ascending Stationary

Ascending Columns – Leading Column Type More Info Article by Fabiano Amorim https://www.simple-talk.com/sql/database-administration/statistics-on-ascending-columns/

TF 2390 – Creating Statistics On The Fly Calculates Estimation on the fly Requires an index on the column Statistics is not persisted Calculates estimations for each column involved in filter expressions if their Leading column Type is Unknown or Ascending TF 2389 The same as with 2390 but only for Ascending TF 4139 Includes Stationary columns too (from SQL Server 2012 SP1 CU10 or SP2 CU1)

TF 2390 – Creating Statistics On The Fly Search for max value for all columns involved in filter expressions Temporary changes statistics objects on the fly Rollback changes and let statistics objects unchanged

TF 2390 – Creating Statistics On The Fly

Key Ascending Problem – Solutions in SQL Server 2012 More Info: https://milossql.wordpress.com/2014/11/24/beyond-statistics-histogram-part-2-tf-2390/ https://milossql.wordpress.com/2014/12/22/beyond-statistics-histogram-part-3-tf-2389-feat-tf-4139/

Key Ascending Problem – Solutions in SQL Server 2014 New CE estimates differently values beyond Statistics Histogram Old CE estimates always 1 row New CE estimates significantly more Trace Flags 2389, 2390 and 4139 are not supported in SQL Server 2014

Key Ascending Problem – Solutions in SQL Server 2014 Due to different estimations made by new CE the initial query performs well in SQL Server 2014 New selectivity for the orderdate predicate is far away from the selectivity for the custid and therefore new QO has chosen the index on the custid as we initially expected

Key Ascending Problem - Another Query Pattern SELECT * FROM dbo.OrderCIDetails WHERE orderid BETWEEN 10999900 AND 11000000 ORDER BY productid; SELECT * FROM dbo.OrderCIDetails WHERE orderid BETWEEN 10999900 AND 11000000 ORDER BY productid OPTION (QUERYTRACEON 2312, RECOMPILE);

Key Ascending Problem - Another Query Pattern (Solution) SELECT * FROM dbo.OrderCIDetails WHERE orderid BETWEEN 10999900 AND 11000000 ORDER BY productid OPTION (QUERYTRACEON 2390, RECOMPILE); This solution is possible in SQL Server 2012, but NOT in SQL Server 2014 – TF 2389 and 2390 don’t exist anymore!

Key Ascending Problem - Another Query Pattern (Solution) Workaround for SQL Server 2014 DECLARE @T AS TABLE(id INT); INSERT INTO @T(id) SELECT 10999900 + n FROM dbo.GetNums(100); SELECT * FROM dbo.OrderCIDetails INNER JOIN @t t ON orderid=t.id ORDER BY productid;

SQL Server 2014 – Problem with KeyAsc New Cardinality Estimator behaves differently beyond Statistics Histogram (not documented) It does not respect literals!!! i.e. orderdate BETWEEN 100000 AND 100001 has the same estimation as orderdate BETWEEN 100000 AND 900000000 Trace Flags 2389, 2390 don‘t work with new CE

SQL Server 2014 KeyAsc Problem – Connect Item SQL Server 2014 New Cardinality Estimator and Ascending Key Columns https://connect.microsoft.com/SQLServer/feedback/details/870089/sql-server-2014-new-cardinality-estimator-and-ascending-key-columns

Summary General Recommendation SQL Server 2012 SQL Server 2014 Update regularly statitsics on the most important ascending columns SQL Server 2012 Use TF 2390 (or 2389 or 4139) SQL Server 2014 TFs not supported For some query patterns new CE performs better (TFs not needed) For some other query patterns performance regression => vote for Connect Item