1. Ascending Key Problem in SQL Server Large Tables
Milos Radivojevic,
Principal Database Consultant, bwin.party

2. Our Main Sponsors:

3. Say Thank you to Volunteers:
They spend their FREE time to give you this event.
Because they are crazy. 
Because they want YOU to learn from the BEST IN THE WORLD.

4. Paulo Matos:

5. Pedro Simões:

6. André Batista:

7. Paulo Borges:

8. André Melancia:

9. Murilo Miranda:

10. Quilson Antunes:

11. 5 Sponsor Sessions at 15:10
Don’t miss them, they might be getting distributing some awesome prizes!
Rumos
BI4ALL
Bold Int
CozyRoc
Pythian

12. Important Activities:
WIT – Women in Technology
15:10 at BizSpark Room (Ground Floor)
SQLClinic Challenges
10:00 DEV (Neil Hambly)
11:50 DBA (Uwe Ricken)
17:00 BI (Steph Locke)

13. About Me Milos Radivojevic SQL Server MVP
Principal Database Consultant, bwin.party Vienna, Austria
MCT, MCITP, MCTS, MCP
Co-Founder: SQL Pass Austria
Conference Speaker: SQLBits, SQL Saturday, SQL Pass Austria
Contact: E W
@MilosSQL

14. Agenda SQL Server and Statistics Key Ascending Problem
Solutions in SQL Server 2012
Solutions in SQL Server 2014
14 |
12/31/201812/31/2018 |
Footer Goes Here

15. Sample Table Sample rows... Size: 1M rows
CREATE TABLE dbo.Orders(
id INT IDENTITY(1,1) NOT NULL,
custid INT NOT NULL,
orderdate DATETIME NOT NULL,
amount MONEY NOT NULL,
CONSTRAINT PK_Orders PRIMARY KEY CLUSTERED (id ASC) );
Sample rows...
CREATE INDEX IXDate ON dbo.Orders(orderdate);
Size: 1M rows
Order dates from 2000 to 2013

16. SQL Server and Statistics
DBCC SHOW_STATISTICS ('dbo.Orders','IXDate');
STATS HEADER
DENSITY VECTOR
STATS HISTOGRAM

17. SQL Server and Statistics
SELECT * FROM dbo.Orders WHERE orderdate = ' ';
Estimation = EQ_ROWS

18. Statistics SELECT * FROM dbo.Orders WHERE orderdate = '20141229';
Estimation = AVG_RANGE_ROWS

19. Statistics DECLARE @d AS DATETIME = '20141230'
SELECT * FROM dbo.Orders WHERE orderdate
Estimation = All density * Rows

20. Statistics SELECT * FROM dbo.Orders WHERE orderdate = '20150101';
Estimation = 1

21. Statistics SELECT * FROM dbo.Orders WHERE orderdate > '20150101';
Estimation = 1

22. Est.Number of Rows - Calculation
Literal equal to an entry from Statistics Histogram => EQ_ROWS
Literal not equal to an entry from Statistics Histogram, but within a histogram => AVG_RANGE_ROWS
Literal wrapped in a variable => All density * Rows
Beyond Statistics Histogram => 1

23. Key Ascending Problem - Demo
Create the same table from previous example with indexes on custid and orderdate
Populate it with 10M rows with orderdate in the range (statistics are up-to-date)
Add additionl 1,5M rows with orderdate from 2014 (statistics are out-of-date, 15% of newly added rows is not enough for automatically stats update)
D E M O

24. Simple Query Get all orders from 2014 for a given customer
SELECT * FROM dbo.Orders
WHERE custid = 160
AND orderdate >= ' ' AND orderdate < ' ';

25. SQL Server 2012

26. SQL Server 2014

27. SQL Server 2012 – Estimation Details
Estimation for custid = 160
Estimation for orderdate in year 2014

28. SQL Server 2014 – Estimation Details
Estimation for custid = 160
SQL 2012: 98
SQL 2014: 113
SQL 2012: 1
SQL 2014:
Estimation for orderdate in year 2014

29. Key Ascending Problem Monotonically increasing values in columns
newly added rows have value greater than values in all existing rows
newly added rows are beyond the range covered by the statistics histogram
SQL Server does not know for recent rows
This can result in bad cardinality estimates and slow query performance
Most recent rows are most frequently accessed

30. Key Ascending Problem – Solutions in SQL Server 2012
Update Statistics
Use TF 2371 to change the default threshold for automatically update
Manually and frequently statistics update (not waiting for 20% of changes)
Trace Flags
2390 (for Leading Column Type: Unknown and Ascending)
2389 (for Leading Column Type: Ascending)
4139 (for all three Leading Column Types)

31. Ascending Columns – Leading Column Type
DBCC TRACEON(2388);
DBCC SHOW_STATISTICS(Orders, IXDate);
DBCC TRACEOFF(2388);
Leading Column Type:
Unknown
Ascending
Stationary

32. Ascending Columns – Leading Column Type
Simple Distribution in one of my databases:
Unknown
Ascending
Stationary

33. Ascending Columns – Leading Column Type
More Info
Article by Fabiano Amorim

34. TF 2390 – Creating Statistics On The Fly
Calculates Estimation on the fly
Requires an index on the column
Statistics is not persisted
Calculates estimations for each column involved in filter expressions if their Leading column Type is Unknown or Ascending
TF 2389
The same as with 2390 but only for Ascending
TF 4139
Includes Stationary columns too (from SQL Server 2012 SP1 CU10 or SP2 CU1)

35. TF 2390 – Creating Statistics On The Fly
Search for max value for all columns involved in filter expressions
Temporary changes statistics objects on the fly
Rollback changes and let statistics objects unchanged

36. TF 2390 – Creating Statistics On The Fly

37. Key Ascending Problem – Solutions in SQL Server 2012
More Info:

38. Key Ascending Problem – Solutions in SQL Server 2014
New CE estimates differently values beyond Statistics Histogram
Old CE estimates always 1 row
New CE estimates significantly more
Trace Flags 2389, 2390 and 4139 are not supported in SQL Server 2014

39. Key Ascending Problem – Solutions in SQL Server 2014
Due to different estimations made by new CE the initial query performs well in SQL Server 2014
New selectivity for the orderdate predicate is far away from the selectivity for the custid and therefore new QO has chosen the index on the custid as we initially expected

40. Key Ascending Problem - Another Query Pattern
SELECT * FROM dbo.OrderCIDetails WHERE orderid BETWEEN AND ORDER BY productid;
SELECT * FROM dbo.OrderCIDetails WHERE orderid BETWEEN AND ORDER BY productid OPTION (QUERYTRACEON 2312, RECOMPILE);

41. Key Ascending Problem - Another Query Pattern (Solution)
SELECT * FROM dbo.OrderCIDetails WHERE orderid BETWEEN AND ORDER BY productid OPTION (QUERYTRACEON 2390, RECOMPILE);
This solution is possible in SQL Server 2012, but NOT in SQL Server 2014 – TF 2389 and 2390 don’t exist anymore!

42. Key Ascending Problem - Another Query Pattern (Solution)
Workaround for SQL Server 2014
AS TABLE(id INT);
INSERT
SELECT n FROM dbo.GetNums(100);
SELECT * FROM dbo.OrderCIDetails
INNER t ON orderid=t.id
ORDER BY productid;

43. SQL Server 2014 – Problem with KeyAsc
New Cardinality Estimator behaves differently beyond Statistics Histogram
(not documented)
It does not respect literals!!!
i.e. orderdate BETWEEN AND has the same estimation as orderdate BETWEEN AND
Trace Flags 2389, 2390 don‘t work with new CE

44. SQL Server 2014 KeyAsc Problem – Connect Item
SQL Server 2014 New Cardinality Estimator and Ascending Key Columns

45. Summary General Recommendation SQL Server 2012 SQL Server 2014
Update regularly statitsics on the most important ascending columns
SQL Server 2012
Use TF 2390 (or 2389 or 4139)
SQL Server 2014
TFs not supported
For some query patterns new CE performs better (TFs not needed)
For some other query patterns performance regression => vote for Connect Item