1. Boosting DWH-Performance with SQL Server 2016 ColumnStore Index

2. Sponsors Gold Sponsors: Bronze Sponsors: Swag Sponsors:

3. Introduction Markus Ehrenmüller-Jensen Business Intelligence Architect @MEhrenmueller markus.ehrenmueller@gmail.com runtastic Pluskaufstraße 7 4061 Pasching Austria, Europe http://www.runtastic.com SQL Server 2005 - 2014 BI Developer Database Developer Database Admin

4. Agenda Column Store Non-Clustered Column Store Index Clustered Column Store Index What’s new in SQL 2016

6. Evolution of Microsoft Data Platform SQL Server 2000 SQL Server 2005 SQL Server 2008 XML ● KPIs Management Studio ● Mirroring Compression ● Policy-Based Mgmt ● Programmability SQL Server 2012 ColumnStore Index ● AlwaysOn ● Data Quality Services ● Power View ● Cloud Connectivity PowerPivot ● SharePoint Integration ● Master Data Services SQL Server 2008 R2 SQL Server 2014 In-Memory Across Workloads ● Performance & Scale ● Hybrid Cloud Optimized ● HDInsight ● Cloud BI

7. In-Memory in SQL Server Cache Buffer Pool Pin-to-Memory DBCC PINTABLE ColumnStore (Non-)Clustered Index SSAS Tabular Power Pivot In-Memory OLTP Memory Optimized Table Natively Compiled Stored Procedure

9. Ordinary Report http://www.flickr.com/photos/marts-pics/3554153954/sizes/l/in/photostream/

10. ColumnStore Index http://commons.wikimedia.org/wiki/User:Malene

11. DEMO Improve Query Performance with ColumnStore Index

12. How? xVelocity (VertiPaq, PowerPivot, BISM Tabular) Compression Column-Elimination Segment/Rowgroup-Elimination Parallel Read Operations Query Processor in Batch-Mode Typically 10x faster

13. xVelocity Column Store Power Pivot Analysis Services SQL Server Non clustered Index SQL 2012+ Clustered Index SQL 2014+

14. Compression X-Velocity In-Memory Compression Engine Creation takes up to 1.5x longer then b-tree Algorithms Value Scale (1023, 1002  scale 1000; 23, 2) Bit Array (x, y, z, x  100, 010, 001, 100) Binary Bitmap Run-Length (x, x, x, y, z, z  3x, 1y, 2z) Dictionary (x, y, z, x  1, 2, 3, 1) Huffman Lempel-Ziv-Welch Decision after sample (1% / max 1 Mio rows) No estimation available sp_estimate_data_compression_savings not supported for ColumnStore

15. CREATE INDEX (elapsed time) NCCI (22943 ms) CCI (22879 ms) CCI Archival (22067 ms) PAGE (5668 ms) Clustered (4925 ms) ROW (3282 ms)

16. INSERT (elapsed time) PAGE (72946 ms) CCI Archival (54221 ms) CCI (54123 ms) NCCI (42596 ms) ROW (21652 ms) Clustered (14905 ms) Heap (6941 ms)

17. Compression Heap & NCCI (1622 MB) Heap (1382 MB) ROW (794 MB) PAGE (202 MB) CCI (25 MB) Archival (11 MB)

18. Dictionary Primary (global) Across all segments Mandatory Preferable Secondary (local) Per segment Optional sys.column_store_dictionaries

19. ARCHIVAL Compression LZ777 compression on top Additional ~30% of space-savings Can be applied per partition

20. Column-Elimination Only needed column are read As opposite to RowStore Where only whole rows can be read

21. KeyAlternateKeyNameStock 1AR-5381Adjustable Race1000 2BA-8327Bearing Ball1000 3BE-2349Ball Bearing Cage800 4BE-2908Ball Bearing Grease800 5BL-2036Blade800 6CA-5965LL Crankarm500 7CA-6738ML Crankarm500 Page 1 of row store Page 2 of row store Segment for column 1 Segment for column 2 Segment for column 3 Segment for column 4 RowStore vs. ColumnStore Segment for column 1 Segment for column 2 Segment for column 3 Segment for column 4 Rowgroup

22. Segment Part of row group for a single column # of rows per row group is the same for all segments No sort order inside a segment Max. 1,048.576 rows Trimmed segement(s) because of to few rows, DOP or memory pressure The bigger, the better compression The smaller, the better segment-elimination Stored as BLOB (through 8k pages) Directory Allocation status, # of rows, min/max value sys.column_store_segments

23. Segment-Elimination Segment is the smallest unit Reading 1 Mio rows or not Min/max value Alignment.sql

24. Parallel Read Operations

25. Batch-Mode SQL 2012+ Chunks of 1000 rows per batch Better CPU efficiency Vs. Row mode Every row processed after each other

26. Typically 10x faster Heap (4291 ms) Clustered (2151 ms) ROW (1695 ms) NCCI (970 ms) PAGE (938 ms) CCI (140 ms) CCI Archival (99 ms)

27. Use Cases > 1 Mio rows Aggregations, groupings & filters (DWH/OLAP) Write once, read multiple times Less distinctive values per column Sweet spot Design (eg. matching data type, no functions), star & snowflake schema, inner joins Can substitute datamarts/aggregation-tables ROLAP & Tabular Model DirectQuery Query Optimizer includes CS Index Suggested_tables.sql

28. Restrictions (2014) Data types: ntext, text, and image, vardecimal, varchar(max) and nvarchar(max), rowversion (and timestamp), sql_variant, CLR types (hierarchyid and spatial types), xml, uniqueidentifier Page/Row compression Replication Change Tracking, Change Data Capture Filestream Enterprise Edition only

29. Nonclustered ColumnStore Index (NCCI) May combined with other indices Decide which column to include Only one NCCI per table Redundant storage No sort order SQL Server 2012+

30. NCCI Restrictions No constraints allowed (unique) Indexed table is read-only

31. NCCI Best Practice Memory, Memory, Memory Include all columns MAXDOP > 1 Fact-tables and big dimension tables Choose Clustered Index wise Update/Insert Disable/enable index Partitioning View/Union all

32. Clustered ColumnStore Index (CCI) Physical columnar storage (instead of row based) Not really clustered (as stored in unsorted segments) No reduntant storage All columns included automatically No other index allowed UPDATE-able Supports more data types Switching between ROW-mode and BATCH-mode allowed SQL Server 2014+

33. CCI: Structure CREATE CLUSTERED COLUMNSTORE INDEX INSERT DELETE REORGANIZE REBUILD Columnstore Deleted Bitmap Deltastore(s)

34. CCI: Deltastore Ordinary Rowstore Uncompressed heap Does not benifit from batch mode Compression is expensive operation OPEN / CLOSED / INVISIBLE / TUMBSTONE / (COMPRESSED) Tuple-Mover

35. CCI: Tuple Mover Closed Delta Store  new Segment (compressed) Closedown 1,048.576 rows for INSERT 102.400 rows for BULK INSERT Every 5 minutes Will pause 15 sec after it has done its job Invoke REBUILD/REORGANIZE after trickle load Update/Insert will be blocked

36. CCI: Deleted Bitmap Deleted rows of ColumnStore 2 storage format Bitmap (in memory) B-Tree (on disk) Fully deleted segments are not eliminated REBUILD to get rid of those rows

37. CCI: Restrictions (2014) No Constraints (unique, primary, foreign key) No Triggers Not all datatypes supported No ISOLATION LEVEL SNAPSHOT Enterprise Edition only

38. CCI Best Practice Memory, Memory, Memory MAXDOP > 1 Fact-tables and big dimension tables suggested_tables.sql Choose Clustered Index wise Go for BULK INSERTs INSERT is locking whole Row group DELETE is locking whole segment

39. ColumnStore Index Non-clustered SQL Server 2012+ Additional index (redundancy) Read-only Subset of columns Max. one NCCI per table Clustered SQL Server 2014+ Master Update-able All columns No additional index allowed

40. ColumnStore vs. In-Memory OLTP DWH vs. OLTP Aggregation vs. Single Rows Disk vs. Memory Compressed vs. uncompressed

41. SQL Server v2016 UPDATE-able & Filtered NCCI Deleted Buffer  deleted Bitmap Non-Clustered Index (B-Tree) on CCI Mapping index in between In-Memory ColumnStore Index Less restrictions Primary key, foreign key, all isolation levels, batch mode support, inline-definition, …

42. RowStore Index on CCI Mapping Index B-tree Tracks movements of rows inside of ColumnStore Row Locator SegmentID:TupleID (vs. FileID:PageID:SlotID)

43. Wrap up Column Store Non-Clustered Column Store Index Clustered Column Store Index What’s new in SQL 2016

44. Call to Action Try Non-Clustered Column Store Index Clustered Column Store Index

45. Sponsors Gold Sponsors: Bronze Sponsors: Swag Sponsors:

46. Questions? Markus Ehrenmüller-Jensen Business Intelligence Architect @MEhrenmueller markus.ehrenmueller@gmail.com runtastic Pluskaufstraße 7 4061 Pasching Austria, Europe http://www.runtastic.com SQL Server 2005 - 2014 BI Developer Database Developer Database Admin