Columnstore Indexing: From SQL Server 2012 to SQL Server 2014

Columnstore Indexing: From SQL Server 2012 to SQL Server 2014
Presented by: Sifiso Ndlovu

A Tale of Two Queries TSQL Script Without Columnstore Index
TSQL Script That Uses Columnstore Index

Speaker BIO Sifiso W Ndlovu Senior BI Tech at Karabina
Member of Johannesburg SQL User Group Mcom: IT Management with UJ Microsoft Certifications MCSE: SQL Server 2012 BI; Data Platform MCSD: ALM MCSA: SQL Server 2012, SQL Server 2008 MCITP: DBA 2008, BI 2008 MCTS: TFS 2010 Blog/Writing SQLShack.com BIDN.com SQLServerCentral.com Social Network @mafiswana sifiso.w.ndlovu facebook.com/sifiso.w.ndlovu Sifiso W Ndlovu Website

SQL Saturday Audience: Assumptions
Developers DBAs Architects Technical Oriented People Managers Business Analysts Product Owners Business Minded People

SQL Saturday Audience: Assumptions
Intermediate Level Clustered Index SQL Server Statistics Understanding Execution Plans Non-Clustered Index Database Diagrams SQL Server Data Storage Data Warehouse Design Principles

Columnstore Index: SQL Server 2012
Agenda Introduction Columnstore Index: SQL Server 2012 What Is a Columnstore Index How to Create Columnstore Index Columnstore Index Internals Advantages to Columnstore Index Usage Limitations to Columnstore Index Usage Tips, Tricks and Best Practice Case Study Columnstore Index: SQL Server 2014

Session Objectives & Takeaway
Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Introduction

What Is a Columnstore Index Several Explanations Enterprise feature that is intended to improve the query response on T-SQL queries against Analytical/Data Warehouse environments Data warehouse workloads typically scan, aggregate, and join large amounts of data Introduced in SQL Server 2012 as an alternative to the traditional row-based indexing (in Data Warehouse environments) that has been dominant in SQL Server versions prior to SQL Server 2012

What Is a Columnstore Index SQL Server 2008 R2 Express

What Is a Columnstore Index Several Explanations Column-based non-clustered index geared toward increasing query performance for workloads that involve large amounts of data, typically found in data warehouse fact tables Column store index stores data in columnar fashion and uses compression aggressively to reduce the disk I/O needed to serve the query request

What Is a Columnstore Index Several Explanations The defining characteristic of columnar storage is the ability to read the values of a particular column of a table without having to read the values of all the other columns In row-oriented storage this is impossible because the individual column values are physically stored grouped in rows on the pages and reading a page in order to read a column value must fetch the entire page in memory, thus automatically reading all the other columns in the row

What Is a Columnstore Index Several Explanations Columnstores are much more like the row store heaps and supports only end-to-end scans. They do not have a key and do not offer any order of rows. Yet, on large data sets, they can offer exceptional performance for analytical workloads due to the factors mentioned above: read only for the columns referenced by the query, high compression, segment elimination and fast batch mode processing

What Is a Columnstore Index Row-based storage

What Is a Columnstore Index Column-based storage

Questions…

How to Create Columnstore Index The T-SQL Syntax

How to Create Columnstore Index Create Columnstore Index: GUI SQL Server 2008 R2 SQL Server 2012

How to Create Columnstore Index Create Columnstore Index: GUI

How to Create Columnstore Index

How to Create Columnstore Index Create Columnstore Index: GUI Potential Issue of Creating Index via GUI I tried to create a columnstore index with SQL Server Management Studio using the Indexes->New Index menu and it timed out after 20 minutes. How can I work around this? Run a CREATE NONCLUSTERED COLUMNSTORE INDEX statement manually in a T-SQL window instead of using the graphical interface. This will avoid the timeout imposed by the Management Studio graphical user interface.

How to Create Columnstore Index Create Columnstore Index: Script

How to Create Columnstore Index Create Columnstore Index: Script CREATE NONCLUSTERED COLUMNSTORE INDEX ix_cs_MyDWTable ON dbo.MyDWTable (col1 , col2 , ... , coln); DROP INDEX dbo.MyDWTable.ix_cs_MyTable; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable DISABLE; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable ENABLE;

How to Create Columnstore Index Supported Data Types Character Strings Unicode Character Strings Exact Numerics Approximate Numerics Date and Time CHAR NCHAR INT FLOAT DATE VARCHAR – except for MAX values NVARCHAR – except for MAX values BIGINT REAL DATETIME SMALLINT DATETIME2 TINYINT SMALLDATETIME BIT TIME MONEY DATETIMEOFFSET – with a scale less than 2 SMALLMONEY DECIMAL NUMERIC

Questions…

Columnstore Index Internals Segment, Segmentation, and Segment Limitation

Columnstore Index Internals Segment, Segmentation, and Segment Limitation Columnstore data is physically stored in one or more segments A segment is a highly compressed Large Object (LOB) that can contain up to one million rows The data within each column’s segment matches row-by-row so that the rows can always be assembled correctly Segment can contain values from one column only, which allows each column’s data to be accessed independently Segments are compressed on disk and remain compressed when cached in memory

Columnstore Index Internals Segment, Segmentation, and Segment Limitation Of course, a column’s data won’t always fit into a single segment, given the one-million-row limitation. In such cases, multiple segments are created for each column and grouped into multiple row groups, one for each set of segments A column can span multiple segments, and each segment can be made up of multiple data pages Data is transferred from the disk to memory by segment, not by page

Columnstore Index Internals Segment, Segmentation, and Segment Limitation The underlying storage mechanism is still the good old 8K pages Segment elimination refers to the skipping of large chunks of data to speed up scans Each segment has metadata that stores the minimum and maximum value of each column for the segment. The storage engine checks filter conditions against the metadata. If it can detect that no rows will qualify then it skips the entire segment without even reading it from disk

Columnstore Index Internals Segment, Segmentation, and Segment Limitation Segment Limitation Row Group Segment

Columnstore Index Internals xVelocity, Vertipac, and Compression

Columnstore Index Internals xVelocity, Vertipac, and Compression formerly known as Vertipac, was first introduced to the BI analytical line of products: Power Pivot and Tabular Mode for Analysis Services enables large amounts of data to be compressed in-memory reduces the number of disk reads and increases buffer cache hit ratios because only the smaller column-based data pages that need to satisfy a query are moved into memory columnstore indexes are possible by leveraging the xVelocity compression technology

Columnstore Index Internals xVelocity, Vertipac, and Compression Because column oriented storage groups values from the same column together there is a new beneficial side effect: data becomes more compressible Compression can be deployed on row-oriented storage too (i.e. Page Compression), however with column oriented storage the data is more homogenous, as it contains only values from a single column

Columnstore Index Internals xVelocity, Vertipac, and Compression Since compression ratio is subject to the data entropy (how homogenous the data is) it follows that columnar storage format is more compressible than the same data represented in row oriented storage format Unlike base tables whereby user can specify page or row compression, columnstore index compression cannot be specified by the user

Columnstore Index Internals Batch, Batches, and Processing

Columnstore Index Internals Batch, Batches, and Processing new query execution engine optimized for BI queries highly efficient, vector-based query execution method A "batch" is an object that contains about 1000 rows. Each column within the batch is represented internally as a vector. Batch processing can reduce CPU consumption 7X to 40X compared to the older, row-based query execution methods. Efficient vector-based algorithms allow this by dramatically reducing the CPU overhead of basic filter, expression evaluation, projection, and join operations

Columnstore Index Internals Batch, Batches, and Processing When the query uses at least one columnstore index, batch mode processing can speed up joins, aggregations, and filtering During batch mode processing, columnar data is organized in vectors during query execution Sets of data are processed a-batch-at-a-time instead of a-row-at-a-time, using highly efficient algorithms designed to take advantage of modern hardware The query optimizer takes care of choosing when to use batch mode processing and when to use traditional row mode query processing

Columnstore Index Internals Batch, Batches, and Processing

Columnstore Index Internals Batch, Batches, and Processing For best performance a query plan has to run in batch mode from top to bottom batch mode execution requires everybody to play the same tune: all operators have to implement the batch mode execution Row mode operators cannot interact with batch mode operators nor the other way around

Columnstore Index Internals Batch, Batches, and Processing The Query Optimizer can resort to a conversion operator from batch mode to row mode and it can create plans that have sub trees in batch mode even if the upper part of the tree is in row mode This mixed mode can still yield significant performance improvements, provided that most data processing still occurs in batch mode (eg. aggregation occurs in batch mode and the result of aggregation is consumed in row mode)

Columnstore Index Internals Batch, Batches, and Processing List of plan operators support batch mode operation Columnstore Index Scan Hash Inner Join Batch Hash Table Build Bitmap Filter Hash Aggregate (not scalar aggregates) Filter Compute Scalar (for projection and expression evaluation)

Columnstore Index Internals Batch, Batches, and Processing Most tricks rely on splitting the query in two, a subquery that does the bulk of the work in batch mode and then project aggregated results into a an outer query that uses some non batch mode friendly operator like OUTER JOIN, NOT IT, IN, EXISTS, Scalar Aggregates or DISTINCT aggregates

Questions…

Advantages to Columnstore Index Usage

Advantages to Columnstore Index Usage Read only for the columns referenced by the query High compression Segment elimination Fast batch mode processing Buffer hit rates are improved because data is highly compressed, and frequently accessed parts of commonly used columns remain in memory, while infrequently used parts are paged out

Questions…

Limitations to Columnstore Index Usage

Limitations to Columnstore Index Usage Restrictions on Column-Store Indexes Columnsstore index is not allowed to be created against a sparse column Columnsstore index is not allowed to have more than 1024 columns Columnsstore index is not allowed to be used as a primary key in a given object Columnsstore index is not allowed to be used as a foreign key in a given object

Limitations to Columnstore Index Usage Restrictions on Column-Store Indexes During a create columnstore index statement, you are not allowed to specify sorting keywords (i.e. ASC, DESC) on the columns that are used as part of the columnstore index Columnsstore index is not allowed to be created against a view or indexed view Columnsstore create index statement is not allowed to contain the INCLUDE keyword You are not allowed to modify the columnsstore index using the ALTER INDEX statement

Limitations to Columnstore Index Usage Restrictions on Column-Store Indexes A columnstore index can only be created in a table that currently has no data You are restricted to creating a single columnstore index per given object You are not allowed to update the data of an object that uses columnstore indexing Although you are allowed to add columns to a table that uses columnstore index, you are not allowed to drop any of the columns that are referenced in a columnstore index

Limitations to Columnstore Index Usage SQL Server 2012 Features that do not support Column-Store Indexes SQL Server Replication Change Tracking Change Data Capture Filestream Compression Page Row SQL Server Vardecimal Storage Format

Limitations to Columnstore Index Usage SQL Server 2012 Data Types that do not support Column-Store Indexes Character Strings Unicode Character Strings Exact Numerics Binary Strings Date and Time Other Data Types VARCHAR (MAX) NVARCHAR (MAX) DECIMAL - with precision greater than 18 digits BINARY DATETIMEOFFSET - with scale greater than 2 XML NUMERIC - with precision greater than 18 digits VARBINARY HIERARCHYID IMAGE SPATIALTYPES TIMESTAMP SQLVARIANT UNIQUEIDENTIFIER ROWVERSION

Limitations to Columnstore Index Usage

Questions…

Tips, Tricks and Best Practice

Tips, Tricks and Best Practice Comparing with traditional row store indexing strategies, the columnstore indexing rules are straight forward: pick a large table that is usually the fact table in a star schema data warehouse and add a columnstore index that includes every column. Without key order, ASCENDING or DESCENDING clauses, fill factors, filtered indexes, INCLUDE columns or any other of the whistles and bells of row store indexes, the decision is much simpler. Consider the columnstore index as an alternative to the base table

Tips, Tricks and Best Practice The key to getting the best performance is to make sure your queries process the large majority of data in batch mode To tell if the main part of your query is running in batch mode, look at the graphical showplan, hover the mouse pointer over the most expensive scan operator (usually a scan of a large fact table) and check the tooltip. It will say whether the estimated and actual execution mode was Row or Batch Columnstore indexes are not (yt) available in SQL Azure

Tips, Tricks and Best Practice Avoid use of OUTER JOIN on columnstore-indexed tables. Outer joins don't benefit from batch processing instead, SQL Server 2012 reverts to row-at-a-time processing Avoid use of NOT IN on columnstore-indexed tables. NOT IN (<subquery>) (which internally uses an operator called "anti-semi-join") can prevent batch processing and cause the system to revert to row mode NOT IN (<list of constants>) typically works fine though If you frequently update your dimension tables, and they are not too large, you may find the maintenance effort outweighs the benefit of a columnstore index

Tips, Tricks and Best Practice Avoid use of UNION ALL to directly combine columnstore-indexed tables with other tables. Batch processing doesn't get pushed down over UNION ALL. So, for example, creating a view vFact that does a UNION ALL of two tables, one with a columnstore indexes and one without, and then querying vFact in a star join query, will not use batch processing Columnstore indexes are designed to accelerate data warehouse queries, not OLTP workloads. Use columnstore indexes when your query workload entails scanning and aggregating large amounts of data or joining multiple tables, especially in a star join pattern

Tips, Tricks and Best Practice Be sure to populate the table with data before you create the columnstore index Microsoft recommends that columnstore indexes are used for fact tables in datawarehouses, for large dimensions (say with more than 10 millions of records), and any large tables designated to be used as read-only Microsoft recommends that you place columnstore indexes on all of the columns in a given table Disable or drop the columnstore index. You will then be able to update the table and then rebuild the columnstore index

Tips, Tricks and Best Practice Use partition switching. If your table is partitioned, you can put new data into a staging table, build a columnstore index on the staging table, and switch the staging table into an empty partition of your main table. Similarly, you could modify existing data by first switching a partition from the main table into a staging table, disable the columnstore index on the staging table, perform your updates, rebuild the columnstore index, and switch the partition back into the main table To build columnstore indexes as fast a possible, make sure to provide plenty of memory

Tips, Tricks and Best Practice Creating a columnstore index is a parallel operation, subject to the limitations on the number of CPUs available and any restrictions set on MaxDOP. Creating a columnstore index takes on the order of 1.5 times as long as building a B-tree on the same columns Use best practices for statistics management and query design. This is independent of columnstore technology. Use good statistics and avoid query design pitfalls to get the best performance

Tips, Tricks and Best Practice Statistics are not created automatically when you create a columnstore index. The optimizer will use statistics that exist on the base table, whether those statistics were created automatically or manually. Hence, you will generally want to create statistics on all the columns you expect to query or leave auto create statistics turned on When you create a columnstore index, a statistics object is created, but the values reported by DBCC SHOW_STATISTICS are all NULL. The statistics object associated with a columnstore index is used only for database cloning.

Tips, Tricks and Best Practice Avoid joins and string filters directly on columns of columnstore-indexed tables. String filters don't get pushed down into scans on columnstore indexes, and join processing on strings is less efficient than on integers. Filters on number and date types are pushed down. Consider using integer codes (or surrogate keys) instead of strings in columnstore indexed fact tables. You can move the string values to a dimension table. Joins on the integer columns normally will be processed very efficiently

Tips, Tricks and Best Practice If your table has less than one million rows, SQL Server will use only one thread to create the columnstore index. Creating the index in parallel requires more memory than creating the index serially. If your table has more than one million rows, but SQL Server cannot get a large enough memory grant to create the index using MAXDOP, SQL Server will automatically decrease DOP as needed to fit into the available memory grant. In some cases, DOP must be decreased to one in order to build the index under constrained memory

Tips, Tricks and Best Practice The memory required for creating a columnstore index depends on the number of columns, the number of string columns, the degree of parallelism (DOP), and the characteristics of the data. SQL Server will request a memory grant before trying to create the index. If not enough memory is available to create the index in parallel with the current max DOP, SQL Server will reduce the DOP as needed to get an adequate memory grant. If SQL Server cannot get a memory grant to build the index with DOP = 1, the index creation will fail

Tips, Tricks and Best Practice A rule of thumb for estimating the memory grant that will be requested for creating a columnstore index is: Memory grant request in MB = [(4.2 *Number of columns in the CS index) + 68]*DOP + (Number of string cols * 34) It's possible for creation of a columnstore index to fail either at the very beginning of execution if it can't get the necessary initial memory grant, or later during execution if supplemental grants can't be obtained If the initial grant fails, you'll see error 8657 or 8658 You may get error 701 or 802 if memory runs out later during execution

Tips, Tricks and Best Practice If out-of-memory error 8657 or 8658 occur at the beginning of columnstore index creation, first, check your resource governor settings. The default setting for resource governor limits a query in the default pool to 25% of available memory even if the server is otherwise inactive. This is true even if you have not enabled resource governor. Consider changing the resource governor settings to allow the create index statement to access more memory

Tips, Tricks and Best Practice Another way to deal with out-of-memory conditions during columnstore index build is to vertically partition a wide table into two or more tables so that each table has fewer columns. If a query touches both tables, the table will have to be joined, which will affect query performance. If you use this option, you will want to allocate columns to the different tables carefully so that queries will usually touch only one of the tables. This option would also affect any existing queries and loading scripts. Another option is to omit some columns from the columnstore index. Good candidates are columns that are infrequently touched by queries that require scanning large amounts of data

Questions…

AdventureWorks Cycles
Case Study AdventureWorks Cycles Fictitious company on which the AdventureWorks sample databases are based Manufactures and sells metal and composite bicycles to North American, European and Asian commercial markets Actually redirects to Microsoft.com Requirement/Product Backlog Item/User Story “As a Procurement Manager, I would like a report that shows Products by their Units In versus Units Out”

Case Study AdventureWorksDW2012

A Tale of Two Queries AdventureWorksDW2012
TSQL Script Without ColumnStore Index TSQL Script That Uses ColumnStore Index

Case Study

Case Study AdventureWorksDW2012

Questions…

Fully Read/Write Clustered Columnstore No need for “trickle-loading” or other workarounds Yet Partition switching & BULK INSERT remain best practices Data type support expanded All data types except: (n)varchar(max), varbinary(max), XML, Spatial, CLR Basically, SQL14 columnstore is compatible with all non-blob datatypes Note: Non-clustered columnstore is still supported & is still a read-only structure Dependency on conventional b-tree structures has been removed

Batch mode Query Processor vector-based (L1 cache resident) operations expanded, improved New support for: All joins (including OUTER, HASH, SEMI (NOT IN, IN), UNION ALL, Scalar aggregates, “Mixed mode” plans, Hash join enhancements Columnstore joins on string columns is better in 2014 than 2012, yet still slower! Compression Adds an additional layer of compression on top of the inherent compression used by ColumnStore Note: Non-clustered columnstore is still supported & is still a read-only structure Dependency on conventional b-tree structures has been removed

How to Create Columnstore Index Create Columnstore Index: Script CREATE CLUSTERED COLUMNSTORE INDEX ix_cs_MyDWTable ON dbo.MyDWTable; --no column list! DROP INDEX dbo.MyDWTable.ix_cs_MyTable; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable DISABLE; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable ENABLE;

Questions…

Columnstore Indexing: From SQL Server 2012 to SQL Server 2014

Similar presentations

Presentation on theme: "Columnstore Indexing: From SQL Server 2012 to SQL Server 2014"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Columnstore Indexing: From SQL Server 2012 to SQL Server 2014

Similar presentations

Presentation on theme: "Columnstore Indexing: From SQL Server 2012 to SQL Server 2014"— Presentation transcript:

Similar presentations

About project

Feedback