1. SQL Server 2016: Real-time operational analytics
Sunil Agarwal
Principal Program Manager SQL Server

2. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

3. Traditional Real-time/Analytics Architecture
Microsoft Ignite 2015
Traditional Real-time/Analytics Architecture
4/28/2017 4:15 PM
Select ProduceName, ExpiryDate, Quantity, Sum(Sales)
From <transactions>
Where ProduceType =‘perishables’
Group By ProduceName, ExpiryDate
Insert into <transactions>
values (‘<upc-code>, ‘flowers’, $20.00)
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
SQL Server
Analysis Server
Key Issues
Complex Implementation
Requires two Servers (CapEx and OpEx)
Data Latency in Analytics
More businesses demand/require real-time Analytics
AS – (a) data model variance from budget, forecasting using MDX and DAX (b) caching
Insert into CreditCardTrans values (‘ ’, ‘Surface’, ‘Seatte’, $450.00)
Select count(*) as transacitons, max(price)
from creditCardTrans
Where CreditCardID = ‘ ’)
Group By City
SQL Server
Relational DW
Database
ETL
Hourly, Daily, Weekly
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Minimizing Data Latency for Analytics
Microsoft Ignite 2015
Minimizing Data Latency for Analytics
4/28/2017 4:15 PM
Select ProduceName, ExpiryDate, Quantity, sum (sales)
From <transactions>
Where ProduceType =‘perishables’
Group By ProduceName, ExpiryDate
Insert into <transactions>
values (‘<upc-code>, ‘flowers’, $20.00)
Benefits
No Data Latency
No ETL
No Separate DW
Challenges
Minimizing Impact on OLTP workload
Delivering Performant Analytics
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
This is Real-Time ANALYTICS
SQL Server
Analysis Server
Add columnstore index
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Real-Time Analytics – What it is NOT for
4/28/2017
Real-Time Analytics – What it is NOT for
OLTP
Operational Data Coming from multiple sources
Extreme Analytics
Needs pre-aggregated cubes
Star-Schema
Challenge with OLTP schema
Data is normalized
Queries require multi-table joins
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

7. Real-time Analytics: Nonclustered Columnstore Index (NCCI)
Dynamics AX 7
Production Preview in
Azure SQLDB
Relational Table (disk-based)
(Clustered Index/Heap)
Btree Index
Nonclustered columnstore index (NCCI)
Delta rowgroups
Delete Bitmap
Delete Buffer
Key Points
Create an updateable non-clustered columnstore index (NCCI) for analytics queries
Drop all other indexes that were created for analytics.
No OLTP Application changes.
ColumnStore index automatically keeps up with DML operations
Query Optimizer will choose columnstore index where needed

8. Real-Time Analytics: Columnstore Index Overhead
DML Operations on OLTP workload
Operation
BTREE (NCI)
Non Clustered ColumnStore Index (NCCI)
Insert
Insert row into btree
Insert row into btree (delta store)
Delete
Seek row(s) to be deleted
Delete the row
Seek for the row in the delta stores
(there can be multiple)
If row found, then delete
Else insert the key into delete row buffer
Update
Seek the row(s)
Delete the row (steps same as above)
Insert the updated row into delta store
Improvement - No PAGE compression
Solution – Exponentially Increasing sized Delta RGs
Reclaiming Deleted Rows: Run ALTER Index <> REORGANIZE (it is ONLINE)

9. Real-time Analytics: Minimizing Columnstore Index overhead
OLTP Workload
Relational Table
(Clustered Index/Heap)
Btree Index
HOT
Nonclustered columnstore index (NCCI)
Delta rowgroups
Delete Bitmap
Key Points
Create Columnstore only on cold data – using filtered predicate to minimize maintenance
Analytics query accesses both columnstore and ‘hot’ data transparently
Example – Order Management Application –
create nonclustered columnstore index ….. where order_status = ‘SHIPPED’

10. Real-time Analytics: Minimizing Columnstore Index overhead
Relational Table
(Clustered Index/Heap)
Btree Index
30 mins
Syntax:
Create nonclustered columnstore index <name> on <table> (<columns>) with (compression_delay = 30 Minutes)
HOT
Nonclustered columnstore index (NCCI)
Delete Bitmap
Delete Buffer
Nonclustered columnstore index (NCCI)
Delta rowgroups
Delete Bitmap
Delete Buffer
Key Points
Delta RG is only compressed after ‘Compression_Delay’ duration
Minimizes/Eliminates index fragmentation

11. Real-time Analytics: Minimizing Columnstore overhead
AlwaysOn Availability Group
Analytics workload
Real-time workload
Primary
Replica
Secondary
Secondary
Replica
Insert into <transactions>
values (‘<upc-code>, ‘flowers’, $20.00)
Select ProduceName, ExpiryDate, Quantity, sum (sales)
From <transactions>
Where ProduceType =‘perishables’
Group By ProduceName, ExpiryDate
Key Points
Mission Critical Real-time Workloads typically configured for High Availability using AlwaysOn Availability Groups
You can offload analytics to readable secondary replica

12. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

13. Real-time Analytics: Columnstore on In-Memory Tables
4/28/2017
In-Memory OLTP Table
Range Index
Hash Index
Deleted Rows Table (DRT) – Tracks deleted rows
No explicit delta rowgroup
Rows (tail) not in columnstore stay in in-memory OLTP table
No columnstore index overhead when operating on tail
Background task migrates rows from tail to columnstore in chunks of 1 million rows
Compression Delay Option for Hot Data
Storage
Columnstore data fully resident in memory
Persisted together with OLTP data in Memory-Optimized FG
No OLTP application changes required.
Deleted Rows Table
Hot
Delta RG
Hot
Delta RG
Columnstore Index
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Session Objectives And Takeaways
Tech Ready 15
4/28/2017
Session Objectives And Takeaways
Session Objective(s):
What/Why Real-Time Analytics?
Rich set of configurations for Real-Time Analytics
First leading commercial database combining in-memory OLTP and in-memory DW
Key Takeaways
I can do real-time analytics with SQL Server with no application changes
I have multiple options to minimize/eliminate impact on my OLTP workload
Superior Real-Time analytics over Oracle12C (In-Memory)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. 4/28/2017
Q&A
Note - It is important for attendees to use the Q&A microphone located in the session room:
TechReadyTV recordings cannot capture Q&A unless it is spoken using the microphone
Attendees in the back of the room may not be able to hear a question from someone in the front of the room
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.