Aleksandar Drašković Enterprise Architect deroso Solutions GmbH Data shredding: a deep dive into SharePoint 2013 storage architecture

about me

agenda Structured and unstructured data Previously on „SharePoint Storage“ Shredded Storage overview Q&A

Inspirirani ljudima. Structured and unstructured data

“on average 20% of data is structured, 80% is unstructured or semi- structured”

Unstructured data No specific format or sequence Not tied to rules Unpredictable Examples: Text, Video, Audio, Images, Word, PowerPoint

Structured data Organized in semantic chunks (entities) Tied to relationships and has attributes Associated with a defined schema All entities have the defined format Have a predefined length Example EDI

Data in SharePoint BLOB = Binary Large Object BLOB is the data stream associated with a file SharePoint file metadata and BLOBs are stored in SQL databases BLOBs do not participate in query operations Sample BLOB operations: Get, Put, Read range, etc. SharePoint is built around the file Document libraries, Record Centers BLOBs generally represent 80% of total content

SQL BLOBS Binary large objects stored in data tables (varbinary(MAX) – 2010 and 2013, image in 2007) Image was limited to 2GB Varbinary virtually unlimited, but SharePoint still has limit of 2GB in code SQL BLOBS are traditional method of storing and retrieving binary large objects with SharePoint

BLOB Storage Challenges Storage SQL storage is usually more expensive SAN versus CAS stores Performance Impacts load on SQL Server box Policy requirements Expunge, BLOB immutability

Inspirirani ljudima. Previously on „SharePoint Storage“

SharePoint Storage History SharePoint Portal Server 2001 Web Storage System SharePoint Portal Server 2003 Relational Database Storage SharePoint Server 2007 External BLOB Storage (EBS) SharePoint Server 2010 Remote Blob Storage (RBS) SharePoint Server 2013 Shredded Storage (Awesome sauce)

SharePoint 2001 Based on Web Storage System Originally implemented in Exchange 2000 Hierarchical model for storing unstructured content One database per site, one table per list

SharePoint 2003 Fundamentally changed approach All documents stored in SQL Server databases All sites stored in one database All document BLOBs stored in one table Tables: dbo.Sites, dbo.Docs, dbo.Lists, dbo.Links, dbo.WebParts

SharePoint 2007 Follows the similar model to SharePoint 2003 Introduces External BLOB Storage (EBS) Extension on the SharePoint side Requires 3rd party components Utilizes COM interface (ISPExternalBinaryProvider) Hooks up to Open and Save commands and invokes redirection calls

SharePoint 2010 Maintained relational database model EBS deprecated in favour to Remote BLOB Storage (RBS) Offloads data externalization to SQL Server FILESTREAM provider OOTB, 3 rd party solutions available

SharePoint 2013 Maintains and extends relational database model EBS support is removed from SharePoint 2013 RBS is still supported Introduced Shredded Storage! Yay!

Inspirirani ljudima. Shredded Storage overview

Shredded Storage Question? What is Shredded Storage? Simple Answer A technology that break apart files into smaller chunks Advanced Answer A platform for other higher level applications to take advantage of

Goals Reduce Storage Saving only modified parts of the file Optimize Bandwidth Office applications use Shredded Storage and Cobalt bandwidth optimization Optimize File I/O Only shreds are saved, not the whole file Security File extraction from the Content DB is harder than earlier

SSL Secure Shredded Store in Office 365

Someone said “Cobalt”? Introduced in SharePoint 2010 Also known as MS-FSSHTTP protocol Used by Office clients Transfers compressed deltas

Someone said “Cobalt”? Takes care of the locking operations Enables multi-user authoring Supported by file format changes introduced by Office XML file format Essentially, a bunch of ZIP-compressed XML files

Content DB schema changes dbo.AllDocStreams renamed to dbo.DocStreams Each row in dbo.DocStreams stores a chunk or portion of the BLOB Columns: BSN (BLOB Sequence Number), Content (subset of a binary data), Rbsid (Remote BLOB Storage identifier) New dbo.DocToStreams table Contains pointers to corresponding rows in dbo.DocStreams Used for rebuilding document stream BLOB Sequence Number (BSN) manages the sequence across dbo.AllDocVersions dbo.DocsToStreams dbo.DocStreams NextBSN is used to manage the last BSN for each BLOB The BLOB access pattern is dbo.AllDocs/dbo.AllDocVersions > dbo.DocsToStreams > dbo.DocStreams.

Content DB schema changes dbo.AllDocs Site association Library association Pointer to the information in dbo.AllDocVersions Bunch of other information about documents Actual metadata is stored in dbo.AllUserData dbo.AllDocVersions Information on document versions

Content DB schema changes

Configuration parameters FileWriteChunkSize The target size of the shreds of a file binary FileReadChunkSize The size of the data returned from each Stored Procedure call to a file binary

FileWriteChunkSize Large values improve throughput, small values improve latency. Should not exceed 4MB as significant hit on I/O operations will occur Should not be set to less than 64 KB Optimal setting is based on workload 1-4 MB, depending on the use case OneDrive is set to 2 MB Size of partitioned BLOB can be adjusted via PowerShell Server Object Model API

FileReadChunkSize Controls the size of incremental reads HTTP range request support Request only a piece of file BLOB cache is required Recommendations: >12.5% of average file size = normal operation 6% 12.5% = 10% hit on read operations 3% 6% = 20% hit on read operations X<3% = 50% hit on read operations Beware Too high of a setting OneDrive for Business will stop working ICsiError: csierrWebService_QuotaExceeded (0x662) Average file size drives the setting

Shredded Storage facts Always on, can‘t be turned off Works only on the item scope (document) Only a (storage) benefit with the versioning turned on Shredding works with all file types but Office XML documents benefits the most "Cobalt" works only with the Office XML and Office BLOB data is not automatically shredded after upgrade to SharePoint 2013 I/O recommendations are the same as for SharePoint 2010

Inspirirani ljudima. Pitanja i odgovori.

Inspirirani ljudima. Thank you!