Data Manipulation Language Deep Dive into Internals of DML Uwe Ricken MCM:Microsoft Certified Master – SQL 2008 MVP:Most Valued Professional – SQL Server Slide: 1

Our Main Sponsors:

Say Thank you to Volunteers:  They spend their FREE time to give you this event.  Because they are crazy.  Because they want YOU to learn from the BEST IN THE WORLD.

Paulo Matos:

Pedro Simões:

André Batista:

Paulo Borges:

André Melancia:

Murilo Miranda:

Quilson Antunes:

5 Sponsor Sessions at 15:10  Don’t miss them, they might be getting distributing some awesome prizes!  Rumos  BI4ALL  Bold Int  CozyRoc  Pythian

Important Activities: WIT – Women in Technology  15:10 at BizSpark Room (Ground Floor) SQLClinic Challenges  10:00 DEV (Neil Hambly)  11:50 DBA (Uwe Ricken)  17:00 BI (Steph Locke)

Slide: 13 www: blog: xing: I am working with IT-systems since early 1990's and with the main focus on Microsoft SQL Server since version 6.0. I started with development of database applications in 1998 with a professional CRM-System based on Microsoft products (Microsoft Office and Microsoft SQL Server). Since 2008 I'm focused exclusively on Microsoft SQL Server and since 2008 I'm working in 3rd level support teams for banks, insurances and global industries. Since May 2013 I'm a Microsoft Certified Master: SQL Server 2008 which was an amazing way into the depth of Microsoft SQL Server. In July 2013 I have been awarded with the MVP Award for Microsoft SQL Server. Uwe Ricken db Berater GmbH

The Microsoft Certified Solutions Master (MCSM) and Microsoft Certified Master (MCM) programs recognize the most experienced IT professionals who demonstrate and validate their deep technical skills to successfully design and implement technical solutions in a complex business environment. Microsoft Certified Solutions Expert (MCSE) certifications are designed to recognize IT professionals who can design and build solutions across multiple technologies, both on-premises and in the cloud. The Microsoft Most Valuable Professional (MVP) Award is our way of saying thank you to exceptional, independent community leaders who share their passion, technical expertise, and real-world knowledge of Microsoft products with others. Uwe Ricken db Berater GmbH Slide: 14

Agenda  internal structures  Special pages in a database  data storage internals  HEAP / Clustered Index  functions and tools for investigation of DML operations  fn_PhysLocFormatter() / fn_PhysLocCracker()  DBCC TRACEON / DBCC PAGE  sys.fn_dblog()

Agenda  INSERT  Insert data into a HEAP  Insert data into Clustered Indexes  UPDATE  Update data in a HEAP  Update data in a Clustered Index  DELETE  DELETE records in a HEAP  Delete data in a Clustered Index

Special Pages - Overview  PFS:Page Free Space  GAM:Global Allocation Map  SGAM:Shared Global Allocation Map  IAM:Index Allocation Map

What functions do we have for monitoring  fn_PhysLocFormatter() / fn_PhysLocCracker()  Get information about the logical location of a record  fn_dbLog()  Decrypt secrets from the transaction log  Running fn_dblog will cause I/Os on the transaction log, which can cause performance issues.  Log clearing is disabled while fn_dblog is running.

What functions do we have for monitoring  DBCC PAGE (db_id, file_id, page_id, Option)  0 = Default; prints the buffer header and page header  1 = Prints the buffer and page header, each row separately and the row offset table  2 = Prints the buffer and page headers, the page as a whole, and the offset table  3 = as 2 (except offset table) + each row followed by each of its column values listed separately  ENABLE TF 3604 before you run DBCC PAGE to output information to the client instead of the error log!

Data Storage internals

Anatomy of a heap A heap is a table without a clustered index. Data is stored in the heap without specifying an order. There is NO relation between the leaf levels for next / previous page! R LLL Demo: 0002 – anatomy of a heap.sql

Anatomy of a clustered Index Leaf nodes contain the data pages. The root node and intermediate nodes contain index pages. Each index row contains a key value and a pointer to either an intermediate level page in the B- tree or a data row in the leaf level of the index. R III L L L L L L L

Anatomy of a non cl. Index Non clustered indexes have the same B-tree structure as clustered indexes, with two significant differences: The data rows are not sorted and stored in order based on their nonclustered keys. The leaf layer of a nonclustered index does not consist of the data pages.

INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes Demo: 0007 – anatomy of a data row.sql

INSERT / UPDATE / DELETE –anatomy of a data  First Status Byte contains properties of the row itself * BitDescription 0Versioning information (always 0) 1Record type 20 = PRIMARY RECORD | 1 = FORWARDED RECORD | 2 = FORWARDING STUB 33 = INDEX RECORD | 4 = BLOB, ROW OVERFLOW | 5 = GHOST INDEX RECORD 6 = GHOST DATA RECORD | 7 = GHOST VERSION RECORD 4NULL-Bitmap (since 2008 always!) 5Columns with variable length exists in the record 6Row contains versioning information 7Not used * Kalen Delaney: SQL Server 2008 internals

INSERT / UPDATE / DELETE –anatomy of a data  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory  f6e61 6c Donald  Daisy  e ’ƒ....à.  d F.married INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', ' ', ‘married') Offset 7Offset 6Offset 5Offset 4Offset 3Offset 2Offset 1Offset Not usedversioni ng var length NULL exists Record TypeVersion 0 = PRIMARY RECORD

INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory  f6e61 6c Donald  Daisy  e ’ƒ....à.  d F.married 0x38 & 0x00 = bytes – 4 bytes (row header) = 52 bytes f6e61 6c f6e61 6c INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', ' ', ‘married')

INSERT / UPDATE / DELETE –anatomy of a data Attrib ute Datat ype Len gth HexValueValue IdInt C1Char20 446f6e61 6c Donald C2Char Daisy C3Datetim e C4Varchar20 C5Image C6Varchar

INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory  f6e61 6c Donald  Daisy  e ’ƒ....à.  d F.married CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', ' ', ‘married')

INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory  f6e61 6c Donald  Daisy  e ’ƒ....à.  d F.married IdC1C2C3C4C5C CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', ' ', ‘married')

INSERT / UPDATE / DELETE – Deep Dive anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory  f6e61 6c Donald  Daisy  e ’ƒ....à.  d F.married CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', ' ', ‘married')

INSERT / UPDATE / DELETE – anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory  f6e61 6c Donald  Daisy  e ’ƒ....à.  d F.married INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', ' ', ‘married')

INSERT / UPDATE / DELETE – anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

INSERT / UPDATE / DELETE – anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory  f6e61 6c Donald  Daisy  e ’ƒ....à.  d F.married m a r r i e d. 6d Demo: Basics about relations indexes partitions allocation units pages etc.sql INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', ' ', ‘married')

INSERT: Heap vs. Clustered Index  A HEAP is an „unsorted bunch of data  an INSERT command scans the PFS to find a page where the data can be inserted.  An INSERT command in a Clustered Index checks the free space on the page itself D E M O

Page Splits PAGE Page 10 Page 11 Page 12 Page 13 Page 14 Page 15

Page Splits Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 D E M O

Actions of Page Split LOP_BEGIN_XACT LOP_INSYSXACTLCX_INDEX_INTERIORMark index page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDMark leaf page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDMark NEXT leaf page as part of system transaction LOP_MODIFY_ROWLCX_PFSSet the value of free space in PFS LOP_HOBT_DELTAIncrease the value for the amount of data pages LOP_FORMAT_PAGELCX_HEAPZero the new page LOP_INSYSXACTLCX_CLUSTEREDMark new created page as part of system transaction LOP_INSERT_ROWSLCX_CLUSTEREDInsert data on the new page LOP_DELETE_SPLITLCX_CLUSTEREDRemove the data from the original page LOP_MODIFY_HEADERLCX_HEAPSet [NEXT PAGE] on original page LOP_MODIFY_HEADERLCX_HEAPSet [PREVIOUS PAGE] on original page LOP_INSERT_ROWSLCX_INDEX_INTERIORUpdate of the root page / B-tree level LOP_INSYSXACTLCX_CLUSTEREDRelease NEWLY added leaf page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDRelease OLD next page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDRelease NEW next page as part of system transaction LOP_INSYSXACTLCX_INDEX_INTERIORRelease B-Tree as part of system transaction LOP_COMMIT_XACT

UPDATE data  UPDATES in a heap may allocate the existing space if the record will have enough space on the page  If an UPDATE will increase the amount of data a FORWARDING RECORD will be generated!  Updating data in a clustered index may lead to a page split if the record does not fit into the page D E M O

Forwarded Records  Forwarded records can only occur in HEAPS  A forwarded record will be generated if the updated record doesn’t fit on the page  A forwarded record does not jump from one forwarding point to another one  The record details itself contain the new address

Slot 1, Offset 0x520, Length 9, DumpStyle BYTE Record Type = FORWARDING_STUB Record Attributes = Record Size = 9 Memory : 04 ca Ê Forwarded Records ValueDescriptionConverte d 04The first byte defines the type of the record (04 = forwarded record) 4 ca Number of the page where the record has been moved to Number of the data file1 00 Number of the slot0

Summary – UPDATE A HEAP  fixed length attributes are best handled by Microsoft SQL Server  variable length attributes may generate forwarded records if the record does not completely fit on the page  If a forwarded record will be updated to a length which will fit to the original page the operation will be reverted (both are separate transactional operations)

Summary – UPDATE a C. I.  UPDATES in C. I. will be handled in the same way as in HEAPS  If a record does not match the available space on the page an expensive PAGE SPLIT will occur.

Summary - DELETE  DELETE removes data from the slot but not from the page!  To delete the data pages in a HEAP you have to use WITH (TABLOCK) to hold and exclusive lock on the table  DELETE in a clustered index will not remove the data but the record will be marked as “GHOST_RECORD”  A background task will delete the record from the page

Thank you for attention  Useful links  DML-Operation: us/library/ff aspx us/library/ff aspx  DBCC: us/library/ms aspxhttp://technet.microsoft.com/en- us/library/ms aspx  SQLSkills:  Contact  blog:  