1. Data Manipulation Language Deep Dive into Internals of DML Uwe Ricken MCM:Microsoft Certified Master – SQL 2008 MVP:Most Valued Professional – SQL Server Slide: 1

2. Our Main Sponsors:

3. Say Thank you to Volunteers:  They spend their FREE time to give you this event.  Because they are crazy.  Because they want YOU to learn from the BEST IN THE WORLD.

4. Paulo Matos:

5. Pedro Simões:

6. André Batista:

7. Paulo Borges:

8. André Melancia:

9. Murilo Miranda:

10. Quilson Antunes:

11. 5 Sponsor Sessions at 15:10  Don’t miss them, they might be getting distributing some awesome prizes!  Rumos  BI4ALL  Bold Int  CozyRoc  Pythian

12. Important Activities: WIT – Women in Technology  15:10 at BizSpark Room (Ground Floor) SQLClinic Challenges  10:00 DEV (Neil Hambly)  11:50 DBA (Uwe Ricken)  17:00 BI (Steph Locke)

13. Slide: 13 www:http://www.db-berater.dehttp://www.db-berater.de email:uwe.ricken@db-berater.deuwe.ricken@db-berater.de blog:http://db-berater.blogspot.dehttp://db-berater.blogspot.de twitter:https://twitter.com/@dbberaterhttps://twitter.com/@dbberater xing:http://www.xing.com/profile/Uwe_Rickenhttp://www.xing.com/profile/Uwe_Ricken I am working with IT-systems since early 1990's and with the main focus on Microsoft SQL Server since version 6.0. I started with development of database applications in 1998 with a professional CRM-System based on Microsoft products (Microsoft Office and Microsoft SQL Server). Since 2008 I'm focused exclusively on Microsoft SQL Server and since 2008 I'm working in 3rd level support teams for banks, insurances and global industries. Since May 2013 I'm a Microsoft Certified Master: SQL Server 2008 which was an amazing way into the depth of Microsoft SQL Server. In July 2013 I have been awarded with the MVP Award for Microsoft SQL Server. Uwe Ricken db Berater GmbH

14. The Microsoft Certified Solutions Master (MCSM) and Microsoft Certified Master (MCM) programs recognize the most experienced IT professionals who demonstrate and validate their deep technical skills to successfully design and implement technical solutions in a complex business environment. Microsoft Certified Solutions Expert (MCSE) certifications are designed to recognize IT professionals who can design and build solutions across multiple technologies, both on-premises and in the cloud. The Microsoft Most Valuable Professional (MVP) Award is our way of saying thank you to exceptional, independent community leaders who share their passion, technical expertise, and real-world knowledge of Microsoft products with others. Uwe Ricken db Berater GmbH Slide: 14

15. Agenda  internal structures  Special pages in a database  data storage internals  HEAP / Clustered Index  functions and tools for investigation of DML operations  fn_PhysLocFormatter() / fn_PhysLocCracker()  DBCC TRACEON / DBCC PAGE  sys.fn_dblog()

16. Agenda  INSERT  Insert data into a HEAP  Insert data into Clustered Indexes  UPDATE  Update data in a HEAP  Update data in a Clustered Index  DELETE  DELETE records in a HEAP  Delete data in a Clustered Index

17. Special Pages - Overview  PFS:Page Free Space  GAM:Global Allocation Map  SGAM:Shared Global Allocation Map  IAM:Index Allocation Map

18. What functions do we have for monitoring  fn_PhysLocFormatter() / fn_PhysLocCracker()  Get information about the logical location of a record  fn_dbLog()  Decrypt secrets from the transaction log  Running fn_dblog will cause I/Os on the transaction log, which can cause performance issues.  Log clearing is disabled while fn_dblog is running.

19. What functions do we have for monitoring  DBCC PAGE (db_id, file_id, page_id, Option)  0 = Default; prints the buffer header and page header  1 = Prints the buffer and page header, each row separately and the row offset table  2 = Prints the buffer and page headers, the page as a whole, and the offset table  3 = as 2 (except offset table) + each row followed by each of its column values listed separately  ENABLE TF 3604 before you run DBCC PAGE to output information to the client instead of the error log!

20. Data Storage internals

21. Anatomy of a heap A heap is a table without a clustered index. Data is stored in the heap without specifying an order. There is NO relation between the leaf levels for next / previous page! R LLL Demo: 0002 – anatomy of a heap.sql

22. Anatomy of a clustered Index Leaf nodes contain the data pages. The root node and intermediate nodes contain index pages. Each index row contains a key value and a pointer to either an intermediate level page in the B- tree or a data row in the leaf level of the index. R III L L L L L L L

23. Anatomy of a non cl. Index Non clustered indexes have the same B-tree structure as clustered indexes, with two significant differences: The data rows are not sorted and stored in order based on their nonclustered keys. The leaf layer of a nonclustered index does not consist of the data pages. http://technet.microsoft.com/en-us/library/ms177484.aspx

24. INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data 123456789 1 Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes Demo: 0007 – anatomy of a data row.sql

25. INSERT / UPDATE / DELETE –anatomy of a data  First Status Byte contains properties of the row itself * BitDescription 0Versioning information (always 0) 1Record type 20 = PRIMARY RECORD | 1 = FORWARDED RECORD | 2 = FORWARDING STUB 33 = INDEX RECORD | 4 = BLOB, ROW OVERFLOW | 5 = GHOST INDEX RECORD 6 = GHOST DATA RECORD | 7 = GHOST VERSION RECORD 4NULL-Bitmap (since 2008 always!) 5Columns with variable length exists in the record 6Row contains versioning information 7Not used * Kalen Delaney: SQL Server 2008 internals

26. INSERT / UPDATE / DELETE –anatomy of a data  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory Dump @0x00000000273DA060  30003800 01000000 446f6e61 6c642020 20202020 0.8.....Donald  20202020 20202020 44616973 79202020 20202020 Daisy  20202020 20202020 00000000 92830000 0700e001....’ƒ....à.  0046006d 61727269 6564.F.married INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', '19920321', ‘married') Offset 7Offset 6Offset 5Offset 4Offset 3Offset 2Offset 1Offset 0 1286432168421 00110000 Not usedversioni ng var length NULL exists Record TypeVersion 0 = PRIMARY RECORD

27. INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data 123456789 1 Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

28. INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory Dump @0x00000000273DA060  30003800 01000000 446f6e61 6c642020 20202020 0.8.....Donald  20202020 20202020 44616973 79202020 20202020 Daisy  20202020 20202020 00000000 92830000 0700e001....’ƒ....à.  0046006d 61727269 6564.F.married 0x38 & 0x00 = 56 56 bytes – 4 bytes (row header) = 52 bytes 01000000 446f6e61 6c642020 20202020 20202020 20202020 44616973 79202020 20202020 20202020 00000000 92830000 01000000 446f6e61 6c642020 20202020 20202020 20202020 44616973 79202020 20202020 20202020 00000000 92830000 INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', '19920321', ‘married')

29. INSERT / UPDATE / DELETE –anatomy of a data Attrib ute Datat ype Len gth HexValueValue IdInt4 010000001 C1Char20 446f6e61 6c642020 20202020 20202020 20202020 Donald.............. C2Char20 44616973 79202020 20202020 20202020 20202020 Daisy............... C3Datetim e 8 00000000 9283000021.03.1992 C4Varchar20 C5Image C6Varchar

30. INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data 123456789 1 Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

31. INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory Dump @0x00000000273DA060  30003800 01000000 446f6e61 6c642020 20202020 0.8.....Donald  20202020 20202020 44616973 79202020 20202020 Daisy  20202020 20202020 00000000 92830000 0700e001....’ƒ....à.  0046006d 61727269 6564.F.married CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', '19920321', ‘married')

32. INSERT / UPDATE / DELETE –anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data 123456789 1 Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

33. INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory Dump @0x00000000273DA060  30003800 01000000 446f6e61 6c642020 20202020 0.8.....Donald  20202020 20202020 44616973 79202020 20202020 Daisy  20202020 20202020 00000000 92830000 0700e001....’ƒ....à.  0046006d 61727269 6564.F.married IdC1C2C3C4C5C6-- 1286432168421 11100000 CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', '19920321', ‘married')

34. INSERT / UPDATE / DELETE – Deep Dive anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data 123456789 1 Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

35. INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory Dump @0x00000000273DA060  30003800 01000000 446f6e61 6c642020 20202020 0.8.....Donald  20202020 20202020 44616973 79202020 20202020 Daisy  20202020 20202020 00000000 92830000 0700e001....’ƒ....à.  0046006d 61727269 6564.F.married CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); CREATE TABLE dbo.tbl_demo ( Id int NOT NULL, c1 char(20) NOT NULL, c2 char(20) NOT NULL, c3 datetime NULL, c4 varchar(20) NULL, c5 image NULL, c6 varchar(max) NULL ); INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', '19920321', ‘married')

36. INSERT / UPDATE / DELETE – anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data 123456789 1 Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

37. INSERT / UPDATE / DELETE –anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory Dump @0x00000000273DA060  30003800 01000000 446f6e61 6c642020 20202020 0.8.....Donald  20202020 20202020 44616973 79202020 20202020 Daisy  20202020 20202020 00000000 92830000 0700e001....’ƒ....à.  0046006d 61727269 6564.F.married INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', '19920321', ‘married')

38. INSERT / UPDATE / DELETE – anatomy of a data row  Status Byte:Information about the kind of row  Status Byte:information about the record is a ghost forwarded record  Fixed Length Size:  Fixed length data  Number of columns  NULL-Bitmap1 bit for each column  Number of variable length columns  Column offset array  variable length data 123456789 1 Byte 2 Bytes x Bytes 2 Bytes x Bytes 2 Bytes x Bytes

39. INSERT / UPDATE / DELETE – anatomy of a data row  Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP Record Size = 70  Memory Dump @0x00000000273DA060  30003800 01000000 446f6e61 6c642020 20202020 0.8.....Donald  20202020 20202020 44616973 79202020 20202020Daisy  20202020 20202020 00000000 92830000 0700e001....’ƒ....à.  0046006d 61727269 6564.F.married m a r r i e d. 6d 61727269 6564 Demo: 000 - Basics about relations indexes partitions allocation units pages etc.sql INSERT INTO dbo.tbl_demo (c1, c2, c3, c4) VALUES ('Donald', 'Daisy', '19920321', ‘married')

40. INSERT: Heap vs. Clustered Index  A HEAP is an „unsorted bunch of data  an INSERT command scans the PFS to find a page where the data can be inserted.  An INSERT command in a Clustered Index checks the free space on the page itself D E M O

41. Page Splits PAGE 10 2 4 6 8 10 12 7 7 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15

42. Page Splits Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 D E M O

43. Actions of Page Split LOP_BEGIN_XACT LOP_INSYSXACTLCX_INDEX_INTERIORMark index page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDMark leaf page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDMark NEXT leaf page as part of system transaction LOP_MODIFY_ROWLCX_PFSSet the value of free space in PFS LOP_HOBT_DELTAIncrease the value for the amount of data pages LOP_FORMAT_PAGELCX_HEAPZero the new page LOP_INSYSXACTLCX_CLUSTEREDMark new created page as part of system transaction LOP_INSERT_ROWSLCX_CLUSTEREDInsert data on the new page LOP_DELETE_SPLITLCX_CLUSTEREDRemove the data from the original page LOP_MODIFY_HEADERLCX_HEAPSet [NEXT PAGE] on original page LOP_MODIFY_HEADERLCX_HEAPSet [PREVIOUS PAGE] on original page LOP_INSERT_ROWSLCX_INDEX_INTERIORUpdate of the root page / B-tree level LOP_INSYSXACTLCX_CLUSTEREDRelease NEWLY added leaf page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDRelease OLD next page as part of system transaction LOP_INSYSXACTLCX_CLUSTEREDRelease NEW next page as part of system transaction LOP_INSYSXACTLCX_INDEX_INTERIORRelease B-Tree as part of system transaction LOP_COMMIT_XACT

44. UPDATE data  UPDATES in a heap may allocate the existing space if the record will have enough space on the page  If an UPDATE will increase the amount of data a FORWARDING RECORD will be generated!  Updating data in a clustered index may lead to a page split if the record does not fit into the page D E M O

45. Forwarded Records  Forwarded records can only occur in HEAPS  A forwarded record will be generated if the updated record doesn’t fit on the page  A forwarded record does not jump from one forwarding point to another one  The record details itself contain the new address

46. Slot 1, Offset 0x520, Length 9, DumpStyle BYTE Record Type = FORWARDING_STUB Record Attributes = Record Size = 9 Memory Dump @0x00000000232CA520 0000000000000000: 04 ca010000 0100 0000.Ê....... Forwarded Records ValueDescriptionConverte d 04The first byte defines the type of the record (04 = forwarded record) 4 ca 01 00 00 Number of the page where the record has been moved to 458 01 00Number of the data file1 00 Number of the slot0

47. Summary – UPDATE A HEAP  fixed length attributes are best handled by Microsoft SQL Server  variable length attributes may generate forwarded records if the record does not completely fit on the page  If a forwarded record will be updated to a length which will fit to the original page the operation will be reverted (both are separate transactional operations)

48. Summary – UPDATE a C. I.  UPDATES in C. I. will be handled in the same way as in HEAPS  If a record does not match the available space on the page an expensive PAGE SPLIT will occur.

49. Summary - DELETE  DELETE removes data from the slot but not from the page!  To delete the data pages in a HEAP you have to use WITH (TABLOCK) to hold and exclusive lock on the table  DELETE in a clustered index will not remove the data but the record will be marked as “GHOST_RECORD”  A background task will delete the record from the page

50. Thank you for attention  Useful links  DML-Operation: http://technet.microsoft.com/en- us/library/ff848766.aspx http://technet.microsoft.com/en- us/library/ff848766.aspx  DBCC:http://technet.microsoft.com/en- us/library/ms188796.aspxhttp://technet.microsoft.com/en- us/library/ms188796.aspx  SQLSkills:http://www.sqlskills.com/http://www.sqlskills.com/  Contact  blog:http://db-berater.blogspot.dehttp://db-berater.blogspot.de  email:uwe.ricken@db-berater.deuwe.ricken@db-berater.de  twitter:@dbberater@dbberater