Presentation is loading. Please wait.

Presentation is loading. Please wait.

Managing Very Large Databases with SQL Server

Similar presentations


Presentation on theme: "Managing Very Large Databases with SQL Server"— Presentation transcript:

1 Managing Very Large Databases with SQL Server
Tips and best practices for DBAs

2 Please Visit Sponsors and Enter Raffles

3 Who Am I? Manager, DBA @ ChannelAdvisor Microsoft Certified Master
Seven years working with SQL Server Live in Raleigh, North Carolina 3 | 9/17/2018 Managing Very Large Databases with SQL Server

4 Agenda Very Large Databases Maintenance Partitioning SQL 2016
Backups & Restores Indexes Integrity checks Statistics Partitioning SQL 2016 High Availability & Disaster Recovery 4 | 9/17/2018 Managing Very Large Databases with SQL Server

5 What constitutes a VLDB?
So what are they? Hundreds of MBs in the ‘80s … Hundreds of GBs in ‘90s … Hundreds of TBs in ‘00s ? Hundreds of PBs today ? When default practices may change due to database size 5 | 9/17/2018 Managing Very Large Databases with SQL Server

6 VLDBs in SQL Server > 1TB or more than one billion rows
SQL is a good choice Need to think more about maintenance Definitely need think more about recovery Where EE delivers value Size hardware appropriately Know VLDB management techniques before your database becomes a VLDB 6 | 9/17/2018 Managing Very Large Databases with SQL Server

7 SQL VLDBs @ ChannelAdvisor
88TB production data 111 production SQL instances 7.2TB largest instance 4.2TB largest database 7 | 9/17/2018 Managing Very Large Databases with SQL Server

8 Before you have a VLDB Consider “scale out” options
Compress your data (not Sharepoint) sp_estimate_data_compression_savings Consider archival / retention Sliding window portioning Stretch database May be more cost effective to have fewer EE instances Scale is hard too! For archival incentivize developers rather than punish them Archival must be tough or StretchDB would not exist Can data reside in multiple instances / databases / tables 8 | 9/17/2018 Managing Very Large Databases with SQL Server

9 Backups Possible initial pain point Implication of long backup windows
Restore strategy Difficulty is not an excuse “The database is very big so we stopped taking backups.” – sqlservercentral.com Track backup times Test restores Long backup times impact RPO/RTO Especially if backup fails Rebuild from source may be an option but make sure you can recreate schema (DACPAC may work) Use backup compression 9 | 9/17/2018 Managing Very Large Databases with SQL Server

10 Backup Strategy Differential backups Filegroup backups
Read-only filegroups Be cautious of “two-step” processes Try avoid backup local then copy to share May not be possible to avoid in public cloud Stripe backups across multiple files Use backup compression Don’t recommend “file” backups, due to restoration difficulties Backup directly to a CIFS share Database compression does not negatively impact backup compression 10 | 9/17/2018 Managing Very Large Databases with SQL Server

11 Optimizing backups Best practices do not negate testing
Test backing up to NUL Our Typical Parameters MAXTRANSFERSIZE = 4MB BUFFERCOUNT (memory tolerance) BLOCKSIZE = 64k (discuss with storage team) Writer threads = number of backup files Reader threads = number of mount points / drives Backup to NUL determines theoretical throughput Internally in CA we use a buffercount of 200 Memory tolerance (xfer * max) = 800MB (outside bufferpool) Single reader thread per drive/mount point Single writer thread per backup file Ideal is 1:1 relationship with CPU cores (but consider any network bottleneck) 11 | 9/17/2018 Managing Very Large Databases with SQL Server

12 Restore Considerations
What do we typically need to restore? Alternate restore options Object-level recovery SQL database snapshots SAN snapshots Online piecemeal restore Far more likely to need to restore accidentally deleted or incorrectly updated data than restore an entire database to a point-in-time, overwriting existing data In many enterprises it is better to just fix the broken thing than start afresh Not uncommon to have a log shipped “delayed” copy of the data 12 | 9/17/2018 Managing Very Large Databases with SQL Server

13 Object-Level Recovery
3rd party tools only Recover objects directly from backups Run SQL queries/scripts against the backup EXEC = 'C:\MSSQL\Backup\MyDB_Backup.BAK' = 'SELECT * FROM dbo.Customers WHERE City=''London''' = 'dbo.CustomersInLondon' = 'MyDB' Useful for recovering from “oops” deletion events Oops deletion events should figure in your BC/DR plans 13 | 9/17/2018 Managing Very Large Databases with SQL Server

14 SQL Database Snapshots
Static copy of source database Copy-on-write Useful validation option Leverages NTFS sparse files By default CHECKDB uses snapshots You “revert” to a snapshot Not compatible with IMOLTP (2014) Transactionally consistent read-only static view of source database at point of creation Before a page is modified the original is copied to the snapshot (copy-on-write) Could be taken before deployments Instant creation (but must reside on same instance) Minimal space required (initially) CheckDB uses snapshots but same process can be undertaken manually. For highly fragmented NTFS volumes, CHECKDB may fail with error 665 but manual snapshot 14 | 9/17/2018 Managing Very Large Databases with SQL Server

15 SQL Database Snapshots
Static copy of source database Copy-on-write Useful validation option Leverages NTFS sparse files By default CHECKDB uses snapshots You can “revert” to a snapshot Does not replace backups Transactionally consistent read-only static view of source database at point of creation Before a page is modified the original is copied to the snapshot (copy-on-write) Could be taken before deployments Instant creation (but must reside on same instance) Minimal space required (initially) CheckDB uses snapshots but same process can be undertaken manually. For highly fragmented NTFS volumes, CHECKDB may fail with error 665 but manual snapshot Can be useful for QA/Dev testing 15 | 9/17/2018 Managing Very Large Databases with SQL Server

16 SAN snapshots “Application-aware” snapshots supported
Entry in MSDB backup history tables Seconds to backup and restore File layout / recovery model restrictions Recovery plans need SAN admin May be useful for “refreshing” QA/Dev Not SQL (do you have friends on the storage team?) SQL VSS Writer vs Windows VSS Writer Impact virtually unnoticeable even under heavy load (Zerto) – freeze IO warning LUNs need to be snapped at same time Equivalent to a full backup usually Vendors vary, only Snapmanager (NetApp) that I am aware of integrates log backups. Delphix slightly different. 16 | 9/17/2018 Managing Very Large Databases with SQL Server

17 Online Piecemeal Restore
Enterprise feature (online) Restore some filegroups while others remain online Must restore PRIMARY & In-memory FG Integrates with partitioning Filegroup with latest data can be brought online before anything else 2005+, assume everything in my talk is, I will identify newer features post-2005, I like to pretend SQL 2000 doesn’t exist because I never worked with it Allows the ability to bring databases online while some filegroups remain offline (recovery pending) For a VLDB with a lot of archive data Typical sequence Restore PRIMARY and any immediately required filegroups Take a tail log backup (do not use NORECOVERY option) Restore log sequence including tail log backup Restore archive filegroups and log sequence For read only filegroups no log sequence required 17 | 9/17/2018 Managing Very Large Databases with SQL Server

18 Restore strategy Design for restores ABT: Always Be Testing
Keep the PRIMARY filegroup small Filegroup design aligns to recovery needs Related objects should be kept in same filegroups ABT: Always Be Testing Fully automated if possible Script generation at a minimum Powershell Cmdlets available No user objects in PRIMARY filegroup Primary filegroup may still get big in 2016 due to Query Store Can be enforced using PBM or DDL triggers 18 | 9/17/2018 Managing Very Large Databases with SQL Server

19 Page Level Restores They rock! Page corruption
Can you wait? Do you have disk space? Do you have Enterprise Edition? Automatic page repair (Mirroring/AG) Recover from corruption without downtime Automatic page repair events should be monitored 19 | 9/17/2018 Managing Very Large Databases with SQL Server

20 Demo Time 20 | 9/17/2018 Managing Very Large Databases with SQL Server

21 Corruption Detection Integrity checks are expensive
Better detection outside of DBCC Checksum on pages Page scan for ECC checksum violations Checksum on tempdb (2008+) Combination of checksum and backup with checksum can detect errors outside page header Consistency checks still needed Running integrity checks on VLDBs is difficult where system is available 24x7 2005+ data purity, indexed view integrity in standard check SQL 2016 enhancements 9/17/2018 | Footer Goes Here

22 Optimizing CHECKDB May need PHYSICAL_ONLY option
Align with partitioning DBCC CHECKFILEGROUP Individual checks DBCC CHECKALLOC; DBCC CHECKCATALOG; DBCC CHECKTABLE Due to internal snapshot can be intensive for write-heavy workloads Use resource governor to limit memory to 1GB on systems with >256GB RAM for improved performance PHYSICAL_ONLY does not examine logical corruption SQL 2016 may be “fast enough” even with logical detection

23 Offload CHECKDB Restore to new location and run
DBCC CHECKDB (MegaBigVLDB) WITH NO_INFOMSGS, EXTENDED_LOGICAL_CHECKS, DATA_PURITY Advantage of checking backups and integrity Can be combined with SAN snapshots May double the amount of additional storage required False positive of catching corruption introduced by new storage Running CHECKDB against a AG secondary replica is not the same (unless used for backups) - Running CHECKDB against secondary is fine if you take your backups on the secondary

24 Index maintenance Fragmentation still matters regardless of storage
Index rebuilds will be expensive Online index rebuilds by partition SQL EE only Wait priority options in SQL 2014 Filtered indexes can be useful Fragmentation still matters Read ahead, page splits causing low page density, bloated memory, wasted disk space Indexes can be reorganized online by partition ETL inserts may cause fragmentation Online index rebuilds will need a LOT of tempdb/transaction log/data file space for sorting

25 Statistics maintenance
Requires manual maintenance Index rebuild “free lunch” not always available (2012+) Auto update statistics asynchronous SQL 2014 has incremental statistics SQL has filtered statistics SQL 2016 makes TF2871 default Monitor row modifications to update With VLDBs you may need manual processes to maintain statistics (use Ola Hallengren’s scripts) Dynamic sampling for online rebuilds of partitioned indexes in 2012+ For large partitioned indexes data distributions difficult to represent accurately with 200 step histogram Incremental statistics – update the partition and merge with the existing histogram Only update statistics if rows have been modified (sys.dm_db_stats_properties DMF) – use Ola Hallengren’s scripts

26 Partitioning EE only Divide large table into multiple B-Trees
Sliding window archival/ETL Limitations Cannot use composite key Difficult to change schema Tricky to align all indexes Partition elimination can be hit-or-miss Do not think of it as a performance feature - Do not ignore possibility of partitioned views (or combine PV and PTs)

27 Trace flags to consider
2371** 2562 2459 610 -E (startup) 834 Default (for me 2014) 3226, 1118**, 1117**, 4199** Default (for me 2016) 3226 * YMMV, ** Not needed in 2016 TF 2371, dynamically adjust threshold for statistics maintenance TF 2562 – optimized for physical_only TF 2459 – optimized for one file per drive 610 Allow minimally logged inserts into non-empty B-Tree -E (startup) Alters proportional fill algorithm to 64 extents before switching 1117 Expands all files in a filegroup when expansion is triggered 1118 Removes mixed extents so all extents become uniform extents 834 Enables large pages

28 HA / DR VLDBs may have more stringent requirements for HA/DR
Problems may take longer to resolve Understand your options Failover clustering Log shipping Mirroring Availability Groups AlwaysON FCI Test recovery times Log shipping Offsite / DR Replication Initialize from backup Mirroring Automatic page repair AlwaysON Availability Groups (2012+) Offload reporting and backups Migration option for upgrades

29 SQL 2016 “Enhanced” CHECKDB Truncate by partition
Ignore persisted computed columns, filtered indexes, UDT columns Truncate by partition GZIP Compression (COMPRESS) Batch mode Columnstore Stretch database

30 Coming up Next Awesome Raffle Prizes Free Beer and BBQ from 17:00


Download ppt "Managing Very Large Databases with SQL Server"

Similar presentations


Ads by Google