Presentation is loading. Please wait.

Presentation is loading. Please wait.

VERY LARGE DATABASES

Similar presentations


Presentation on theme: "VERY LARGE DATABASES"— Presentation transcript:

1 VERY LARGE DATABASES ADMINISTRATION @murilocmiranda http://www.sql.pt/ murilo.miranda@gmail.com

2 AGENDA

3 1.What is a VLDB? 2.Typical Troubles 3.OS Config 4.Instance Config 5.DB Config 6.Maintenance

4 VLDB??

5 Theres no official definition.

6 VLDB?? Theres no official definition. Typically occupying TB range.

7 VLDB?? Theres no official definition. Typically occupying TB range. Billions of rows.

8 VLDB?? Theres no official definition. Typically occupying TB range. Billions of rows. Typically: OLAP or OLTP with large amount of users.

9 VLDB?? A very large database, or VLDB, is a database that contains an extremely high number of tuples (database rows), or occupies an extremely large physical filesystem storage space. The most common definition of VLDB is a database that occupies more than 1 terabyte or contains several billion rows, although naturally this definition changes over time. Wikipedia…

10 SQL VS. VLDB

11 Maximum database size

12 SQL VS. VLDB Maximum database size 524,272 TB

13 SQL VS. VLDB Maximum data file size 16 TB Maximum log file size 2 TB A limit of 32.767 files which can be distributed between 32.767 filegroups.

14 TYPICAL TROUBLES

15 Maintenance

16 TYPICAL TROUBLES Maintenance Backups

17 TYPICAL TROUBLES Maintenance Backups Indexes

18 TYPICAL TROUBLES Maintenance Backups Indexes Statistics

19 TYPICAL TROUBLES Maintenance Backups Indexes Statistics Disaster Recovery

20 TYPICAL TROUBLES Maintenance Backups Indexes Statistics Disaster Recovery Performance

21 OS CONFIG

22 Perform Volume Maintenance

23 OS CONFIG Turning on Instant Initialization to speed up data file growth and restores.

24 OS CONFIG Storage Layout

25 OS CONFIG Plan an efficient storage layout.

26 OS CONFIG Plan an efficient storage layout. Normally, the more spread, the more effective.

27 OS CONFIG Plan an efficient storage layout. Normally, the more spread, the more effective. Suggestion: SQL BINSQL DATASQL IDXSQL LOGSSQL TMP

28 OS CONFIG Mountpoints

29 OS CONFIG Mountpoints could be a good strategy.

30 OS CONFIG Mountpoints could be a good strategy. Mountpoints are persistent directories that point to disk volumes.

31 OS CONFIG Pros: Scalable. Save drive letters (limited to 26). Easy to add. No need to restart SQL Server.

32 OS CONFIG Cons: Looks like a simple folder. Need a different approach to monitor.

33 OS CONFIG So, if you dont know the server….

34 OS CONFIG Partition Alignment

35 OS CONFIG Setting the partition offset properly can improve up to 30% the performance.

36 OS CONFIG Setting the partition offset properly can improve up to 30% the performance. Partition alignment increases throughput (bytes/sec) and reduce disk queues.

37 OS CONFIG Setting the partition offset properly can improve up to 30% the performance. Partition alignment increases throughput (bytes/sec) and reduce disk queues. A partition that is track misaligned will occasionally cause 2 I/O operations instead of one.

38 OS CONFIG Unless performed at the time of partition creation, the default alignment offset (31,5 Kb) will result in unaligned partitions on versions of Windows up to and including Windows Server 2003.

39 OS CONFIG This offset is associated with hidden sectors, which basically store partition information.

40 OS CONFIG This offset is associated with hidden sectors, which basically store partition information. Considering that: -Each disk sector has 512 bytes. -Win. 2003 has 63 hidden sectors.

41 OS CONFIG This offset is associated with hidden sectors, which basically store partition information. Considering that: -Each disk sector has 512 bytes. -Win. 2003 has 63 hidden sectors. 512 * 63 = 31,5 Kb

42 OS CONFIG Example: Stripe Unit Size: 64Kb* Allocation Unit Size: 64Kb * Defined by storage team. Optimal values

43 OS CONFIG Example: Stripe Unit Size: 64Kb* Allocation Unit Size: 64Kb * Defined by storage team. Optimal values Stripe Size Data (Alloc. Unit Size)

44 OS CONFIG Optimal solution: Stripe Size Data (Alloc. Unit Size)

45 OS CONFIG Best Practice: -Set an offset of 1024 Kb. -This value works for mostly disks out there. -Allocation Unit Size = Stripe Unit Size. The rule: Offset / Allocation unit = INTEGER Eg: 1024/64=16

46 Some I/O subsystem vendors intercepting what Windows is trying to do and are still creating partitions with the incorrect offset – Even for Windows 2008+. WARNIG ALWAYS check!

47 OS CONFIG Anti-Virus in servers… is really a need?

48 OS CONFIG Cost money to license. Maintenance costs. Can cause problems in Prod. Cant protect to zero-day exploits.

49 OS CONFIG What can we do instead?

50 OS CONFIG Keep the servers patched. Configure the firewall properly. Restrict servers access. You can install AV… in workstations!

51 OS CONFIG Whats the big problem for SQL Server?

52 OS CONFIG One more app fighting for resources. SQL Server files can be locked.

53 OS CONFIG How can AV and SQL Server live together?

54 OS CONFIG Add exceptions!

55 OS CONFIG Basically the AV should ignore: SQL Server data and log files (.mdf,.ndf and.ldf). Backup files (.bak and.trn). Full-text Catalog files. Trace files (.trc). ERRORLOG files. SQL Server binaries folder. Filestream folder. More on: http://support.microsoft.com/kb/309422

56 INSTANCE CONFIG

57 Memory

58 INSTANCE CONFIG Memory This is a very open subject.

59 INSTANCE CONFIG Memory This is a very open subject. There are lots of discussions about that…

60 INSTANCE CONFIG Memory This is a very open subject. There are lots of discussions about that… Theres no perfect formula, because the correct awnser is….

61 INSTANCE CONFIG Memory This is a very open subject. There are lots of discussions about that… Theres no perfect formula, because the correct answer is…. … it depends !!

62 INSTANCE CONFIG Memory Baseline: 1 GB for the OS Up to 16 GB available 1 GB for each 4 GB More than 16 GB 1 GB for every 8 GB An efficient general rule…

63 INSTANCE CONFIG Memory This is for 64 bit servers… For 32 bit, here is a good article to follow: http://www.eraofdata.com/understanding-and-configuring-sql-servers-memory-settings/

64 INSTANCE CONFIG TempDB

65 INSTANCE CONFIG TempDB Two common behaviors:

66 INSTANCE CONFIG TempDB Two common behaviors: Ignore. Overvalue.

67 INSTANCE CONFIG TempDB As per Brent Ozar: TempDb is the SQLs public toilet

68 INSTANCE CONFIG TempDB And this is true!

69 INSTANCE CONFIG TempDB

70 INSTANCE CONFIG TempDB Theres a myth: tempdb should always have one data file per processor core.

71 INSTANCE CONFIG TempDB Theres a myth: tempdb should always have one data file per processor core. Again….

72 INSTANCE CONFIG TempDB Theres a myth: tempdb should always have one data file per processor core. Again…. It depends!

73 INSTANCE CONFIG TempDB Execute large operations, like a sort or store a huge temporary table, may be slowed down because of the round-robin operation. The more files, the more costly.

74 INSTANCE CONFIG TempDB Common wait types on TempDB: PAGELATCH_*: Contention for In-memory allocation bitmaps. PAGEIOLATCH_*: Contention at the I/O subsystem level.

75 INSTANCE CONFIG TempDB How many tempdb data files should we have?

76 INSTANCE CONFIG TempDB How many tempdb data files should we have? A recommended approach is: Up to 8 cores: Number of files = Number of cores. More than 8 cores: 1.Add 8 files. 2.Monitor PAGELATCH_*. 3.Add 4 more files at a time, if necessary.

77 INSTANCE CONFIG TempDB Other TempDB best practices: Isolate the TempDB in a different storage system. Depending of the load, you might need to separate LDF and M(N)DF. Use a fast drive (SSD :). Set an initial size, equally to all the files. Set the auto-growth accordingly. If you have a heavy operation using constantly the TempDB, consider create a staging table into your own database.

78 INSTANCE CONFIG TempDB From SQL Server 2012, local disk TempDB in SQL Server cluster.

79 INSTANCE CONFIG TempDB From SQL Server 2012, local disk TempDB in SQL Server cluster. More flexibility. Use PCIe bus instead of HBA, and have more throughput. Data and Log are in SAN, TempDB locally: Avoid congestion or contention on a shared storage network or array.

80 DB CONFIG

81 Dont rely on auto-grow. You can manage file growth and control the free disk space and avoids performance problems.

82 DB CONFIG Dont rely on auto-grow. You can manage file growth and control the free disk space and avoids performance problems. Have page checksums turned on. To detect damaged pages.

83 DB CONFIG Dont rely on auto-grow. You can manage file growth and control the free disk space and avoids performance problems. Have page checksums turned on. To detect damaged pages. Make sure auto-stats update is turned on. For OLTP consider turning auto-stats update off only for heavily updated tables, and schedule a job that periodically updates the statistics for those tables.

84 DB CONFIG

85 Make sure youre managing the transaction log correctly: Full recovery requires log backups. No advantage in have multiple log files. Control the file growth or this could cause VLF fragmentation. Performance issues. Slow backup time. Dont set the log file growth size to a multiple of 4 in older SQL Server versions. http://connect.microsoft.com/SQLServer/feedback/details/481594/log-growth-not-working- properly-with-specific-growth-sizes-vlfs-also-not-created-appropriately

86 MAINTENACE

87 MAINTENANCE Few questions…

88 MAINTENANCE Is data-loss acceptable? What about the recovery time? How to meet your SLAs dealing with a TB database? Are you able to UPDATE STATS, do INDEX MAINTENANCE and run a INTEGRITY CHECK in time and WITHOUT PROBLEMS?

89 MAINTENANCE DISASTER RECOVERY

90 MAINTENANCE First of all, think in a Disaster Recovery plan! SQL Server is not Oracle, we have free included options: Log Shipping (HA and DR) Database Mirroring (HA and DR) DB Snapshot advantage Replication (HA, DR and LB) AlwaysOn (HA, DR and LB) We can still be safe with a storage level replication.

91 MAINTENANCE Partition Compress Clean

92 MAINTENANCE Partition, Compress and Clean Using the partitioning feature you can devise the maintenance.

93 MAINTENANCE Partition, Compress and Clean Using the partitioning feature you can devise the maintenance. You can use the DBCC CHECKFILEGROUP command. DBCC CHECKFILEGROUP and DBCC CHECKDB are. The main difference is that DBCC CHECKFILEGROUP is limited to the single specified filegroup and required tables.

94 MAINTENANCE Partition, Compress and Clean Using the partitioning feature you can devise the maintenance. Devising a filegroup architecture allows piecemeal restores with low TTR Online piecemeal restore: After the PRIMARY FG restore the DB can be online. The tables will come available while each FG is restored. Design the database accordingly: Keep the necessary into the PRIMARY FG. Configuration tables, indispensable data, etc… Think in the consistency: keep related tables in the same FG.

95 MAINTENANCE Partition, Compress and Clean Compress backups Vs. Compress Data Backup compression: More CPU usage to backup/restore (avg ~20%). Less time to backup/restore (avg ~40%). Good compression ratio. SELECT backup_size/compressed_backup_size FROM msdb..backupset; A backup set will not be able to contain both compressed and uncompressed backups. No advantage with TDE enabled.

96 MAINTENANCE Partition, Compress and Clean Compress backups Vs. Compress Data Data compression (ROW and PAGE): One time operation. Reduce the physical database size. Reduce the I/O required for a workload. Allows more data to be stored in the buffer cache. More CPU usage. Usually good for DW systems For OLTP may also benefit. FILESTREAM data is not compressed.

97 MAINTENANCE Partition, Compress and Clean Compress backups Vs. Compress Data Data compression (ROW and PAGE): TDE and Data Compression play together! Backup and Data Compression can coexist!

98 MAINTENANCE Partition, Compress and Clean Compress backups Vs. Compress Data Data compression (ROW and PAGE): ROW or PAGES compression? You can use SQL Server Compression Estimator

99 MAINTENANCE Partition, Compress and Clean Purge and Archive the data Purging data: If data is needed no more… Save storage. Faster backups. Improves the performance.

100 MAINTENANCE Partition, Compress and Clean Purge and Archive the data Archiving data: If data is still needed… Isolate in a different FG. Set as Read-Only: Avoids locking. For faster scans: 100% fill factor. Update statistics with FULLSCAN. You can adapt the backup strategy. You can adapt the backup strategy using Partial Backups. This allows you to exclude read-only filegroups.

101 MAINTENANCE More about DBCC CHECKDB CHECKDB takes time and uses resources. Run a DBCC CHECKDB using the WITH PHYSICAL_ONLY option. Limits the checking to the integrity of the physical structure of the page and record headers and the allocation consistency of the database. Faster, but a full CHECKDB is required periodically.

102 MAINTENANCE More about DBCC CHECKDB We can divide up the consistency checking over several days, Paul Randals prescription is: Divide tables in two buckets (bigger ones and the rest) On Sunday: Run a DBCC CHECKALLOC Run a DBCC CHECKCATALOG Run a DBCC CHECKTABLE on each table in the first bucket On Monday, Tuesday, Wednesday: Run a DBCC CHECKTABLE on each table in the 2nd, 3rd, 4th buckets, respectively On Thursday: Run a DBCC CHECKALLOC Run a DBCC CHECKTABLE on each table in the 5th bucket On Friday and Saturday: Run a DBCC CHECKTABLE on each table in the 6th and 7th buckets, respectively More on: http://www.sqlskills.com/blogs/paul/checkdb-from-every-angle-consistency-checking-options-for-a-vldb/

103 MAINTENANCE More about BACKUPS Besides doing PARTIAL BACKUPS we have more options… A MULTISTREAM BACKUP is an option to run faster: DB File 1 File 2 File 3 E: G: F:

104 MAINTENANCE More about BACKUPS To make sure it will be well stored, we can use a MIRROR. DB File 1 File 2 File 3 E: G: F: File 1 File 2 File 3

105 MAINTENANCE More about BACKUPS If storing to the network: Use a separate network card to avoid network congestion. Dont forget about T-LOG backups! Create a good backup strategy. Verify the backups periodically.

106 MAINTENANCE INDEXES MAINTENANCE Only rebuild/defrag indexes that are really fragmented (avoid unnecessary work in short maintenance windows) If you defrag instead of rebuild, make sure you manually update stats. Be wary of doing large index maintenance jobs if you use log shipping or DBM They contribute to large log backups Index rebuilds are always full-logged when DBM is present

107 QUESTIONS?

108 OBRIGADO! @murilocmiranda http://www.sql.pt/ murilo.miranda@gmail.com


Download ppt "VERY LARGE DATABASES"

Similar presentations


Ads by Google