16 Solution to what? Load Speed (ETL) Query Speed Data Management Backup / RestoreDBCC CHECKDB, remove Fragmentation16
17 Solutions Use Multiple FileGroups/Files Spread Data to maximize resource useSliding Window if there is a time dimensionPartitioned Tables and/or ViewsETL – Insert into empty unindexed tablesUse READ_ONLY FileGroups to minimize maintenance needs.
18 I/O Performance Little has changed in 50 years Watch out for bottlenecks in the I/O PathMemory reduces the need for I/ODisks can only do so many I/O operations per secondThe more disk heads you have the higher the I/O throughput.
19 At 3 PM on the 1st of the month: Where do you want your data to be? 19
20 Spread to as many disk resources as possible. 20
21 Sliding Window Always There Data Temporal Data 2008-01 Temporal Data TemporalDataTemporalDataTemporalData
22 Read_Only FileGroups Require only one Backup ALTER DATABASE <database> MODIFY FILEGROUP <filegroup> SET READ_ONLYRequire only one BackupDon’t require page or row locksDon’t require maintenanceThe ALTER requires exclusive access to the database before SQL 2008
23 Concern - Load Performance (ETL) 4 Hour maximum window for any loadLoad into large indexed tables is unacceptably long.Example: 2 million row insert into 400 million row table with 10 indexes took 12 hours.23
24 Concern – Query Performance Users have little patienceData warehouse QueriesFrequent small to medium to support UILess frequent large queries on fact tables may access 10’s of GB
25 Fact Table Queries Concentrated time period Most recentYear agoMay go against full table to get year-against-year25
26 Dimension Table Queries Smaller than fact table queriesSometimes involve millions of rowsFrequent – support the UI
28 Partitioned Views Available in SQL Server Standard Created like any viewCheck constraints tell SQL Server which data is in which tableCREATE VIEW Fact ASSELECT * FROM Fact_UNION ALL SELECT * FROM Fact_ALTER TABLE Fact_ ADD CONSTRAINT CK_FACT_ _Date CHECK (FactDate >= ‘ ’ and FactDate < ‘ ’28
29 Partitioned View - 2 Looks to a query like any table or view Can take advantage of parallel execution.Limited to 256 tablesCan cross servers (Performance Warning)SELECT FactDate, … FROM Fact WHERE CustID= AND FactDate = ‘ ’
30 Partitioned View SQL Server Storage View Fact Table1 Table2 FileGroupAFileGroupBFGF1FGF1FGF2FGF2FGF3FGF3FGF4FGF4FileA1FileB1FileB2F1F1F2F2F3F3F4F4Logical Disk System – Windows DrivesDrive C:Drive D:Drive E:Physical IO - subsystemDisk
31 Partition Elimination The query compiler can eliminate partitions from consideration in the planPartition elimination happens at query compile time.Values matching the partitioning column must be constants to allow partition elimination.
33 Partitioned Tables SQL Server Enterprise SQL Server 2005 and Above Require a non-null partitioning columnCheck constraints tell SQL Server what data is in each parturitionAll tables are partitioned!33
34 Partitioned Tables 2 Partition Function Partition Scheme Defines how to split dataPartition SchemeDefines where to store each range of dataCREATE Partitioned View Fact_PF(smalldatetime) RANGE RIGHT FOR VALUES (‘ ’, ‘ ’)CREATE PARTITION SCHEME Fact_PF AS PARTITION Fact_pf TO (PRIMARY, FG_ , FG_ )
35 Partitioned Table SQL Server Storage Table Fact Table1 Table2 Fact.$Partition=1Fact.$Partitoin=3Fact.$Partition=4Fact.$Partition=2FileGroupAFileGroupBFGF1FGF2FGF3FGF4FileA1FileB1FileB2F1F2F3F4Logical Disk System – Windows DrivesDrive C:Drive D:Drive E:Physical IO - subsystemDisk
37 Partitioning Goals Adequate Import Speed Maximize Query Performance Make use of all available resourcesData ManagementMigrate data to cheaper resourcesDelete old data easily37
38 Achieving Load Speed Insert into empty tables Index and add foreign keys after the insertAdd the Slices toPartitioned ViewsPartitioned Tables38
39 Achieving Query SpeedEliminate access to partitions during query compileAll disk resources should be usedParallel accessAll available memory should be usedAll available CPUs should be usedParallel query39
40 Solution Partition at a sufficiently high grain Spread dimension data to all useable disksSeparate Data and Index FileGroupsMultiple files per FileGroupSpread Fact data by partition key to all useable disksRotate file locations to maximize dispersion40
41 Concern – Data Management (Backup) Let’s say you have a 10 TB database.Now back that up.
42 Backup Calculation 10 TB = 10000 GB Typical Backup speed Low end 1 GB per minuteHigh end 10 GB per minuteAt 10 GB/MinuteWho’s got minutes?
43 Achieving Backup Performance Backup less!Maintain data in a READ_ONLY stateCompress Backups
44 Partial Backup Partial Base Partial Differential Backs up read_write filegroupsPartial DifferentialDifferential backup of read_write filegroupsBACKUP DATABASE <db name>READ_WRITE_FILEGROUPS …..BACKUP DATABASE <db name>READ_WRITE_FILEGROUPSWITH DIFFERENTIAL ….
46 SQL Server 2008 – What’s New Row, page, and backup compression Filtered IndexesOptimization for star joinsMERGE T-SQL DMLResource GovernorFewer operations require exclusive access to the database
47 New England Visual Basic Pro Focused on VB.Net developmentMS Waltham – MPR C1st Thursday - 6:15 to 8:30 PMSept 4 – Jim O’Neil – ASP.Net Dynamic DataSept 25 – Chris Hammond – DotNetNukeOct 2 – Kathleen Dollard – XML Litterals in VB 9Nov 6 – Joe Stagner – Stupid Hacker Tricks and How 2 DefendFeb 5 ’09 – Joe Hill – Novell – Mono/VB/etc….
48 Thanks for Coming Andrew Novick anovick@NovickSoftware.com