Presentation is loading. Please wait.

Presentation is loading. Please wait.

Storage Performance for Data Warehousing

Similar presentations


Presentation on theme: "Storage Performance for Data Warehousing"— Presentation transcript:

1 Storage Performance for Data Warehousing
Joe Chang © 2010 Elemental Inc. All rights reserved.

2 About Joe Chang SQL Server Execution Plan Cost Model
True cost structure by system architecture Decoding statblob (distribution statistics) SQL Clone – statistics-only database Tools ExecStats – cross-reference index use by SQL-execution plan Performance Monitoring, Profiler/Trace aggregation

3 Storage

4 Organization Structure
In many large IT departments DB and Storage are in separate groups Storage usually has own objectives Bring all storage into one big system under full management (read: control) Storage as a Service, in the Cloud One size fits all needs Usually have zero DB knowledge Of course we do high bandwidth, 600MB/sec good enough for you?

5 Data Warehouse Storage
OLTP – Throughput with Fast Response DW – Flood the queues for maximum through-put Do not use shared storage for data warehouse! Storage system vendors like to give the impression the SAN is a magical, immensely powerful box that can meet all your needs. Just tell us how much capacity you need and don’t worry about anything else. My advice: stay away from shared storage, controlled by different team.

6 Nominal and Net Bandwidth
PCI-E Gen 2 – 5 Gbit/sec signaling x8 = 5GB/s, net BW 4GB/s, x4 = 2GB/s net SAS 6Gbit/s – 6 Gbit/s x4 port: 3GB/s nominal, 2.2GB/sec net? Fibre Channel 8 Gbit/s nominal 780GB/s point-to-point, 680MB/s from host to SAN to back-end loop SAS RAID Controller, x8 PCI-E G2, 2 x4 6G 2.8GB/s Depends on the controller, will change!

7 Storage – SAS Direct-Attach
Many Fat Pipes Very Many Disks SAS x4 RAID PCI-E x8 . SAS x4 RAID PCI-E x8 Option A: 24-disks in one enclosure for each x4 SAS port. Two x4 SAS ports per controller Option B: Split enclosure over 2 x4 SAS ports, 1 controller . SAS x4 RAID PCI-E x8 . SAS x4 RAID PCI-E x8 . RAID Balance by pipe bandwidth PCI-E x4 SAS x4 PCI-E x4 2 x10GbE Don’t forget fat network pipes PCI-E x4 2 x10GbE

8 Storage – FC/SAN PCI-E x8 Gen 2 Slot with quad-port 8Gb FC
HBA PCI-E x8 Gen 2 Slot with quad-port 8Gb FC If 8Gb quad-port is not supported, consider system with many x4 slots, or consider SAS! SAN systems typically offer 3.5in 15-disk enclosures. Difficult to get high spindle count with density. disk enclosures per 8Gb FC port, 20-30MB/s per disk? PCI-E x8 . 8Gb FC HBA PCI-E x8 . 8Gb FC PCI-E x4 HBA . PCI-E x4 HBA 8Gb FC PCI-E x4 HBA 8Gb FC PCI-E x4 HBA 8Gb FC . PCI-E x4 HBA 8Gb FC PCI-E x4 2 x10GbE PCI-E x4 2 x10GbE

9 Storage – SSD / HDD Hybrid
No RAID w/SSD? SSD SAS x4 SAS PCI-E x8 . SSD SAS x4 SAS PCI-E x8 . SSD SAS x4 SAS . PCI-E x8 SSD SAS x4 SAS PCI-E x8 . PCI-E x4 RAID Storage enclosures typically 12 disks per channel. Can only support bandwidth of a few SSD. Use remaining bays for extra storage with HDD. No point expending valuable SSD space for backups and flat files SAS x4 PCI-E x4 2 x10GbE Log: Single DB – HDD, unless rollbacks or T-log backups disrupts log writes. Multi DB – SSD, otherwise to many RAID1 pairs to logs PCI-E x4 2 x10GbE

10 PDW Gigabit Ethernet Fiber Channel Infini-Band Compute Nodes Control
SP FC Compute Nodes Control SP FC SP FC Management SP FC Landing Zone SP FC Backup Node

11 ODM X2-2 IB SSD IB SSD Database Server Exadata Storage

12 2 x10GbE 2 x10GbE RAID SAS . . . . SAS x4 SAS x4 PCI-E x4 IB SSD SP FC

13

14

15 SSD Current: mostly 3Gbps SAS/SATA SDD
Some 6Gbps SATA SSD Fusion IO – direct PCI-E Gen2 interface 320GB-1.2TB capacity, 200K IOPS, 1.5GB/s No RAID ? HDD is fundamentally a single point failure SDD could be built with redundant components HP report problems with SSD on RAID controllers, Fujitsu did not?

16 Big DW Storage – iSCSI Are you nuts?
Well, maybe if you like frequent long coffee-cigarette breaks

17 Storage Configuration - Arrays
Shown: two 12-disk Arrays per 24-disk enclosure Options: between 6-16 disks per array SAN systems may recommend R or R5 7+1 Very Many Spindles Comment on Meta LUN

18

19 Data Consumption Rate: Xeon
TPC-H Query 1 Lineitem scan, SF1 1GB, 2k8 875M Processors Total Cores Q1 sec SQL MB/s per core GHz Mem GB SF 2 Xeon 5355 2 Xeon 5570 2 Xeon 5680 8 12 85.4 42.2 21.0 5sp2 8sp1 8r2 1,165.5 2,073.5 4,166.7 145.7 259.2 347.2 2.66 2.93 3.33 64 144 192 100 4 Xeon 7560 32 37.2 7,056.5 220.5 2.26 640 300 Nehalem Westmere Neh.-EX Conroe 8 Xeon 7560 183.8 14,282 223.2 512 3000 DCR for 16-way 6-core Xeon ,666MB/sec system level, 111MB/sec per core Data consumption rate is much higher for current generation Nehalem and Westmere processors than Core 2 referenced in Microsoft FTDW document. TPC-H Q1 is more compute intensive than the FTDW light query. © 2010 Elemental Inc. All rights reserved.

20 Data Consumption Rate: Opteron
TPC-H Query 1 Lineitem scan, SF1 1GB, 2k8 875M Processors GHz Total Cores Mem GB SQL Q1 sec SF Total MB/s MB/s per core 4 Opt 8220 2.8 8 128 5rtm 309.7 300 868.7 121.1 8 Opt 8360 2.5 32 256 8rtm 91.4 300 2,872.0 89.7 Barcelona 8 Opt 8384 2.7 32 256 8rtm 72.5 300 3,620.7 113.2 Shanghai 8 Opt 8439 48 49.0 8sp1 5,357.1 111.6 2.8 256 300 Istanbul 8 Opt 8439 48 166.9 8rtm 5,242.7 109.2 2.8 512 1000 2 Opt 6176 24 20.2 8r2 4,331.7 180.5 2.3 192 100 Magny-C 4 Opt 6176 48 31.8 8,254.7 172.0 512 300 - Expected Istanbul to have better performance per core than Shanghai due to HT Assist. Magny-Cours has much better performance per core! (at 2.3GHz versus 2.8 for Istanbul), or is this Win/SQL 2K8 R2?

21 Data Consumption Rate TPC-H Query 1 Lineitem scan, SF1 1GB, 2k8 875M
Processors Total Cores Q1 sec SQL MB/s per core GHz Mem GB SF 2 Xeon 5355 2 Xeon 5570 2 Xeon 5680 2 Opt 6176 8 12 24 85.4 42.2 21.0 20.2 5sp2 8sp1 8r2 1165.5 2073.5 4166.7 4331.7 145.7 259.2 347.2 180.5 2.66 2.93 3.33 2.3 64 144 192 100 4 Opt 8220 8 Opt 8360 32 309.7 91.4 5rtm 8rtm 868.7 2872.0 121.1 89.7 2.8 2.5 128 256 300 8 Opt 8384 8 Opt 8439 48 72.5 49.0 3620.7 5357.1 113.2 111.6 2.7 4 Opt 6176 31.8 8254.7 172.0 512 8 Xeon 7560 183.8 14282 223.2 2.26 3000 Barcelona Shanghai Istanbul Magny-C

22 Storage Targets Processors 2 Xeon X5680 4 Opt 6176 4 Xeon X7560
2U disk enclosure 24 x 73GB 15K 2.5in disks $14K, $600 per disk Processors 2 Xeon X5680 4 Opt 6176 4 Xeon X7560 8 Xeon X7560 Total Cores 12 48 32 64 BW Core 350 175 250 225 Target MB/s 4200 8400 8000 14400 PCI-E x8-x4 5 - 1 6 - 4 9 - 5 SAS HBA 2 4 6 11† Storage Units/Disks 2 - 48 4 - 96 Storage Units/Disks 4 - 96 Actual Bandwidth 5 GB/s 10 GB/s 15 GB/s 26 GB/s † 8-way : 9 controllers in x8 slots, 24 disks per x4 SAS port 2 controllers in x4 slots, 12 disk 24 15K disks per enclosure, 12 disks per x4 SAS port requires 100MB/sec per disk, possible but not always practical 24 disks per x4 SAS port requires 50MB/sec, more achievable in practice Think: Shortest path to metal (iron-oxide)

23 Your Storage and the Optimizer
Sequential IOPS 1,350 350,000 Model Optimizer SAS 2x4 Disks - 24 48 BW (KB/s) 10,800 2,800,000 “Random” IOPS 320 9,600 19,200 Sequential- Rand IO ratio 4.22 36.5 18.2 45,000 FC 4G 30 360,000 12,000 3.75 SSD 8 280,000 1.25 Assumptions 2.8GB/sec per SAS 2 x4 Adapter, Could be 3.2GB/sec per PCI-E G2 x8 HDD 400 IOPS per disk – Big query key lookup, loop join at high queue, and short-stroked, possible skip-seek. SSD 35,000 IOPS The SQL Server Query Optimizer make key lookup versus table scan decisions based on a 4.22 sequential-to-random IO ratio A DW configured storage system has a ratio, 30 disks per 4G FC about matches the QO, SSD is in the other direction

24 Data Consumption Rates
TPC-H SF100 Query 1, 9, 13, 21 TPC-H SF300 Query 1, 9, 13, 21

25

26 Fast Track Reference Architecture
My Complaints Several Expensive SAN systems (11 disks) Each must be configured independently $1,500-2,000 amortized per disk Too many 2-disk Arrays 2 LUN per Array, too many data files Build Indexes with MAXDOP 1 Is this brain dead? Designed around 100MB/sec per disk Not all DW is single scan, or sequential Scripting?

27 Fragmentation Weak Storage System
Table Weak Storage System 1) Fragmentation could degrade IO performance, 2) Defragmenting very large table on a weak storage system could render the database marginally to completely non-functional for a very long time. Powerful Storage System 3) Fragmentation has very little impact. 4) Defragmenting has mild impact, and completes within night time window. What is the correct conclusion? File Partition LUN Disk

28 Operating System View of Storage

29 Operating System Disk View
Controller 1 Port 0 Controller 1 Port 1 Disk 2 Basic 396GB Online Disk 3 Controller 2 Port 0 Controller 2 Port 1 Disk 4 Disk 5 Controller 3 Port 0 Controller 3 Port 1 Disk 6 Disk 7 Additional disks not shown, Disk 0 is boot drive, 1 – install source?

30 File Layout Each File Group is distributed across all data disks
Disk 2, Partition 0 File Group for the big Table File 1 Partition 1 File Group for all others Partition 2 Tempdb Partition 4 Backup and Load Disk 3 Partition 0 File 2 Small File Group Disk 4 Partition 0 File 3 Disk 5 Partition 0 File 4 Disk 6 Partition 0 File 5 Disk 7 Partition 0 File 6 Log disks not shown, tempdb share common pool with data

31 File Groups and Files Dedicated File Group for largest table
Never defragment One file group for all other regular tables Load file group? Rebuild indexes to different file group

32 Partitioning - Pitfalls
Disk 2 File Group 1 Disk 3 File Group 2 Disk 4 File Group 3 Disk 5 File Group 4 Disk 6 File Group 5 Disk 7 File Group 6 Table Partition 1 Table Partition 2 Table Partition 3 Table Partition 4 Table Partition 5 Table Partition 6 Common Partitioning Strategy Partition Scheme maps partitions to File Groups What happens in a table scan? Read first from Part 1 then 2, then 3, … ? SQL 2008 HF to read from each partition in parallel? What if partitions have disparate sizes?

33


Download ppt "Storage Performance for Data Warehousing"

Similar presentations


Ads by Google