1. Implementing SQL Server in Large Storage Environments
Best Practices & Lessons Learned
© 2004 Hitachi Data Systems

2. Agenda Common questions/Pitfalls to avoid Performance/Monitoring
Host configuration considerations
Summarize: Key Takeaways
Question & Answers
© 2004 Hitachi Data Systems

3. Common Questions How do I optimize performance?
What are the most common pitfalls?
How do I monitor performance and, more importantly, how do I determine if performance is good or bad?
Is physical isolation needed between data/log/tempdb as was recommended with more traditional storage?
What are the considerations if using SQL Server clustering?
What do I need to think about when configuring the host server?
How do I manage growth?
What are my options for high availability?
© 2004 Hitachi Data Systems

4. Terminology Terminology Array
Cabinet containing the physical disks, cache, controllers, etc..
Consolidated storage resources for centralized management
Block level access between host server and storage over network (fiber)
Scalable performance with data integrity
DAS (Direct Attached Storage) vs. SAN (Storage Area Network)
SAN implies a switch is present
LUN (Logical Unit Number)
Logical grouping of pieces or entire physical disks exposed to the OS as a single logical device
HBA (Host Bus Adapter)
PCI card used to establish physical fiber channel connection to the array
Tutorials
© 2004 Hitachi Data Systems

5. Storage Area Networks (Architecture)
Cache
Fiber Channel Ports
Controllers/Processors
Switch
Host
Storage Array
© 2004 Hitachi Data Systems

6. Virtualized Storage
Pieces of many physical disks are combined to create a Logical Unit Number (LUN)
Multiple hosts can share the same underlying physical disks
Server A
Implications
-Each host may be connected to the same set of physical disks
-Important to understand the i/o characteristics of hosts which share the same physical disks
-Makes monitoring more challenging.
Server B
© 2004 Hitachi Data Systems

7. Performance (General)
There is no one “right” way to configure storage for optimal performance – this is dependent on SQL Server usage scenario
Size Matters! Design considerations for large databases can differ greatly from smaller database deployments or consolidated storage enviornments.
Understanding the I/O charcteristics and performance requirements of your SQL Server workload is key in determining the correct storage design.
General Guidelines
More/Faster Spindles is always better for performance.
Engaging the right engineers from the Vendors early in the design phase is key!
Try not to over optimize, simpler desings generally offer good performance and more flexiblity
Benchmark or performance test all storage configuration prior to deployment
© 2004 Hitachi Data Systems

8. SQL Servers I/O Characteristics
Understanding the I/O characteristics of common SQL Server operations/scenarios can help you determine how to configure storage
Some of the more common scenarios below (more detail on others in the appendix)
Monitor I/O to determine specifics of each scenario
Operation
Random / Sequential
Read / Write
Size Range
OLTP – Log
Sequential
Write
Up to 64K
OLTP – Data
Random
Read/Write 8K
Bulk Insert
Any multiple of 8K up to 128K
Read Ahead (DSS, Index Scans)
Read
Any multiple of 8KB up to 256K
**Note these values may change in new versions as optimizations are made to take advantage of modern storage enhancements.
© 2004 Hitachi Data Systems

9. Performance (RAID Level)
Best Practice: log files on RAID 1+0 disks
Best Practice: Isolate log from data at the physical disk level (more on isolation later)
Performance may benefit if Tempdb is placed on RAID 1+0 (dependent on Tempdb usage)
Our results indicate performance gain on RAID 1+0 for write intensive workloads but at a higher cost ($)
The performance difference between RAID 1+0 and RAID 5 can vary by vendor
Benchmarking of the storage can give a clear indication of the performance differences between RAID Levels before SQL Server is deployed
© 2004 Hitachi Data Systems

10. Performance (Storage Design)
How many and what size should my LUNs be?
LUN Size: Recommendation of the storage vendor balanced with the needs of the particular deployment
How many LUN’S?
There are some parallelism gains with SQL Server when multiple logical volumes (not necessarily LUNs) are used
Backup and Restore 1 Reader, 1 Writer thread per Logical Volume
Create Database 1 Thread per data file for initialization
May only be a consideration for very large databases
SQL Server 2005 alleviates this concern (data file created instantly, only log file requires initilization)
DBCC – Round Robin I/O with equal amount issued to each logical volume
Consider the number of processors on the host
Use of logical volumes can offer better granularity of monitoring
© 2004 Hitachi Data Systems

11. Performance (# Data Files/Groups)
How many data files/filegroups should I have?
More data files does not necessarily equal better performance – this is determined by hardware capacity & characteristics of SQL Server’s I/O
Disaster recovery requirements
Will the target environment for a disaster recovery restore accommodate the file sizes?
In a few, extreme cases, the number of data files may impact scalability
Consider the number of processors on the host server (mainly be concerned for >= 8 processors)
Best Practice: have .25 to .5 data files (per filegroup) for each CPU on the host server
Consider Tempdb usage and apply recommendation to it as well
Utilize maximum # of spindles - Multiple data files can be used to span as many underlying disks as necessary
Multiple filegroups may be optimal for backup / recovery scenarios of larger datasets
© 2004 Hitachi Data Systems

12. Performance (Creating Partitions)
Sector Align new partitions at creation time
Discuss the need and correct offset with your vendor
In our test lab the following has worked well
Start at an offset that is a power of two (and >32)
For EMC and most other arrays, 64 sector offsets are fine
Not required for HDS Storage Arrays
Sector alignment cannot be controlled using Windows Disk Manager
DISKPAR (note the missing t, available as part of the Windows 2000 Resource Kit) can be used to create ‘aligned’ partitions
New in Windows 2003 Server Service Pack 1
DiskPart.exe will contain a new /ALIGN option eliminating the need for Diskpar.exe
NTFS Allocation Unit Size
No impact on performance of accessing the file once on disk
Avoid values less than 8K (reduces risk of torn pages)
© 2004 Hitachi Data Systems

13. Isolation
Best practice: Always separate log from data files at the physical disk level
However, that said, this may not be practical or even possible in some deployments on large storage arrays (i.e. consolidation)
Dependent on specific configuration & I/O performance requirements of the deployment
For consolidation consider I/O characteristics and group similar I/O characteristics (i.e. all logs )
Combining heterogeneous workloads (workloads with very difference latency characteristics) can have negative effects on overall performance
Tempdb – unless you understand tempdb I/O characteristics it may be better to place tempdb on common disks with data and utilize more cumulative disks
© 2004 Hitachi Data Systems

14. Isolation (2)
Consolidation of different workload types (OLTP & DSS) results in higher disk latencies and reduced transactional throughput
© 2004 Hitachi Data Systems

15. Managing Growth Depends on features offered by storage array
Many newer storage arrays offer the ability to dynamically grow a LUN – consult with your storage vendor
Two types of growth
Capacity vs. Additional performance (more physical disks)
Windows perspective
Basic or Dynamic disks – Either can be expanded
Basic disks can be expanded using Diskpart.exe (see appendix for KB articles)
Changes to underlying LUNs are automatically recognized by Windows
© 2004 Hitachi Data Systems

16. Clustering Considerations
Mount Point Volume support for SQL clusters
Not supported for SQL Server 2000 Failover Clusters on either Windows 2000 or 2003
Support for mount point volumes will be added in SQL Server 2005 (requiring Windows 2003)
Refer to the Windows HCL to make sure the hardware has been certified
Other considerations for clustering
Microsoft Windows Clustering: Storage Area Networks
Also see the Additional References slide
© 2004 Hitachi Data Systems

17. Host Server Configuration
HBA placement
Hosts will have multiple PCI Buses, overloading a single bus can result in a bottleneck
Multipathing I/O software
Recommended to use this when multiple HBA’s are involved.
Simplifies the configuration and offers advantages for availability
Microsoft Multipath I/O (MPIO)
Microsoft now provides core multipathing functionality
Vendors build Device Specific Modules (DSM) on top of MPIO
Drivers and Firmware
Storport vs. SCSIport vs. Full Port?
Use drivers which have been qualified with storage
Storage vendor specific areas on the HBA manufactures website
Driver settings should come from the storage vendor (i.e. Queue Depth, etc…)
© 2004 Hitachi Data Systems

18. Monitoring Host: Performance Monitor & Fn_virtualfilestats
Establish ongoing monitoring plan and capture baseline data
Identify I/O problems
Fn_virtualfilestats provides I/O details on a per file basis
Storage Array: Vendor specific
Cache Utilization, Hotspots, Disk & Processor Utilization
Useful for troubleshooting & trend analysis
Used to determine root cause of I/O problems or if capacity of storage has been reached
Examples: HDS GraphTrack
© 2004 Hitachi Data Systems

19. Controllers/Processors
Monitoring
Understand potential throughput, know your hardware
Each component in the path has associated bandwidth
Know where the potential bottlenecks exist
Cache
Fiber Channel Ports
Controllers/Processors
Switch
Host
PCI Bus  HBA  Fiber Channel Ports  Array Processors  Disks
© 2004 Hitachi Data Systems

20. Monitoring (Perfmon I/O Counters)
Disk Reads/Writes per Second
Measures the Number of I/O’s per second (practical limit of 100/sec for a single 10K Rpm disk and 130/sec for a single 15K Rpm disk)
Average Disk/sec Read & Write
Measures disk latency. Numbers will vary (optimal values < 2-4 ms for Log, 4-15 ms for Data – OLTP, 25+ ms for DSS)
Average Disk Bytes/Read & Write
Measures the size of I/O’s being issued. Larger I/O tend to have higher latency
Average Disk Queue Length
Hard to interpret due to virtualization of storage. Ideal value <=2-4 per physical disk.
Disk Read & Write Bytes/sec
Measure of total disk throughput.
© 2004 Hitachi Data Systems

21. Testing Storage Configurations
Test and validate the performance of each storage configuration before deploying SQL Server application (common pitfall)
SQLIO.exe is an unsupported tool provided by Microsoft that can be used for this (see Tools slide in Appendix)
Benchmark performance and “shake out” hardware/driver problems early in the configuration
Test a variety of I/O types and sizes
Make sure your test files are significantly larger than the amount of cache on the array
Test each I/O path individually and then combinations of the I/O paths
Relatively short tests are okay, however, longer runs may give a more complete understanding of how the storage will perform
Allow time in between tests to allow the hardware to reset (cache flush)
Keep all of the benchmark data to refer to after the SQL implementation has taken place
Share the results with your vendor – good method for comparing different configurations
© 2004 Hitachi Data Systems

22. Remote Copy Between Arrays
Provides replication across storage arrays
HDS – TrueCopy
Get assurance from hardware vendor
“Write Dependency” will be honored
“Torn Pages” will be avoided following vendor provided best practices
Microsoft will provide support for non-storage device related issues; Storage related issues will be referred to the storage vendor for support
© 2004 Hitachi Data Systems

23. Snapshot Backup/Restore
Virtual Device Interface (VDI)
API designed to allow ISV to integrate SQL Server backup and restore operations in to their software.
Solutions which use VDI are provided by the storage vendor
More on this in the sample configurations section
Volume Shadow Copy Service (VSS)
New in Windows 2003
Inbox writer shipped with Windows 2003 providing limited support for SQL Server 2000
SQL Server 2005 will include a writer with additional support for VSS
© 2004 Hitachi Data Systems

24. Large Database Backup/Restore using VDI (HDS)
Backup and restore of large database running OLTP workload
Using HDS Shadow Image Copy & SplitSecond
Provides VDI backup/restore
Service offering provided by HDS
Currently called Splitsecond
Moving to a product named Protection Manager
3 copies of database maintained
Primary / Shadow Image A (SIA) / Shadow Image B (SIB)
5 LUNs in each shadow image group (4 data, 1 log)
HDS Command Control Interface (CCI) used to create mirrored pairs
PRIMARY
SIB
SIA
© 2004 Hitachi Data Systems

25. Large Database Backup/Restore using VDI (HDS)
SQL
HDS
SplitSecond
(1) Request backup
(2) Response ‘ready to freeze’
(3) Request – ‘freeze writes to DB’
(4) Response - ‘writes to DB frozen’
(6) Request - ‘Resume all I/O’
Only writes to the database and transaction log are frozen during the backup.
(5) Mirror pair split
© 2004 Hitachi Data Systems

26. Large Database Backup/Restore using VDI (HDS)
Approximately 8 seconds to backup ~500GB total data size.
© 2004 Hitachi Data Systems

27. Key Takeaways Engage the storage vendor early in the design
Get input on designing for growth & performance
Attempt to keep storage configurations as simple as practical
No single way to design for performance, this is dependent on the specifics of the scenario
Understand system I/O components capacities and potential bottlenecks
Benchmark and resolve any issues with I/O system prior to SQL Server application deployment
Monitor your I/O performance
Windows / SQL Tools to identify problem space and utilize storage vendor tools for debugging and resolution
© 2004 Hitachi Data Systems

28. More on SQL Server I/O Characteristics
Operation
Random / Sequential
Read / Write
Size Range
CREATE DATABASE
Sequential
Write
512KB
(Only log file is initialized in SQL Server 2005)
Backup
Read/Write
Multiple of 64K (up to 4MB)
Restore
DBCC – CHECKDB
Read
8K – 64K
DBCC – DBREINDEX (Read Phase)
(see Read Ahead)
DBCC – DBREINDEX (Write Phase)
Any multiple of 8K up to 128K
DBCC – SHOWCONTIG
© 2004 Hitachi Data Systems

29. Tools Diskpar.exe Windows 2000 Companion CD
SQLIO
Used to stress an I/O subsystem – Test a configuration’s performance
SQLIOSim
Simulates SQL Server I/O – Used to isolate hardware issues
HOW TO: Use the SQLIOStress Utility to Stress a Disk Subsystem
Fiber Channel Information Tool
Command line tool which provides configuration information (Host/HBA)
© 2004 Hitachi Data Systems

30. Additional References
Scalability and VLDB Resources on Microsoft.com
Troubleshooting Storage Area Network (SAN) Issues
Microsoft Windows Clustering: Storage Area Networks
Support for Multiple Clusters Attached to the Same SAN Device
How to Configure Volume Mount Points on a Clustered Server
INF: SQL Server support for mounted volumes
How to Extend the Partition of a Cluster Shared Disk
How to Use Diskpart.exe to Extend a Data Volume
Virtual Device Interface Specification
adfe15e850fc&DisplayLang=en
© 2004 Hitachi Data Systems

31. Additional References (2)
SQL Server Consolidation on the 64-Bit Platform
SQL Server Consolidation on the 32-Bit Platform using a Clustered Environment
Windows Server System Storage Home
Microsoft Storage Technologies – Multipath I/O
Windows 2003 Storport Dirver
us/storage/hh/storage/portdg_88563b10-e292-48f3-92de-65bf xml.asp
INF: Support for Network Database Files
© 2004 Hitachi Data Systems