SQL Server Optimization for Developers Anil Desai (Anil@AnilDesai.net | http://AnilDesai.net) Austin .NET User’s Group, 04/14/2014

Presentation Overview Database Performance Goals and Challenges Monitoring and Optimizing Performance Understanding indexes SQL Profiler and Database Engine Tuning Advisor Tuning Database Queries Understanding the Query Optimizer and Execution Plans Seeing the effects of indexes Application Design Best Practices

Presentation and Demo Notes For Developers, not database administrators No focus on backups, fault-tolerance, data protection, replication, security, etc. Demo Script Run SQL Load Simulator with various queries (need to write queries) Build WAIT FOR statements into the script Disable indexes on Demo schema tables? Use Profiler to capture trace data Use Database Tuning Advisor to test workloads

Performance Monitoring Process Best Practices: Optimize for real-world workloads Monitor/review performance regularly Focus on specific issues Establish a baseline Identify bottlenecks Make one change at a time Measure performance (if desired) Repeat

Monitoring and Troubleshooting Scenarios Server-Level Issues Users are reporting slow performance Intermittent transaction timeouts “The server/application seem sluggish” Application-Specific Issues “Application A is running more slowly than usual” “The End-of-Month report is taking too long to run.” Other Issues Ad-hoc reports are running slowly CPU, memory, disk, or network alerts are being generated

Common Datbaase Questions Which queries are taking the longest to run? Which queries are using the most system resources? Why does database/web site access slow down during particular times or during certain patterns of activity? Is indexing configured optimally for my live (production) database workloads? How will my development updates affect production performance? Which are the most cost-effective upgrades I can make to improve performance? How can I optimize a specific query?

SQL Server Editions SQL Server Compact Works with mobile devices (small footprint) Fewer features vs. full SQL Server SQL LocalDB Creates a "user instance" of SQL Server (no installation or services) Supports AttachDbFileName option More functionality vs. SQL Server Compact SQL Server Express Free version of SQL Server (up to 10GB database) SQL Server Standard, Enterprise, Business Intelligence, Developer

Database Performance Tools System/OS Performance Monitor Resource Monitor Task Manager SQL Server SQL Server Management Studio SQL Profiler / SQL Trace Database Engine Tuning Advisor Dynamic Management Views (DMVs) Query-Level Query Execution Plans

Useful SQL Server Performance Counters MCP TechMentor 2002 (San Diego) 9/3/2002 Useful SQL Server Performance Counters Backup Device Device throughput Bytes/sec Buffer Manager Buffer cache hit ratio Page reads / sec Buffer Partition Free Pages Cache Manager Cache Hit Ratio Databases Active Transactions Data File Size Log Growths Percent Log Used Transactions / sec General Statistics Logins | logouts / sec User Connections Latches Latch waits / sec Locks Lock requests / sec Lock waits / sec Memory Manager Everything! Replication Depends on configuration SQL Statistics Batch requests / sec SQL compilations / sec The key to managing SQL Server performance is an understanding of the various performance objects and counters that are available. Each of these statistics can provide a lot of info regarding how you server and applications are running. Anil Desai

Database File Structure SQL Server database file structure: Primary data file (.mdf) Secondary data files (*.ndf) Transaction log files (*.ldf) File groups Moving database files: Attaching and detaching databases Storage Goals: Reduce disk contention Optimize read/write performance

Database Design Issues Transaction processing (OLTP) Favors normalized schema Many tables, each with fewer columns Optimized for write (transactional) activity Reporting and Analysis Centralized, consistent storage of required data Favored by denormalized schema Fewer tables with many columns in each Data is aggregated from multiple sources into a data mart or data warehouse May store aggregates in warehouse

Understanding Indexes Index types Clustered Index Non-Clustered Indexes Columnstore indexes Indexing strategies Goal is ideal index coverage Index maintenance can slow-down write operations (Insert, Update, Delete) Referential Integrity Primary Key (default = clustered index) Foreign Key references Constraints Statistics (manual vs. automatic)

General Index Tuning Best Practices Make tuning a part of your development process Dev: Use synthetic workloads and test cases Test: Use real-world databases, whenever possible Production: Capture real usage statistics for analysis Collect a representative workload, whenever possible Consider all applications and workloads when tuning a database Use naming conventions for indexes and related objects Use query hints sparingly (NOLOCK)

Using SQL Profiler Purpose / Features: Examples: GUI for managing SQL Trace Monitor important events Capture performance data / resource usage Replaying of workloads / transactions Identifying performance bottlenecks Correlation of data with System Monitor Workloads for Database Tuning Advisor Examples: Generate a list of the 100 slowest queries Monitor all failed logins (Security)

Database Engine Tuning Advisor Automatic workload analysis for Physical Design Structures (PDS) Data Source: File (Profiler Trace or .SQL files) Table Plan Cache Tuning Options Keep existing PDS Advanced Features: Partitioning, indexed views, etc.

Demo: Optimizing Indexes Generate sample queries / tables View query execution plans View the effects of indexes on common queries Capture Performance Data with SQL Profiler SQL Profiler traces, events, and filters Using SQL Load Generator to generate database load Capturing and storing Analyzing and optimizing with Database Tuning Advisor Analyzing index usage reports Saving and applying index recommendations

Tuning Individual Queries Query Analyzer Features Execution Plan (estimated and actual) Include Client Statistics (multiple trials) Analyze in Database Engine Tuning Advisor (single query) Trace query in SQL Profiler (single query) Keep query logic close to the database Filter returned data at the database layer Minimize the size of result sets Minimize round-trips to the server Use standard (inner) joins, where possible Consider strategic denormalization for core sets of data

Query Optimizer Details Goal: Find the most efficient method to return the data Come up with a plan quickly Minimize CPU, memory, and I/O requirements Use statistics and index details to improve plans Query plan caching Relational engine vs. storage engine Execution Plan output Save as .sqlplan file for later analysis Output in graphical, text, and XML formats Can store and export plans using SQL Profiler (ShowPlan XML event) Can use query hints

Understanding Execution Plans Optimizing individual queries Rewrite query logic Use other objects (views, stored, procedures, etc.) Strategic demoralization Data Retrieval: Table scan, index seek/scan Index Usage Covering indexes Join conditions

Execution Plan Example

Execution Plans in Windows Azure

Client Statistics Example

Demo: Tuning Queries Viewing the effects of indexes on specific queries

Application Design Best Practices Create an abstraction layer between business and database objects ADO.NET Microsoft Enterprise Library NHibernate Entity Framework Use caching wherever possible Server-side (web services) Application-level (middle tier) Client-side (desktop or mobile apps) Minimize transaction times

Dev Best Practices: Application and Data Architecture Standards-based communications Loosely-coupled tiers and components Architecture Test harness Unit tests that use data Query performance monitoring Development / Testing Windows PerfMon Counters Instrumentation and Logging Alerts / pro-active corrections Auto-scaling Performance and Monitoring Error and event handling Transaction retry (random interval); Deadlock management User notifications (responsive UI / cancel and retry options) Reliability

Windows Azure and Cloud Databases Practical cloud benefits Data redundancy and geographic distribution Lower management overhead Potential issues Keeping data close to applications and services Data synchronization Network performance issues Data security, legal issues, and regulatory compliance Determine where/how to use cloud-based services SaaS vs. PaaS vs. IaaS

Azure Database Services SQL Azure Database Cost-effective, managed database instances Can be managed with standard tools (Visual Studio and SSMS) Some limitations (CLR, Mirroring, Partitioning, Replication, Extended SP’s) Other Services Azure Virtual Machines (SQL Server templates) Azure Web Sites (with gallery templates) Azure HDInsight, Cache Service Azure Backup and Recovery Manager SQL Azure Reporting Network, Active Directory, Service Bus, etc.

Managing SQL Azure Instances

ORM Considerations General issues Development efficiency vs. hardware/software efficiency Latency, query inefficiency (outer joins), platform-specific optimizations Frequency and number of server round-trips ORM-generated queries can be inefficient Difficult to tune or modify individual queries Potential Solutions Make sure entity relationships are correct Can use views or stored procedures to improve performance in some cases Bypass the ORM for some types of operations

New Features in SQL Server 2014 Memory-optimized tables (In-Memory OLTP) Buffer Pool Extension (for SSD usage) Delayed durability Async log writes can result in data loss Enable at database-level; use with BEGIN ATOMIC … COMMIT Resource Governor storage I/O limits Updateable Clustered ColumnStore indexes Primarily for data warehousing; supports data index compression Azure storage for SQL Server data/log files Backup to Azure; Backup encryption

Dev Best Practices: Managing Data Large UPDATE or DELETE operations: Use loops to minimize locking and transaction log growth Large INSERT operations Disable indexes and triggers (if present) Use BULK INSERT, bcp, SSIS, or DTS Change transaction isolation level (if appropriate) Change recovery model Use SQL to generate SQL Example: INSERT statements Schedule or delay non-critical operations

Dev Best Practices: Schema Changes Generate Scripts Script specific objects using SQL Server Management Studio Script the entire database using Generate Scripts Can include schema and/or data Schema changes Use ALTER commands when possible Drop and recreate objects, as needed Make all scripts re-runnable Check before and after state of all objects

Transaction Isolation Levels Balance of concurrency (performance) vs. consistency Affects SELECT queries SET TRANSACTION ISOLATION LEVEL Transaction Isolation Levels READ UNCOMMITTED READ COMMITTED (default) REPEATABLE READ SERIALIZABLE SNAPSHOT Row-Versioning: ALLOW_SNAPSHOT_ISOLATION READ_COMMITTED_SNAPSHOT

Dev Best Practices: Performance Testing Build performance testing/optimization into the dev process Develop load tests or test “harnesses” Using synthetic load generation tools Use representative test data Consider caching effects: Index maintenance (fragmentation) DBCC DropCleanBuffers DBCC FreeProcCache

Advanced Performance Approaches Database Federations Vertical and horizontal data partitioning Cross-Server queries Use Linked Servers to query across databases Potential performance issues Data compression (row- or page-level) Resource governor SQL Server Analysis Services (SSAS) Pre-aggregation for performance Dependent on a denormalized schema (optimized for reporting)

Links and References Presenter: Microsoft TechNet Virtual Labs http://AnilDesai.net | Anil@AnilDesai.net Presentation slides and sample code Microsoft TechNet Virtual Labs Sample Databases AdventureWorks Sample Databases (CodePlex) Microsoft Contoso BI Demo Dataset Database-related tools SQL Load Generator by David Darden (CodePlex) Glimpse Red Gate Software Spotlight

Summary and Conclusion