1. Does Your Performance Tuning Need a 12-step Program?
Janis Griffin
Senior DBA / Performance Evangelist

2. Senior DBA / Performance Evangelist for Solarwinds
Who Am I?
Senior DBA / Performance Evangelist for Solarwinds
Twitter
Current – 25+ Years in Oracle, SQL Server, MySQL
DBA and Developer
Specialize in Performance Tuning
Review Database Performance for Customers and Prospects
Common Thread – How do I tune it? 2

3. Utilize Response Time Analysis (RTA) 12 Steps To Follow
Agenda
Challenges Of Tuning
Which SQLs to tune
Utilize Response Time Analysis (RTA)
Wait Types / Wait Time
12 Steps To Follow
Several Case Studies 3

4. Requires Expertise in Many Areas
Challenges Of Tuning
SQL Tuning is Hard
Who should tune – DBA or Developer
Requires Expertise in Many Areas
Technical – Plan, Data Access, SQL Design
Business – What is the Purpose of SQL?
Tuning Takes Time
Large Number of SQL Statements
Each Statement is Different
Low Priority in Some Companies
Vendor Applications
Focus on Hardware or System Issues
Never Ending 4

5. Image courtesy of Gentle-Stress-Relief.com5

6. 1. Find Which SQL to Tune Methods for Identifying
User / Batch Job Complaints
Known Poorly Performing SQL
Trace Session/Process
Queries Performing Most I/O (Logical I/O, Physical I/O)
Table or Index Scans
Queries Consuming CPU
Highest Response Times (RTA)
DPA (formally Ignite) 6

7. Response Time Analysis (RTA)
Understand the total time a Query spends in Database
Measure time while Query executes
SQL Server helps by providing Wait Types
Focus on Response Time 7 7

8. Wait Time Tables (DMVs - 2005 & Up)
2014
dm_exec_requests
start_time
status
sql_handle
plan_handle
start/stop offset
database_id
user_id
blocking_session
wait_type
wait_time
dm_os_wait_stats
wait_type
waiting_tasks_count
wait_time_ms
2000 only
sysprocesses
loginame
hostname
programname
spid
dbid
waittype
waittime
Lastwaittype
waitresource
sql_handle
stmt_start
stmt_end
cmd
dm_exec_sessions
login_time
login_name
host_name
program_name
session_id
dm_exec_sql_text
text
dm_exec_query_stats
execution_count
total_logical_writes
total_physical_reads
total_logical_reads
dm_exec_query_plan
query_plan 8

9. Wait Type DMV Problem No Time Based View Need to use other tools
What happened between 3am-5am this morning
not possible to get from the DMV objects
Need to use other tools
Run example query (on next slide), include timestamp and save in table
Extended Events Session to gather waits and query results
System_health session gathers SQLs & Waits – hard to see trends
Solarwinds Database Performance Analyzer (DPA)
does a great job of trending both SQLs and Waits
Different DMV Problem 9

10. View Wait Types at Session Level
sys.dm_exec_requests
Real-time view into what each session/SQL is waiting on
Status Column tells you what state it’s in
Suspended - means the session is waiting on the wait_type
Running/Runnable - means the session is on the CPU or in queue
Sleeping - means the session is idle
SELECT r.session_id, r.wait_time, r.status, r.wait_type, r.blocking_session_id,
s.text, r.statement_start_offset, r.statement_end_offset, p.query_plan
FROM sys.dm_exec_requests r
OUTER APPLY sys.dm_exec_sql_text (r.sql_handle) s
OUTER APPLY sys.dm_exec_text_query_plan (r.plan_handle, r.statement_start_offset, r.statement_end_offset) p
WHERE r.status <> ’background’ AND r.status <> ’sleeping’ AND r.session_id <> 10

11. Top Wait Types at Instance Level
sys.dm_os_wait_stats
Cumulative since instance startup
DBCC SQLPERF (N'sys.dm_os_wait_stats', CLEAR);
Exclude idle wait types
Query on next slide 11

12. Note: 2012 Idle Waits Only Query for Top Waits
WITH Waits AS (SELECT wait_type, wait_time_ms / AS WaitS (wait_time_ms - signal_wait_time_ms) / AS ResourceS, signal_wait_time_ms / AS SignalS, waiting_tasks_count AS WaitCount, * wait_time_ms / SUM (wait_time_ms) OVER() AS Percentage, ROW_NUMBER() OVER(ORDER BY wait_time_ms DESC) AS RowNum FROM sys.dm_os_wait_stats WHERE wait_type NOT IN ( N'BROKER_EVENTHANDLER',N'BROKER_RECEIVE_WAITFOR',N'BROKER_TASK_STOP',N'BROKER_TO_FLUSH', N'BROKER_TRANSMITTER',N'CHECKPOINT_QUEUE', N'CHKPT',N'CLR_AUTO_EVENT', N'CLR_MANUAL_EVENT',N'CLR_SEMAPHORE', N'DBMIRROR_DBM_EVENT',N'DBMIRROR_EVENTS_QUEUE',N'DBMIRROR_WORKER_QUEUE',N'DBMIRRORING_CMD',N'DIRTY_PAGE_POLL', N'DISPATCHER_QUEUE_SEMAPHORE',N'EXECSYNC',N'FSAGENT',N'FT_IFTS_SCHEDULER_IDLE_WAIT',N'FT_IFTSHC_MUTEX', N'HADR_CLUSAPI_CALL',N'HADR_FILESTREAM_IOMGR_IOCOMPLETION',N'HADR_LOGCAPTURE_WAIT',N'HADR_NOTIFICATION_DEQUEUE', N'HADR_TIMER_TASK',N'HADR_WORK_QUEUE',N'KSOURCE_WAKEUP',N'LAZYWRITER_SLEEP',N'LOGMGR_QUEUE',N'ONDEMAND_TASK_QUEUE’, N'PWAIT_ALL_COMPONENTS_INITIALIZED',N'QDS_PERSIST_TASK_MAIN_LOOP_SLEEP', N'QDS_CLEANUP_STALE_QUERIES_TASK_MAIN_LOOP_SLEEP',N'REQUEST_FOR_DEADLOCK_SEARCH',N'RESOURCE_QUEUE', N'SERVER_IDLE_CHECK',N'SLEEP_BPOOL_FLUSH',N'SLEEP_DBSTARTUP',N'SLEEP_DCOMSTARTUP', N'SLEEP_MASTERDBREADY',N'SLEEP_MASTERMDREADY',N'SLEEP_MASTERUPGRADED',N'SLEEP_MSDBSTARTUP', N'SLEEP_SYSTEMTASK',N'SLEEP_TASK',N'SLEEP_TEMPDBSTARTUP',N'SNI_HTTP_ACCEPT', N'SP_SERVER_DIAGNOSTICS_SLEEP',N'SQLTRACE_BUFFER_FLUSH',N'SQLTRACE_INCREMENTAL_FLUSH_SLEEP', N'SQLTRACE_WAIT_ENTRIES',N'WAIT_FOR_RESULTS',N'WAITFOR',N'WAITFOR_TASKSHUTDOWN', N'WAIT_XTP_HOST_WAIT',N'WAIT_XTP_OFFLINE_CKPT_NEW_LOG',N'WAIT_XTP_CKPT_CLOSE',N'XE_DISPATCHER_JOIN', N'XE_DISPATCHER_WAIT',N'XE_TIMER_EVENT') AND waiting_tasks_count > 0) SELECT MAX (W1.wait_type) AS WaitType, CAST (MAX (W1.WaitS) AS DECIMAL (16,2)) AS Wait_S, CAST (MAX (W1.ResourceS) AS DECIMAL (16,2)) AS Resource_S, CAST (MAX (W1.SignalS) AS DECIMAL (16,2)) AS Signal_S, MAX (W1.WaitCount) AS WaitCount, CAST (MAX (W1.Percentage) AS DECIMAL (5,2)) AS Percentage, CAST ((MAX (W1.WaitS) / MAX (W1.WaitCount)) AS DECIMAL (16,4)) AS AvgWait_S, CAST ((MAX (W1.ResourceS) / MAX (W1.WaitCount)) AS DECIMAL (16,4)) AS AvgRes_S, CAST ((MAX (W1.SignalS) / MAX (W1.WaitCount)) AS DECIMAL (16,4)) AS AvgSig_S FROM Waits AS W1 INNER JOIN Waits AS W2 ON W2.RowNum <= W1.RowNum GROUP BY W1.RowNum HAVING SUM (W2.Percentage) - MAX (W1.Percentage) < 95;
Note: 2012 Idle Waits Only

13. RTA Benefits

14. Wait Types – give great Clues14

15. 2. Gather Run-Time Details
Get baseline metrics
How long does it take now
What is acceptable (10 sec, 2 min, 1 hour)
Record Logical Reads for tuning comparison
Collect Wait Type Information
Locking / Blocking (LCK)
I/O problem (PAGEIOLATCH)
Latch contention (LATCH)
Network slowdown (NETWORK)
May be multiple issues
All have different resolutions 15

16. Baseline Metrics 11 11 64% is spent on a Memory/CPU,
26 % on pageiolatch_sh

17. 3. Get Execution Plan SQL Server Management Studio (SSMS)
SET SHOWPLAN_XML { ON | OFF }
3. Get Execution Plan
SQL Server Management Studio (SSMS)
Estimated Execution Plan - can be wrong for many reasons
SET SHOWPLAN_XML { ON | OFF }
Actual Execution Plan – must execute query
Can be dangerous in production / Wrong in test ?
SET STATISTICS XML { ON | OFF }
SQL Server Profiler Tracing / Extended Events
Performance: Execution Plan (2000)
Showplan XML (2005+)
Works when you know a problem will occur
DM_EXEC_QUERY_PLAN
Cached execution plan of executed query 17

18. DM_EXEC_QUERY_PLAN Example

19. 4. Examine the Execution Plan
Find Expensive Operators
Examine CPU / IO costs, row counts
Look for full table or clustered index scans
Review the Predicate Information
Know how parameters are being interpreted
Review the data types / Implicit conversions?
Know which step filtering predicate is applied
Review Join Methods
Nested Loops join: Usually efficient for smaller data sets
Merge Join: Efficient for larger data sets.
Hash Match join: Useful on very large data sets (DW) 19

20. Review Logical IO 20

21. Execution Plan 21 CREATE NONCLUSTERED INDEX
<Name of Missing Index, sysname
ON Sales.OrderHeader(OnlineOrderFlag)
INCLUDE
(OrderID,OrderDate,CustomerID,SubTotal) 21

22. 5. Review Table & COLUMN Info
Get Table Definition
View / Table valued function?
Where does it physically reside
Review Keys & Constraints
Size / Selectivity / Datatypes of Columns
Data Profile Task / Viewer – Integration Services
Statistic Gathering?
alter
set auto_update_statistics on;
SSMS can help… 22

23. Review Table & COLUMN Info - cont.
Get All Row Counts
Review the ‘Selected’ Columns
Use of ‘*’ or scalar functions
Look performance degraders
CASE, CAST, CONVERT
Sub-queries
SARG-able vs non-SARG-able
Examine WHERE & JOIN clauses
Bad signs
No filters
Majority of table returned 23

24. 6. Review Indexes Get Index definitions
Know the order of columns and their selectivity
Review existing keys and constraints
Know Multi-Table Relationships (ERD)
Primary key and foreign definitions
Check and not null constraints
Tip: Keys & constraints help create better execution plans 24

25. Make sure the optimizer can use the index
Review Indexes – Cont.
Make sure the optimizer can use the index
Functions on indexed columns can turn off index
Consider rewriting
Look for implicit conversions
Get sample parameter values
Know left leading column if multi-column
Know Index Rebuild Schedules 25

26. 7. Understand Selectivity
Need to know the size of the actual data sets of each step
In Joins (Right, Left, Outer)
What are the filtering predicates
When is each filtering predicate applied
Try to filter earlier rather than later
Compare size of final result set with data
Find the driving table
To reduce logical reads 26

27. Understand Selectivity – cont.
SELECT c.CustomerID ,
p.FirstName ,
p.LastName ,
oh.OrderID ,
oh.OrderDate ,
oh.SubTotal,
pr.Name,
pr.Color,
od.OrderQty
FROM Sales.OrderHeader AS oh
INNER JOIN Sales.Customer AS c ON c.CustomerID = oh.CustomerID
INNER JOIN Person.Person AS p ON p.BusinessEntityID = c.PersonID
INNER JOIN Sales.OrderDetail AS od ON od.OrderID = oh.OrderID
INNER JOIN Production.Product AS pr ON pr.ProductID = od.ProductID
WHERE oh.OnlineOrderFlag = 1
and p.LastName like 'C%'
and pr.Name like 'Mountain%42'
and pr.ProductID like '9%'
Joins
Filtering Predicates 27

28. Selectivity – Use SQL Diagramming
Great Book “SQL Tuning” by Dan Tow
Great book that teaches SQL Diagramming OD
.6% OH C P PR
88% 7% 1 67 3
18699
Join Selectivity
select avg(cnt)
from (select productid, count(*) cnt
from sales.orderdetail oh
group by productid) a
Filtering Selectivity
select count(1) from sales.OrderHeader oh where oh.OnlineOrderFlag = 1
select / *100 = 87.90
select count(1) from person.Person p where p.LastName like 'C%‘
select / * 100 = 6.93
select count(1) from production.Product pr where pr.Name like 'Mountain%42‘ and pr.ProductID like '9%'
select 3.00 / * 100 = 0.59 28

29. Find Driving Table 29

30. 8. Tune the Query Focus on most expensive operations first
Try to reduce high-cost steps
Read less rows
Filter earlier than later
Find the driving table
Consider a more selective or covering index
Seeks vs scans—which is more expensive?
Implicit conversions? 30

31. Execution Plan 31 CREATE NONCLUSTERED INDEX
<Name of Missing Index, sysname
ON Sales.OrderHeader(OnlineOrderFlag)
INCLUDE
(OrderID,OrderDate,CustomerID,SubTotal) 31

32. Review Logical IO 32

33. 9. Engineer out the Stupid
Look for Performance Inhibitors
Cursor or row by row processing
Parallel processing
Hard-coded Hints
Nested views
Linked servers
Abuse of Wild Cards (*) or No Where Clause
Code-based SQL Generators (e.g. EMF; LINQ; nHibernate)
Implicit Data Conversions
Non-SARG-able / Scalar Functions
Select… where upper(first_name) = ‘JANIS’ 33

34. Using the correct index?34

35. Using the correct index – Cont.35

36. Using the correct index?36

37. 10. Adjust the index Try to reduce Logical IO
Adding indexes is not always the right thing to do!
Consider Insert, Update & Delete activity
Covering index
CREATE NONCLUSTERED INDEX cIX_OrderHeader_OnlineOrderFlag
ON Sales.OrderHeader(OnlineOrderFlag)
INCLUDE (OrderID,OrderDate,CustomerID,SubTotal)
Filtered index
CREATE NONCLUSTERED INDEX FIX_OrderHeader_OnlineOrderFlag
WHERE OnlineOrderFlag = 1
Combination
CREATE NONCLUSTERED INDEX FcIX_OrderHeader_OnlineOrderFlag 37

38. Add Covering Index on OrderDetail
CREATE NONCLUSTERED INDEX cIX_OrderDetail_ProductID
ON Sales.OrderDetail(ProductID)
INCLUDE (OrderID,OrderQty) 38

39. 11. Re-Run the Query Make Small Changes Consider adjusting indexes
Re-run & check run-time details
Compare results with baseline metrics
Use ‘Logical Reads’ as a key measurement
Did you improve it? No? Rinse & Repeat 39

40. Still high Cost? CREATE NONCLUSTERED INDEX cIX_OrderDetail_ProductID
ON Sales.OrderDetail(ProductID)
INCLUDE (OrderID,OrderQty)

41. Add Index in OrderHeader41

42. 12. Monitor Your Tuning Results
Monitor the improvement
Be able to prove that tuning made a difference
Take new metric measurements
Compare them to initial readings
Brag about the improvements – no one else will
Monitor for next tuning opportunity
Tuning is iterative
There is always room for improvement
Make sure you tune things that make a difference
Shameless Product Pitch - DPA 42

43. Before Performance Tuning
Query before tuning:
1 sec / avg exes
2500 exes / hour
100,000 rows processed
300,000,000 Logical IOs 43

44. After Performance Tuning
Added 1st index
Added 2nd index
Query before tuning:
490ms / avg exes
6000+ exes / hour
200,000+ rows processed
24,000,000 Logical IOs

45. Case Study Flights by City & Day of Week
CREATE PROCEDURE get_city_day_flights
@beg_date VARCHAR(10) AS
SELECT
o.carrier, uc.description AS carrier_name, o.fl_date,o.fl_num,o.tail_num
,ao.description AS origin_airport,co.Description AS origin_city
,ad.description AS destination_airport
,cd.Description AS destination_city ,w.Description Day_of_Wed
FROM t_ontime_2015 o
INNER JOIN L_UNIQUE_CARRIERS AS uc ON uc.Code = o.UNIQUE_CARRIER
INNER JOIN L_AIRPORT_ID AS ao ON ao.Code = o.ORIGIN_AIRPORT_ID
INNER JOIN L_AIRPORT_ID AS ad ON ad.Code = o.DEST_AIRPORT_ID
INNER JOIN L_CITY_MARKET_ID AS co ON co.Code = o.ORIGIN_CITY_MARKET_ID
INNER JOIN L_CITY_MARKET_ID AS cd ON cd.Code = o.DEST_CITY_MARKET_ID
INNER JOIN L_WEEKDAYS AS w ON w.Code = o.DAY_OF_WEEK
WHERE fl_date
AND co.Description
AND w.Description

46. US DOT - On-time Performance
Case Study
US DOT - On-time Performance

47. Query spending the most time (RTA)
Run-Time Details:
Over 50% spent on CPU,
Latch_EX & CXPACKET
LIO = 381,217,970

48. Indicates a SPID is waiting on parallel processes to complete or start
CXPACKET Wait Type
Indicates a SPID is waiting on parallel processes to complete or start
It’s more of a status - not necessarily a problem
select session_id, exec_context_id, wait_type,
wait_duration_ms, resource_description
from sys.dm_os_waiting_tasks
where session_id in
(select session_id
from sys.dm_exec_requests
where wait_type='CXPACKET')
order by session_id, exec_context_id;
ACCESS_METHODS_DATASET_PARENT:
Points to MAXDOP setting
Default is:
max degree of parallelism = 0 (unlimited)
Great article by Paul Randall
Most-common-latch-classes 48

49. Get & Examine the Execution Plan

50. Review Table & Column Info
0 = Heap
1 = Clustered Index
2 or greater = nonclustered index 50

51. Add Clustered Indexes
CREATE UNIQUE CLUSTERED INDEX l_airport_id_code ON l_airport_id(code); CREATE UNIQUE CLUSTERED INDEX l_city_market_id_code ON l_city_market_id(code); CREATE UNIQUE CLUSTERED INDEX l_unique_carriers_code ON l_unique_carriers(code); CREATE UNIQUE CLUSTERED INDEX l_weekdays_code ON l_weekdays(code); CREATE UNIQUE CLUSTERED INDEX t_ontime_flight ON t_ontime_2015 (origin_city_market_id,carrier,fl_num,fl_date,origin_airport_seq_id);

52. Selectivity – Use SQL Diagramming
800k O 1 8% w
405k
13%
18k ad 1
19k 1 uc ao 1 co cd
.02%
Filtering Selectivity
select count(1) from t_ontime_2015 where fl_date
between ' :00:00.000' and' :00:00.000‘;
select / * 100 = 8.23
select count(1) from L_CITY_MARKET_ID where description = 'Chicago, IL'
select 1.00 / * 100 = 0.017
select count(*) from L_WEEKDAYS where description = 'Friday'
select 1.00 / 8 * 100 = 12.50

53. Tune the Query

54. Fix the Stupid
CREATE PROCEDURE get_city_day_flights NVARCHAR(255) AS SELECT o.carrier, uc.description AS carrier_name, o.fl_date,o.fl_num,o.tail_num ,ao.description AS origin_airport,co.Description AS origin_city ,ad.description AS destination_airport ,cd.Description AS destination_city ,w.Description Day_of_Wed FROM t_ontime_2015 o INNER JOIN L_UNIQUE_CARRIERS AS uc ON uc.Code = o.UNIQUE_CARRIER INNER JOIN L_AIRPORT_ID AS ao ON ao.Code = o.ORIGIN_AIRPORT_ID INNER JOIN L_AIRPORT_ID AS ad ON ad.Code = o.DEST_AIRPORT_ID INNER JOIN L_CITY_MARKET_ID AS co ON co.Code = o.ORIGIN_CITY_MARKET_ID INNER JOIN L_CITY_MARKET_ID AS cd ON cd.Code = o.DEST_CITY_MARKET_ID INNER JOIN L_WEEKDAYS AS w ON w.Code = o.DAY_OF_WEEK WHERE fl_date AND co.Description AND w.Description

55. Make Small Changes

56. Adjust the index
create index IX_L_City_market_id_desc on L_CITY_MARKET_ID(Description);

57. Re-run the Query Run-Time Details: Over 85% spent on ASYNC_NETWORK_IO
Rest on CPU & PAGEIOLATCH
LIOs = 29,839,105

58. Monitor Tuning Results

59. Summary of the 12 Step Program
1. Find Which SQL to Tune
2. Gather Run-Time Details
3. Examine the Execution Plan
4. Examine the Execution Plan
5. Review Table & COLUMN Info
6. Review Indexes
7. Understand Selectivity
8. Tune the Query
9. Engineer out the Stupid
10. Adjust the index
11. Re-Run the Query
12. Monitor to Check Tuning Results
Download Poster at: 12_Step_Tuning
A 12 Step Program for Cats

60. Resolve performance issues quickly
Try Database Performance Analyzer FREE for 14 days
Improve root cause of slow performance
Quickly identify root cause of issues that impact end-user response time
See historical trends over days, months, and years
Understand impact of VMware® performance
Agentless architecture with no dependence on Oracle Packs, installs in minutes