1. SQL Server Internals - Performance Troubleshooting Hands on Approach -
Mohit K. Gupta
SQL Server Internals - Performance Troubleshooting Hands on Approach -

2. SQL Summit Annual International Conference November 6 -9 | Seattle, WA
2 Days of Pre-Cons
200+ sessions over 3 days
Over 5,000 SQL Professionals
Evening Networking Activities

3. Please Support Our Sponsors

4. Let’s Not Distract Others
Please silent your phone
We can chat later
Please don’t snore

5. Troubleshooting Motto
Any thing you discover running in non-normal conditions is a victim, until proven otherwise.

6. "My" Methodology

7. Methodology Seek to understand Build list of assumptions
Check if SQL Instance is responsive
Check for resource contention issue
Verify, Prove, or Disprove Assumptions

8. Seek To Understand

9. Do you have a SQL related error message(s)? Is it reproducible?
What is running slow?
Is it an application, a job, a process, or a report?
How fast it used to run (i.e. quantify “slow” for me please)?
Where are you getting timeouts (is it application, web application, or SQL timeout)?
Do you have a SQL related error message(s)?
Is it reproducible?
How often “it” happens?

10. Build List of Assumptions

11. Introduction How do you fix a problem?
Build a list of assumptions, what could the issue be?
Assumptions come from experience, if your new to performance troubleshooting; you might have to rely on industry experts or colleagues.
Prove or disprove each until you find the root-cause.
After you find the root cause, check, check, and check again!
Make sure the root-cause is not a victim of something else.
I’ll use “+” for each assumption I introduce.

12. Lets Build Assumptions
If it’s an application, general behavior of well designed application is ..
OLTP Workload
Short transactions
Completes quickly
+ resource contention?
+ blocking?
+ poor statistics?

13. Lets Build Assumptions
If it’s a job and/or task, it’s general behavior is ..
Will take time; however question is what is norm?
Sometime jobs are governed by SLA/business requirements; so ask, how long it supposed to take?
+ sufficient data changed?
+ resource or hardware contention?
+ poor statistics?
+ poor execution plans?
+ poor code?
+ blocking?

14. Lets Build Assumptions
If it’s a report, it’s general behavior is ..
Some reports are quick (on-demand); some reports long (on-schedule).
How often is the report run?
How long it generally runs?
How long has it been running for?
+ long report?
+ blocking?
+ resource contention issue?

15. SQL Server Responsive

16. Try logging in from your desktop SSMS to SQL Instance in question …
Were you able to login?
Yes! But it took a long time …
+ resource contention issue
Yes! No problem …
+ blocking issue
No! Connection timed out …
What kind of timeout was it? Instance not found or Login time out?
Instance Not Found! …
+ one or more SQL services are not online
Login Time Out! …

17. Resource Contention Issue

18. What are our resources? Tools available? Task Manager Resource Monitor
Processor
Memory
Disk
Network
Tools available?
Task Manager
Resource Monitor
Performance Monitor

19. Resource Contention Issue - PROCESSOR -

20. Is it SQL Server instance causing the utilization?
Check it out by using task manager or ..
Performance counters ..
Processor\% Processor Time
Process\% User Time (sqlservr)
Yes! It’s SQL …
What uses CPU in-sider SQL Server?
Answer …
Everything!
+ compilations / re-compilations?
+ large read (memory scan)?
+ physical activity (reads / writes)?
+ hash operations?
No! It’s not SQL …
Track down application and/or service shut it down.
Migrate SQL Server from shared platform to dedicated platform.
Fix offending application.

21. Resource Contention Issue - MEMORY -

22. Is it SQL Server instance causing the utilization?
Check it out by using task manager or ..
No … No … No … No …
Performance counters ..
Memory\Available MBytes
Page File\% Usage (_Total)
SQLServer:Buffer Manager\Page Life Expectancy
SQLServer:Memory Manager\Target Server Memory
SQLServer:Memory Manager\Total Server Memory
Yes! It’s SQL …
Do I have internal memory pressure issue?
+ max memory is not configured correctly?
+ do I have procedure plan bloating?
+ is memory being used efficiently?

23. Do I have external memory pressure issue?
+ do I have memory leak?
+ is SQL Server max memory set correctly?
+ do I have enough memory?

24. Resource Contention Issue - DISK -

25. Is it SQL Server instance causing the utilization?
Check it out by using resource monitor or ..
Performance counters ..
Logical Disk\Avg. Disk sec/Read | Write
Logical Disk\Avg. Disk Read | Write/sec
Yes! It’s SQL …
+ if its reads; do I have a memory pressure issue?
+ if its reads; do I have a large scan going on?
+ if its reads; did someone flush my cache?
+ if its reads; is there maintenance going on?
+ if its write; do I have memory pressure issue?
+ if its write; do I have excessive check-points?
+ if its write; do I have maintenance going on?

26. Resource Contention Issue - NETWORK -

27. Check it out by using resource monitor or .. Performance counters ..
Network Interface\Total Bytes/sec
Its rare for network to cause bottleneck in performance; however if it your bottleneck …
+ do I have a large read?
+ do I have a client side process that is using wait-and-pull operation?
+ do I have a network issue?

28. Resource Contention Issue - VICIOUS CYCLE -

29. Processor Memory Disk Network Low Memory Large Reads Configuration
Maintenance
Large Data Request
Poor Application
Network Issue
Poor Code
Large Read
Computation
Processor
Memory
Disk
Network

30. Verify, Prove, or Disprove Assumptions

31. Pick off one at a time … Service Not Online or Responsive Maintenance
11/12/2018
Pick off one at a time …
Service Not Online or Responsive
Maintenance
Blocking
Resource Contention
Poor Statistics
Poor Execution Plans
Compilations
Large Reads
Physical Disk Activity
Hash Operations
Memory Configuration / Efficient Memory Utilization / Memory Pressures
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Service Not Responsive
Try connecting to server from SSMS.
Try using DAC (SQLCMD –A) or (SSMS Query Connection with ADMIN:ServerName) from your computer.
If it doesn’t work, RDP to server and try same options.

33. Maintenance Always Evaluate what is running on SQL Server.
Could it be that an scheduled task is causing the problem?
Could it be that necessary tasks are not running?
Such as Index Maintenance, Statistics Maintenance, etc.

34. Blocking Connect to SSMS. Connect to SQL Server Instance in Question.
Right-click on the server.
Go to Reports.
Go to Standard Reports.
Run All Transactions – Blocking Report.
Identify the head blocker…
If head blocker is doing select it generally safe to kill it. However be careful!

35. Resource Contention
External resource contention can be assed in limited fashion from Task Manager.
However to assess detail contention related issues you will have to rely on Performance Monitor (Perfmon) and evaluate the performance counters discussed to date.
Within SQL Server, you can assess resource contention issue by evaluating wait types:
Open SSMS.
Connect to SQL Server.
Right click on Server Name.
Click on Activity Monitor.
Expand Processes section.
Filter on Wait Type field.
Look up the wait type at MSDN – sys.dm_os_wait_stats document.
Not all wait types are documented. Some are critical, some are informational, and some are system.
General wait types to be concerned about LCK_M_*, PAGELATCH_*, PAGEIOLATCH_*, SOS_SCHEDULER_YEILD, CXPACKET, RESOURCE_SEMAPHORE, CXPACKET, etc.
Always look at the wait resource description to understand what the wait is on.

36. Poor Statistics
Difficult to evaluate if poor statistics are causing the problem.
However if statements start performing poorly out of the blue, it is generally first thing to try.
Therefore, look at executing sp_updatestats against the database in question.
Or, if you know the tables in question use UPDATE STATISTICS.

37. Poor Execution Plans
Generally poor execution plans are result of poor statistics.
Sometimes these plans are also generated due to Parameter Sniffing functionality.
If you suspect an execution plan problem – because something is taking longer then run then normal.
Get list of current running statements:
SELECT session_id, plan_handle, sql_handle, status, command, cpu_time, reads, writes, total_elapsed_time FROM sys.dm_exec_requests WHERE status = 'RUNNING‘
Get the plan_handle.
Clear the plan from memory, to force SQL Server to re-generate the execution plan on next execution.
DBCC FREEPROCACHE(plan_handle) --- DO NOT EXECUTE THIS WITHOUT PLAN_HANDLE ---

38. Compilations Compilations rarely are a root cause but can happen.
Some of the factors that affect compilations are frequent statistics update, poor coding techniques, object changes, etc.
If compilations rate is high, i.e. # of BR/sec to Compilations/sec or Re-Compilations/sec is greater then 20; please evaluate type of code running.
If re-compilations are higher, look at the last time statistics were updated.
SELECT object_name(object_id) as ObjectName, stats_id, STATS_DATE(object_id, stats_id) FROM sys.stats
Consider disabling auto-update statistics; if you suspect the recompilations are happening because of auto-update stats.
If statistics update are causing issue, generally wait type COMPILEOBJ will be shows.
If compilations are higher, look at type of code executing.
Can go to SSMS -> Right Click Server Name -> Activity Monitor -> Expand Processes
Right Click on Spid -> Show Details.

39. Large Reads
Generally when there are large number of reads, it will cause high-cpu load as large amount of memory is being accessed.
However after read completes it generally will lead to ASYNC_NETWORK_IO as the large amount of data needs to be sent.
In either cases to evaluate if reads are the problem you will need to assess the execution plan of the offending statement.
Get plan_handle from the sys.dm_exec_requests.
Get the execution plan from sys.dm_exec_query_plans(plan_handle).

40. Physical Disk Activity
Generally caused by memory pressure issue.
However, can be also caused by poor code running.
If you suspect poor code, look at Activity Monitor for statements causing large waits. If disk is being a bottleneck, then PAGEIOLATCH_* type wait times will show.
Assess if there is internal or external memory pressure issue.
Might be able to resolve the issue by increasing MAX MEMORY SETTING.
Might need to terminate the SQL code executing.
Might be due to poor disk subsystem configuration.
SQL Server we would like to disk latency of 4ms/s for writes and 8ms/s for reads.

41. Hash Operations
Heavy has operations will cause pressure on tempdb and processor.
Therefore best way to evaluate hash operations are causing performance issue is by looking at SQLServer:Access Method\Workfiles Created/sec.
Need to be evaluated with Batch Requests/sec; if the WFC/s to BR/s is higher then 20 and you have processor issue; please evaluate the code being executed.
If hash operations are issue, look at activity monitor. Look for request in running state, right click and details.
This might be caused by missing indexes or large reads.

42. Memory Configuration
SQL Server will consume all the memory available on the server, unless configured correctly.
Max server memory setting does not control all the components, there is additional overhead that needs to be assed.
If you are having memory pressure issue, look at following 5 counters together to understand what is going on.
SQL Server:Buffer Manager\Page Life Expectancy
SQL Server:Memory Manager\Total Server Memory
SQL Server:Memory Manager\Target Server Memory
Memory\Available Mbytes
Page File\% Usage(_Total)
Together these counters will help evaluate if you have external memory pressure issue or internal memory pressure issue.
In SQL Server we might see these issues because we have Low Number of Free Pages, Large Number of Lazy Writes per Second, Pending Memory Grants, Outstanding Memory Grants, RESOURCE_SEMAPHORE wait types.

43. Scenario