Download presentation
Presentation is loading. Please wait.
Published byTheodore Harrison Modified over 7 years ago
1
Janis Griffin Database Performance Evangelist
Geekin’ Out on Extended Events Deconstructing system_health events to solve SQL Server performance Janis Griffin Database Performance Evangelist
2
Who Am I Senior DBA for SolarWinds (formally Confio)
@DoBoutAnything Current – 25+ Years DBA Primarily SQL Server, Sybase, Oracle Former – Database Design & Implementation Specialize in Performance Tuning Review Database Performance for Customers and Prospects Interest in System Health Events
3
Agenda Extended Events Overview
System_Health - Event Session & Targets Viewing Info via SSMS Review what’s collected? All Event details SP_SERVER_DIAGNOSTICS How to query XML data type 2014 has it incorporated into System Health Examples & Usages Appendix – Error Descriptions / More Code
4
Extended Events Overview
Lightweight event-handling mechanism Captures event information like SQL Profiler / SQL Trace More information plus you can now configure When events are triggered They can be sent to a target for further analysis Introduced in SQL Server 2008 Very complex to code and read (parse xml) Much Improved in 2012 / 2014 with many more Events SSMS Extended Event Interface Deprecation Announcement Default Trace is No More Won’t be in Versions > 2016 select instance_name, cntr_value from sys.dm_os_performance_counters where object_name like '%deprecate%' and instance_name like '%trace%'
5
System_Health Session
Included by default beginning with SQL Server 2012 Session starts when database engine starts No noticeable performance effects Collects system data to troubleshoot issues Uses the ring buffer & event file targets Microsoft recommends you leave it running If session dropped, recreate by script: u_tables.sql C:\Program Files\Microsoft SQL Server\MSSQL11.<instanceid>\MSSQL\Install
6
System_Health (SH) Targets
Query to view the session data from different targets: SELECT CAST(xst.target_data as xml) target_session_data FROM sys.dm_xe_session_targets xst JOIN sys.dm_xe_sessions xs ON (xs.address = xst.event_session_address) WHERE xs.name = 'system_health' View data for specific targets in SSMS Extended Events UI Ring buffer info is in XML format
7
View the ring_buffer data in xml format
Viewing Info via SSMS View the ring_buffer data in xml format Select the ring_buffer target Data will appear in the xml editor Options to view event_file target, (.XEL file): Use File -> Open in SSMS. Drag and Drop the file into SSMS. Double click the .XEL file. Default location – SQL Server log directory In SSMS > Extended Events session > select ‘View Target Data’. fn_xe_file_target_read_file To view more than one .XEL file: Merge Extended Event Files from File -> Open menu.
8
Viewing SH via SSMS UI
9
Viewing SH via Function
sys.fn_xe_file_target_read_file ( path, mdpath, initial_file_name, initial_offset ) SELECT object_name,count(*) FROM sys.fn_xe_file_target_read_file ('C:\Program Files\Microsoft SQL Server\MSSQL11.SPEEDTY\MSSQL\Log\system_health*.xel',null,null,null) group by object_name;
10
Merging SH Files
11
What’s Collected? System Data Collected:
Sql_text & session_id for sessions that encounter: Memory-related errors , 701, 802, 8645, 8651, 8657 and 8902 Or that have encountered errors with a severity >= 20 Non-yielding scheduler issues ( i.e. error in sql server log) Deadlocks Callstack, sql_text, & session_id for sessions waiting on: on resources for > 15 seconds. on locks for > 30 seconds. for a long time for preemptive waits Duration varies by wait type. Preemptive wait is when SQL Server is waiting for external API calls Callstack & session_id for CLR and virtual allocation failures System component results from sp_server_diagnostics System Health, Memory, Connectivity & Security errors using: Their respective ring buffers - e.g. connectivity_ring_buffer_recorded
12
What’s Collected in detail?
System_Health Session – Events
13
Detailed Events scheduler_monitor_system_health_ring_buffer_recorded
wait_info or wait_info_external (preemptive) What the heck is a ‘callstack’? Review Jonathan Kehayias’s reference: security_error_ring_buffer_recorded
14
More Detailed Events error_reported memory_broker_ring_buffer_recorded
Only shows errors with severity >= 20 Consider changing for Development environments to check that code! memory_broker_ring_buffer_recorded Shows Memory allocations between various components
15
Detailed Events “(cont.)”
Memory_broker_type definitions MEMORYBROKER_FOR_CACHE Memory allocations used by cached objects MEMORYBROKER_FOR_STEAL Memory used for compilations Stolen from buffer pool MEMORYBROKER_FOR_RESERVE Memory used for query execution Others with little or no documentation MEMORYBROKER_FOR_XTP 2012+ MEMORYBROKER_FOR_HASHED_DATA_PAGES 2014+ MEMORYBROKER_FOR_COMMITTED Memory used by all other caches except above
16
Detailed Events “(cont.)”
connectivity_ring_buffer_recorded xml_deadlock_report
17
Even More Information sp_server_diagnostics_component_result
Originally used for AlwaysOn:AvailabilityGroup Great Info Regardless! Can be run manually Captures diagnostic data & health information to detect potential failures. The procedure runs in repeat mode and sends results periodically. It can be invoked from either a regular or a DAC (dedicated admin) connection. sp_server_diagnostics 5 Interval has to be 5 seconds or more
18
SP_SERVER_DIAGNOSTICS
How to run the sp_server_diagnostics package Default is 0 – run once May want to set up as a job that continuously runs States: Unknown, clean, warning, error
19
SP_SERVER_DIAGNOSTICS
SYSTEM Section <RingBufferTarget truncated="1" processingTime="98772" ... Etc… <event name="sp_server_diagnostics_component_result" package="sqlserver" timestamp=" T02:41:47.499Z"> <data name="component"> <type name="sp_server_diagnostics_component" package="sqlserver"></type> <value>0</value> <text><![CDATA[SYSTEM]]></text> </data> <data name="state"> <type name="sp_server_diagnostics_state" package="sqlserver"></type> <value>1</value> <text><![CDATA[CLEAN]]></text> <data name="data"> <type name="xml" package="package0"></type> <value> <system spinlockBackoffs="0" sickSpinlockType="none" sickSpinlockTypeAfterAv="none" latchWarnings="0" isAccessViolationOccurred="0" writeAccessViolationCount="0" totalDumpRequests="0" intervalDumpRequests="0" nonYieldingTasksReported="0" pageFaults="158" systemCpuUtilization="18" sqlCpuUtilization="3" BadPagesDetected="0" BadPagesFixed="0" LastBadPageAddress="0x0" /> </value> </event>
20
System XML Query (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component as [System CPU Utilizaton] as [SQL Cpu Utilization] as [Latch Warnings] as [Access Violations] as [Non Yields Reported] as [Bad Pages Detected] as [Bad Pages Fixed] AS t(e) WHERE 'varchar(50)') = ’sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'SYSTEM‘;
21
System Data System
22
SP_SERVER_DIAGNOSTICS
<text><![CDATA[CLEAN]]></text> <resource lastNotification="RESOURCE_MEMPHYSICAL_HIGH" outOfMemoryExceptions="0" isAnyPoolOutOfMemory="0" processOutOfMemoryPeriod="0"> <memoryReport name="Process/System Counts" unit="Value"> <entry description="Available Physical Memory" value=" " /> <entry description="Available Virtual Memory" value=" " /> <entry description="Available Paging File" value=" " /> <entry description="Working Set" value=" " /> <entry description="Percent of Committed Memory in WS" value="73" /> <entry description="Page Faults" value=" " /> <entry description="System physical memory high" value="1" /> <entry description="System physical memory low" value="0" /> <entry description="Process physical memory low" value="0" /> <entry description="Process virtual memory low" value="0" /> </memoryReport> <memoryReport name="Memory Manager" unit="KB"> <entry description="VM Reserved" value=" " /> <entry description="VM Committed" value="410592" /> <entry description="Locked Pages Allocated" value="0" /> <entry description="Large Pages Allocated" value="0" /> <entry description="Emergency Memory" value="1024" /> <entry description="Emergency Memory In Use" value="16" /> <entry description="Target Committed" value="606656" /> <entry description="Current Committed" value="410592" /> <entry description="Pages Allocated" value="207536" /> <entry description="Pages Reserved" value="1024" /> <entry description="Pages Free" value="125120" /> <entry description="Pages In Use" value="241216" /> <entry description="Page Alloc Potential" value="839616" /> <entry description="NUMA Growth Phase" value="0" /> <entry description="Last OOM Factor" value="0" /> <entry description="Last OS Error" value="0" /> </memoryReport> </resource>
23
Resource XML Query XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component as [Last Notification] as [OOM Exceptions] as [AvaiilPhysical Mem(GB)] as [Avail Paging File (GB)] as [Percent of Committed Memory in WS] as [Page Faults] as [VM Committed (MB)] as [Locked Pages Allocated (GB)] as [Large Pages Allocated (GB)] as [Target Committed (GB)] as [Current Committed (GB)] AS t(e) WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'RESOURCE‘;
24
Resource Data RESOURCE_MEMPHYSICAL_HIGH - SQL can grow memory usage RESOURCE_MEMPHYSICAL_LOW - System or internal physical memory - shrink RESOURCE_MEM_STEADY – Maintain memory – no change RESOURCE_MEMVIRTUAL_LOW – Virtual address range for SQL Server process is becoming exhausted. Commonly the largest free block is less than 4MB
25
SP_SERVER_DIAGNOSTICS
Query Processing Section <queryProcessing maxWorkers="512" workersCreated="103" workersIdle="67" tasksCompletedWithinInterval="231" pendingTasks="0" oldestPendingTaskWaitingTime="0" hasUnresolvableDeadlockOccurred="0" hasDeadlockedSchedulersOccurred="0" trackingNonYieldingScheduler="0x0"> <topWaits> <nonPreemptive> <byCount> <wait waitType="PAGEIOLATCH_SH" waits=" " averageWaitTime=" " maxWaitTime="5691" /> <wait waitType="WRITELOG" waits="465276" averageWaitTime="761491" maxWaitTime="4446" /> <wait waitType="IO_COMPLETION" waits="453320" averageWaitTime="800811" maxWaitTime="3369" /> <wait waitType="LCK_M_S" waits="369871" averageWaitTime=" " maxWaitTime=" " /> <wait waitType="PAGEIOLATCH_UP" waits="281647" averageWaitTime="107218" maxWaitTime="2779" /> <wait waitType="ASYNC_NETWORK_IO" waits="242675" averageWaitTime="582969" maxWaitTime="4392" /> <wait waitType="SLEEP_BPOOL_FLUSH" waits="204757" averageWaitTime="102009" maxWaitTime="809" /> <wait waitType="HADR_FILESTREAM_IOMGR_IOCOMPLETION" waits="68978" averageWaitTime=" " maxWaitTime=" " /> <wait waitType="PAGEIOLATCH_EX" waits="68281" averageWaitTime="128510" maxWaitTime="2751" /> <wait waitType="FFT_RECOVERY" waits="45976" averageWaitTime="883294" maxWaitTime="16094" /> </byCount> <byDuration> <wait waitType="CLR_AUTO_EVENT" waits="324" averageWaitTime=" " maxWaitTime=" " /> <wait waitType="FT_IFTSHC_MUTEX" waits="10637" averageWaitTime=" " maxWaitTime=" " /> <wait waitType="LCK_M_SCH_S" waits="198" averageWaitTime=" " maxWaitTime=" " /> <wait waitType="LCK_M_U" waits="29596" averageWaitTime=" " maxWaitTime="45376" /> <wait waitType="XE_LIVE_TARGET_TVF" waits="2050" averageWaitTime=" " maxWaitTime="14034" /> <wait waitType="LCK_M_IS" waits="33" averageWaitTime=" " maxWaitTime="659798" /> </byDuration> </nonPreemptive>
26
SP_SERVER_DIAGNOSTICS
Query Processing Section – “cont.” <preemptive> <byCount> <wait waitType="PREEMPTIVE_OS_GENERICOPS" waits="811791" averageWaitTime="106905" maxWaitTime="1349" /> <wait waitType="PREEMPTIVE_OS_FILEOPS" waits="432272" averageWaitTime="596381" maxWaitTime="15461" /> <wait waitType="PREEMPTIVE_XE_CALLBACKEXECUTE" waits="398977" averageWaitTime="1082" maxWaitTime="12" /> <wait waitType="PREEMPTIVE_OS_CREATEFILE" waits="373487" averageWaitTime="221855" maxWaitTime="15461" /> <wait waitType="PREEMPTIVE_OS_WAITFORSINGLEOBJECT" waits="239927" averageWaitTime="575229" maxWaitTime="4392" /> <wait waitType="PREEMPTIVE_OS_GETFILEATTRIBUTES" waits="229899" averageWaitTime="55036" maxWaitTime="1998" /> <wait waitType="PREEMPTIVE_FILESIZEGET" waits="183914" averageWaitTime="15479" maxWaitTime="46" /> <wait waitType="PREEMPTIVE_OS_AUTHORIZATIONOPS" waits="96115" averageWaitTime="998" maxWaitTime="62" /> <wait waitType="PREEMPTIVE_OS_FLUSHFILEBUFFERS" waits="91945" averageWaitTime="114737" maxWaitTime="5071" /> <wait waitType="PREEMPTIVE_OS_REPORTEVENT" waits="47370" averageWaitTime=" " maxWaitTime=" " /> </byCount> <byDuration> <wait waitType="PREEMPTIVE_XE_DISPATCHER" waits="6" averageWaitTime=" " maxWaitTime=" " /> <wait waitType="PREEMPTIVE_XE_GETTARGETSTATE" waits="10015" averageWaitTime="732191" maxWaitTime="1531" /> <wait waitType="PREEMPTIVE_OS_WRITEFILE" waits="2368" averageWaitTime="225333" maxWaitTime="4521" /> </byDuration> </preemptive>
27
Query Processing XML Query
XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component as [Max Workers] as [Workers Created] as [Idle Workers] as [Pending Tasks] as [Unresolvable Deadlock] as [Deadlocked Schedulers] AS t(e) WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'QUERY_PROCESSING‘;
28
Query Processing Data
29
Query Processing Waits Query
Non-Preemptive & Preemptive Waits DROP TABLE #ServerDiag CREATE TABLE #ServerDiag (create_time datetime, component_type sysname, component_name sysname, state int, state_desc sysname, data xml) INSERT INTO #ServerDiag execute sp_server_diagnostics -- Nonpreemptive waits by duration SELECT TOP 5 'Non Preemptive by duration' AS "Wait", AS "WaitType", AS "WaitCount", AS "AvgWaitTime", AS "MaxWaitTime" FROM #ServerDiag CROSS APPLY data.NODES('/queryProcessing/topWaits/nonPreemptive/byDuration/wait') AS t(e) WHERE component_name LIKE 'query_processing' ORDER BY DESC -- Preemptive waits by duration select TOP 5 'Preemptive by duration' as "Wait", as "Wait Type", as "Waits", as "Avg Wait Time", as "Max Wait Time" from #ServerDiag CROSS APPLY data.nodes ('/queryProcessing/topWaits/preemptive/byDuration/wait') AS t(e) where component_name like 'query_processing' order by desc Can report ‘byCount’ instead of ‘byDuration’
30
Top 5 Wait Types at Server Level
Non-Preemptive Preemptive
31
SP_SERVER_DIAGNOSTICS
IO_SUBSYSTEM Section <event name="sp_server_diagnostics_component_result" package="sqlserver" timestamp=“snipped” <data name="component"> <type name="sp_server_diagnostics_component" package="sqlserver"></type> <value>3</value> <text><![CDATA[IO_SUBSYSTEM]]></text> </data> <data name="state"> <type name="sp_server_diagnostics_state" package="sqlserver"></type> <value>1</value> <text><![CDATA[CLEAN]]></text> <data name="data"> <type name="xml" package="package0"></type> <value> <ioSubsystem ioLatchTimeouts="0" intervalLongIos="0" totalLongIos="0"> <longestPendingRequests></longestPendingRequests> </ioSubsystem> </value> </event>
32
IO_Subsystem XML XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component [state] as [ioLatchTimeout] as [intervalLongIos] as [totalLongIos] as [longestPendingRequests] AS t(e) WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'IO_SUBSYSTEM‘; you-may-not-see-the-data-you-expect-in-extended-event-ring-buffer-targets.aspx
33
IO_Subsystem Data
34
Cool ‘free’ view Check out Extended Events ‘system health’ Dashboards for SQL Server 2012
35
Add SH metrics to DPA
36
Add SH metrics to DPA
37
Alerts in DPA
38
Historical view of all alerts for 1 year
Alerts in DPA Historical view of all alerts for 1 year
39
How to Alert in DPA
40
XE Wait types in DPA XE_LIVE_TARGET_TVF - Watch Live Data
SQL2K14 XE_LIVE_TARGET_TVF - Watch Live Data XE_BUFFERMGR_ALLPROCESSED_EVENT - Happens when Extended Events session buffers are flushed to targets This wait Happens on a background thread
41
System_health session - use to trouble shoot issues
Summary System_health session - use to trouble shoot issues Can view easily in SSMS or query XML to alert & record history over time. Show security & connectivity errors, system health & memory broker info Deadlocks & many errors are captured Consider changing error severity levels for deve/test instances Consider changing file target size to capture more history Can set up sp_server diagnostics for more info (2012) Remember Trace has been deprecated Sql Server Profile for Trace Capture Replaces Sql Trace Stored procedures, functions and catalog views
42
Resolve performance issues quickly—free trial
Try Database Performance Analyzer FREE for 14 days Improve root cause of slow performance Quickly identify root cause of issues that impact end-user response time See historical trends over days, months, and years Understand impact of VMware® performance Agentless architecture with no dependence on Oracle Packs, installs in minutes
43
About Solarwinds
44
Thank You!
45
APPENDIX
46
Errors Collected Memory-related errors 17803, 701, 802, 8645, 8651, 8657 and 8902 Non-yielding scheduler issues - error 17883
47
Non-yielding scheduler issues - error 17883
Errors Collected Memory-related errors – “Cont.” 17803, 701, 802, 8645, 8651, 8657 and 8902 Non-yielding scheduler issues - error 17883
48
System Health Severity Levels
Severity level 10 messages are informational indicate problems caused by mistakes in the information you have entered. Severity levels from 11 through 16 are generated by the user can be corrected by the user. Severity levels from 17 through 25 indicate software or hardware errors when a level 17, 18, or 19 error occurs, you can continue working might not be able to execute a particular statement. . Severity levels from 20 through 25 indicate system problems fatal errors, which means that the process is no longer running. process freezes before it stops client connection closes, and the client might not be able to reconnect. Severity Level 20: SQL Server Fatal Error in Current Process Severity Level 21: SQL Server Fatal Error in Database (dbid) Processes Severity Level 22: SQL Server Fatal Error Table Integrity Suspect Severity Level 23: SQL Server Fatal Error: Database Integrity Suspect Severity Level 24: Hardware Error Severity Level 25: Some type of system error
49
Scheduler_monitor Query
XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer') SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime 'int') AS process_utilization 'int') as system_idle 'float') as page_faults 'float') as working_set_delta 'int') as memory_utilization AS t(e) WHERE 'varchar(50)') = 'scheduler_monitor_system_health_ring_buffer_record'
50
Wait_info Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer') SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime 'int') AS Duration AS Session_id 'varchar(max)') AS wait_type 'varchar(max)') AS wait_type_no 'varchar(max)') AS ErrMsg 'varchar(max)') AS sql_text AS t(e) WHERE 'varchar(50)') = 'wait_info' ORDER BY Duration desc
51
Error_reported Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer') SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime 'int') AS ErrNo 'int') AS Severity 'varchar(max)') AS ErrMsg 'varchar(max)') AS sql_text AS t(e) WHERE 'varchar(50)') = 'error_reported' '
52
Security Error Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer') select 'varchar(50)') AS EventName 'datetime') AS DateAndTime 'varchar(max)') AS calling_api_name 'int') AS error_code 'int') AS session_i AS t(e) WHERE 'varchar(50)') = 'security_error_ring_buffer_recorded'
53
Connectivity Error Query
XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer') select 'varchar(50)') AS EventName 'datetime') AS DateAndTime 'varchar(max)') AS type 'varchar(max)') AS tds_flags 'int') AS session_id 'varchar(max)') AS source_type 'varchar(max)') AS source_text 'varchar(max)') AS local_host 'varchar(max)') AS remote_host 'int') AS tds_input_buffer_bytes AS t(e) WHERE 'varchar(50)') = 'connectivity_ring_buffer_recorded'
54
Xml_dead_lock Query DECLARE @XMLData xml SELECT TOP 1
@XMLData = CAST(target_data AS xml) FROM sys.dm_xe_session_targets xst JOIN sys.dm_xe_sessions xs ON xs.address = xst.event_session_address WHERE xs.name = 'system_health' and xst.target_name = 'ring_buffer' SELECT COUNT(1) AS cnt AS X (Col)
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.