1. Janis Griffin Database Performance Evangelist
Geekin’ Out on Extended Events Deconstructing system_health events to solve SQL Server performance
Janis Griffin
Database Performance Evangelist

2. Who Am I Senior DBA for SolarWinds (formally Confio)
@DoBoutAnything
Current – 25+ Years DBA
Primarily SQL Server, Sybase, Oracle
Former – Database Design & Implementation
Specialize in Performance Tuning
Review Database Performance for Customers and Prospects
Interest in System Health Events

3. Agenda Extended Events Overview
System_Health - Event Session & Targets
Viewing Info via SSMS
Review what’s collected?
All Event details
SP_SERVER_DIAGNOSTICS
How to query XML data type
2014 has it incorporated into System Health
Examples & Usages
Appendix – Error Descriptions / More Code

4. Extended Events Overview
Lightweight event-handling mechanism
Captures event information like SQL Profiler / SQL Trace
More information plus you can now configure
When events are triggered
They can be sent to a target for further analysis
Introduced in SQL Server 2008
Very complex to code and read (parse xml)
Much Improved in 2012 / 2014 with many more Events
SSMS Extended Event Interface
Deprecation Announcement
Default Trace is No More
Won’t be in Versions > 2016
select instance_name, cntr_value
from sys.dm_os_performance_counters
where object_name like '%deprecate%'
and instance_name like '%trace%'

5. System_Health Session
Included by default beginning with SQL Server 2012 Session starts when database engine starts No noticeable performance effects Collects system data to troubleshoot issues Uses the ring buffer & event file targets Microsoft recommends you leave it running If session dropped, recreate by script: u_tables.sql C:\Program Files\Microsoft SQL Server\MSSQL11.<instanceid>\MSSQL\Install

6. System_Health (SH) Targets
Query to view the session data from different targets:
SELECT CAST(xst.target_data as xml) target_session_data
FROM sys.dm_xe_session_targets xst
JOIN sys.dm_xe_sessions xs
ON (xs.address = xst.event_session_address)
WHERE xs.name = 'system_health'
View data for specific targets in SSMS Extended Events UI
Ring buffer info is in XML format

7. View the ring_buffer data in xml format
Viewing Info via SSMS
View the ring_buffer data in xml format
Select the ring_buffer target
Data will appear in the xml editor
Options to view event_file target, (.XEL file):
Use File -> Open in SSMS.
Drag and Drop the file into SSMS.
Double click the .XEL file.
Default location – SQL Server log directory
In SSMS > Extended Events session > select ‘View Target Data’.
fn_xe_file_target_read_file
To view more than one .XEL file:
Merge Extended Event Files from File -> Open menu.

8. Viewing SH via SSMS UI

9. Viewing SH via Function
sys.fn_xe_file_target_read_file ( path, mdpath, initial_file_name, initial_offset )
SELECT object_name,count(*)
FROM sys.fn_xe_file_target_read_file
('C:\Program Files\Microsoft SQL Server\MSSQL11.SPEEDTY\MSSQL\Log\system_health*.xel',null,null,null)
group by object_name;

10. Merging SH Files

11. What’s Collected? System Data Collected:
Sql_text & session_id for sessions that encounter:
Memory-related errors , 701, 802, 8645, 8651, 8657 and 8902
Or that have encountered errors with a severity >= 20
Non-yielding scheduler issues ( i.e. error in sql server log)
Deadlocks
Callstack, sql_text, & session_id for sessions waiting on:
on resources for > 15 seconds.
on locks for > 30 seconds.
for a long time for preemptive waits
Duration varies by wait type.
Preemptive wait is when SQL Server is waiting for external API calls
Callstack & session_id for CLR and virtual allocation failures
System component results from sp_server_diagnostics
System Health, Memory, Connectivity & Security errors using:
Their respective ring buffers - e.g. connectivity_ring_buffer_recorded

12. What’s Collected in detail?
System_Health Session – Events

13. Detailed Events scheduler_monitor_system_health_ring_buffer_recorded
wait_info or wait_info_external (preemptive)
What the heck is a ‘callstack’? Review Jonathan Kehayias’s reference:
security_error_ring_buffer_recorded

14. More Detailed Events error_reported memory_broker_ring_buffer_recorded
Only shows errors with severity >= 20
Consider changing for Development environments to check that code!
memory_broker_ring_buffer_recorded
Shows Memory allocations between various components

15. Detailed Events “(cont.)”
Memory_broker_type definitions
MEMORYBROKER_FOR_CACHE
Memory allocations used by cached objects
MEMORYBROKER_FOR_STEAL
Memory used for compilations
Stolen from buffer pool
MEMORYBROKER_FOR_RESERVE
Memory used for query execution
Others with little or no documentation
MEMORYBROKER_FOR_XTP
2012+
MEMORYBROKER_FOR_HASHED_DATA_PAGES
2014+
MEMORYBROKER_FOR_COMMITTED
Memory used by all other caches except above

16. Detailed Events “(cont.)”
connectivity_ring_buffer_recorded
xml_deadlock_report

17. Even More Information sp_server_diagnostics_component_result
Originally used for AlwaysOn:AvailabilityGroup
Great Info Regardless!
Can be run manually
Captures diagnostic data & health information to detect potential failures. The procedure runs in repeat mode and sends results periodically. It can be invoked from either a regular or a DAC (dedicated admin) connection.
sp_server_diagnostics 5
Interval has to be 5 seconds or more

18. SP_SERVER_DIAGNOSTICS
How to run the sp_server_diagnostics package
Default is 0 – run once
May want to set up as a job that continuously runs
States: Unknown, clean, warning, error

19. SP_SERVER_DIAGNOSTICS
SYSTEM Section
<RingBufferTarget truncated="1" processingTime="98772" ... Etc…
<event name="sp_server_diagnostics_component_result" package="sqlserver" timestamp=" T02:41:47.499Z">
<data name="component">
<type name="sp_server_diagnostics_component" package="sqlserver"></type>
<value>0</value>
<text><![CDATA[SYSTEM]]></text>
</data>
<data name="state">
<type name="sp_server_diagnostics_state" package="sqlserver"></type>
<value>1</value>
<text><![CDATA[CLEAN]]></text>
<data name="data">
<type name="xml" package="package0"></type>
<value>
<system spinlockBackoffs="0" sickSpinlockType="none" sickSpinlockTypeAfterAv="none" latchWarnings="0" isAccessViolationOccurred="0" writeAccessViolationCount="0" totalDumpRequests="0" intervalDumpRequests="0" nonYieldingTasksReported="0" pageFaults="158" systemCpuUtilization="18" sqlCpuUtilization="3" BadPagesDetected="0" BadPagesFixed="0" LastBadPageAddress="0x0" />
</value>
</event>

20. System XML Query
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component as [System CPU Utilizaton] as [SQL Cpu Utilization] as [Latch Warnings] as [Access Violations] as [Non Yields Reported] as [Bad Pages Detected] as [Bad Pages Fixed] AS t(e) WHERE 'varchar(50)') = ’sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'SYSTEM‘;

21. System Data
System

22. SP_SERVER_DIAGNOSTICS
<text><![CDATA[CLEAN]]></text>
<resource lastNotification="RESOURCE_MEMPHYSICAL_HIGH" outOfMemoryExceptions="0" isAnyPoolOutOfMemory="0" processOutOfMemoryPeriod="0">
<memoryReport name="Process/System Counts" unit="Value">
<entry description="Available Physical Memory" value=" " />
<entry description="Available Virtual Memory" value=" " />
<entry description="Available Paging File" value=" " />
<entry description="Working Set" value=" " />
<entry description="Percent of Committed Memory in WS" value="73" />
<entry description="Page Faults" value=" " />
<entry description="System physical memory high" value="1" />
<entry description="System physical memory low" value="0" />
<entry description="Process physical memory low" value="0" />
<entry description="Process virtual memory low" value="0" />
</memoryReport> <memoryReport name="Memory Manager" unit="KB">
<entry description="VM Reserved" value=" " />
<entry description="VM Committed" value="410592" />
<entry description="Locked Pages Allocated" value="0" />
<entry description="Large Pages Allocated" value="0" />
<entry description="Emergency Memory" value="1024" />
<entry description="Emergency Memory In Use" value="16" />
<entry description="Target Committed" value="606656" />
<entry description="Current Committed" value="410592" />
<entry description="Pages Allocated" value="207536" />
<entry description="Pages Reserved" value="1024" />
<entry description="Pages Free" value="125120" />
<entry description="Pages In Use" value="241216" />
<entry description="Page Alloc Potential" value="839616" />
<entry description="NUMA Growth Phase" value="0" />
<entry description="Last OOM Factor" value="0" />
<entry description="Last OS Error" value="0" />
</memoryReport>
</resource>

23. Resource XML Query
XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component as [Last Notification] as [OOM Exceptions] as [AvaiilPhysical Mem(GB)] as [Avail Paging File (GB)] as [Percent of Committed Memory in WS] as [Page Faults] as [VM Committed (MB)] as [Locked Pages Allocated (GB)] as [Large Pages Allocated (GB)] as [Target Committed (GB)] as [Current Committed (GB)] AS t(e) WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'RESOURCE‘;

24. Resource Data
RESOURCE_MEMPHYSICAL_HIGH - SQL can grow memory usage RESOURCE_MEMPHYSICAL_LOW - System or internal physical memory - shrink RESOURCE_MEM_STEADY – Maintain memory – no change RESOURCE_MEMVIRTUAL_LOW – Virtual address range for SQL Server process is becoming exhausted. Commonly the largest free block is less than 4MB

25. SP_SERVER_DIAGNOSTICS
Query Processing Section
<queryProcessing maxWorkers="512" workersCreated="103" workersIdle="67" tasksCompletedWithinInterval="231" pendingTasks="0" oldestPendingTaskWaitingTime="0" hasUnresolvableDeadlockOccurred="0" hasDeadlockedSchedulersOccurred="0" trackingNonYieldingScheduler="0x0">
<topWaits>
<nonPreemptive>
<byCount>
<wait waitType="PAGEIOLATCH_SH" waits=" " averageWaitTime=" " maxWaitTime="5691" />
<wait waitType="WRITELOG" waits="465276" averageWaitTime="761491" maxWaitTime="4446" />
<wait waitType="IO_COMPLETION" waits="453320" averageWaitTime="800811" maxWaitTime="3369" />
<wait waitType="LCK_M_S" waits="369871" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="PAGEIOLATCH_UP" waits="281647" averageWaitTime="107218" maxWaitTime="2779" />
<wait waitType="ASYNC_NETWORK_IO" waits="242675" averageWaitTime="582969" maxWaitTime="4392" />
<wait waitType="SLEEP_BPOOL_FLUSH" waits="204757" averageWaitTime="102009" maxWaitTime="809" />
<wait waitType="HADR_FILESTREAM_IOMGR_IOCOMPLETION" waits="68978" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="PAGEIOLATCH_EX" waits="68281" averageWaitTime="128510" maxWaitTime="2751" />
<wait waitType="FFT_RECOVERY" waits="45976" averageWaitTime="883294" maxWaitTime="16094" />
</byCount>
<byDuration>
<wait waitType="CLR_AUTO_EVENT" waits="324" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="FT_IFTSHC_MUTEX" waits="10637" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="LCK_M_SCH_S" waits="198" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="LCK_M_U" waits="29596" averageWaitTime=" " maxWaitTime="45376" />
<wait waitType="XE_LIVE_TARGET_TVF" waits="2050" averageWaitTime=" " maxWaitTime="14034" />
<wait waitType="LCK_M_IS" waits="33" averageWaitTime=" " maxWaitTime="659798" />
</byDuration>
</nonPreemptive>

26. SP_SERVER_DIAGNOSTICS
Query Processing Section – “cont.”
<preemptive>
<byCount>
<wait waitType="PREEMPTIVE_OS_GENERICOPS" waits="811791" averageWaitTime="106905" maxWaitTime="1349" />
<wait waitType="PREEMPTIVE_OS_FILEOPS" waits="432272" averageWaitTime="596381" maxWaitTime="15461" />
<wait waitType="PREEMPTIVE_XE_CALLBACKEXECUTE" waits="398977" averageWaitTime="1082" maxWaitTime="12" />
<wait waitType="PREEMPTIVE_OS_CREATEFILE" waits="373487" averageWaitTime="221855" maxWaitTime="15461" />
<wait waitType="PREEMPTIVE_OS_WAITFORSINGLEOBJECT" waits="239927" averageWaitTime="575229" maxWaitTime="4392" />
<wait waitType="PREEMPTIVE_OS_GETFILEATTRIBUTES" waits="229899" averageWaitTime="55036" maxWaitTime="1998" />
<wait waitType="PREEMPTIVE_FILESIZEGET" waits="183914" averageWaitTime="15479" maxWaitTime="46" />
<wait waitType="PREEMPTIVE_OS_AUTHORIZATIONOPS" waits="96115" averageWaitTime="998" maxWaitTime="62" />
<wait waitType="PREEMPTIVE_OS_FLUSHFILEBUFFERS" waits="91945" averageWaitTime="114737" maxWaitTime="5071" />
<wait waitType="PREEMPTIVE_OS_REPORTEVENT" waits="47370" averageWaitTime=" " maxWaitTime=" " />
</byCount>
<byDuration>
<wait waitType="PREEMPTIVE_XE_DISPATCHER" waits="6" averageWaitTime=" " maxWaitTime=" " />
<wait waitType="PREEMPTIVE_XE_GETTARGETSTATE" waits="10015" averageWaitTime="732191" maxWaitTime="1531" />
<wait waitType="PREEMPTIVE_OS_WRITEFILE" waits="2368" averageWaitTime="225333" maxWaitTime="4521" />
</byDuration>
</preemptive>

27. Query Processing XML Query
XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component as [Max Workers] as [Workers Created] as [Idle Workers] as [Pending Tasks] as [Unresolvable Deadlock] as [Deadlocked Schedulers] AS t(e) WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'QUERY_PROCESSING‘;

28. Query Processing Data

29. Query Processing Waits Query
Non-Preemptive & Preemptive Waits
DROP TABLE #ServerDiag
CREATE TABLE #ServerDiag (create_time datetime,
component_type sysname,
component_name sysname,
state int,
state_desc sysname,
data xml)
INSERT INTO #ServerDiag execute sp_server_diagnostics
-- Nonpreemptive waits by duration
SELECT TOP 5 'Non Preemptive by duration' AS "Wait",
AS "WaitType",
AS "WaitCount",
AS "AvgWaitTime",
AS "MaxWaitTime"
FROM #ServerDiag CROSS APPLY
data.NODES('/queryProcessing/topWaits/nonPreemptive/byDuration/wait') AS t(e)
WHERE component_name LIKE 'query_processing'
ORDER BY DESC
-- Preemptive waits by duration
select TOP 5 'Preemptive by duration' as "Wait",
as "Wait Type",
as "Waits",
as "Avg Wait Time",
as "Max Wait Time"
from #ServerDiag CROSS APPLY data.nodes
('/queryProcessing/topWaits/preemptive/byDuration/wait') AS t(e)
where component_name like 'query_processing'
order by desc
Can report ‘byCount’ instead of ‘byDuration’

30. Top 5 Wait Types at Server Level
Non-Preemptive Preemptive

31. SP_SERVER_DIAGNOSTICS
IO_SUBSYSTEM Section
<event name="sp_server_diagnostics_component_result" package="sqlserver" timestamp=“snipped”
<data name="component">
<type name="sp_server_diagnostics_component" package="sqlserver"></type>
<value>3</value>
<text><![CDATA[IO_SUBSYSTEM]]></text>
</data>
<data name="state">
<type name="sp_server_diagnostics_state" package="sqlserver"></type>
<value>1</value>
<text><![CDATA[CLEAN]]></text>
<data name="data">
<type name="xml" package="package0"></type>
<value>
<ioSubsystem ioLatchTimeouts="0" intervalLongIos="0" totalLongIos="0">
<longestPendingRequests></longestPendingRequests>
</ioSubsystem>
</value>
</event>

32. IO_Subsystem XML
XML = (select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet join sys.dm_xe_sessions xe on (xe.address = xet.event_session_address) where xe.name = 'system_health' and xet.target_name='ring_buffer'); SELECT 'varchar(50)') AS EventName 'datetime') AS DateAndTime ,t.e.value('(data/text)[1]','varchar(255)') as Component [state] as [ioLatchTimeout] as [intervalLongIos] as [totalLongIos] as [longestPendingRequests] AS t(e) WHERE 'varchar(50)') = 'sp_server_diagnostics_component_result' AND t.e.value('(data/text)[1]','varchar(255)') = 'IO_SUBSYSTEM‘; you-may-not-see-the-data-you-expect-in-extended-event-ring-buffer-targets.aspx

33. IO_Subsystem Data

34. Cool ‘free’ view
Check out Extended Events ‘system health’ Dashboards for SQL Server 2012

35. Add SH metrics to DPA

36. Add SH metrics to DPA

37. Alerts in DPA

38. Historical view of all alerts for 1 year
Alerts in DPA
Historical view of all alerts for 1 year

39. How to Alert in DPA

40. XE Wait types in DPA XE_LIVE_TARGET_TVF - Watch Live Data
SQL2K14
XE_LIVE_TARGET_TVF - Watch Live Data
XE_BUFFERMGR_ALLPROCESSED_EVENT - Happens when Extended Events session buffers are flushed to targets
This wait Happens on a background thread

41. System_health session - use to trouble shoot issues
Summary
System_health session - use to trouble shoot issues
Can view easily in SSMS or query XML to alert & record history over time.
Show security & connectivity errors, system health & memory broker info
Deadlocks & many errors are captured
Consider changing error severity levels for deve/test instances
Consider changing file target size to capture more history
Can set up sp_server diagnostics for more info (2012)
Remember Trace has been deprecated
Sql Server Profile for Trace Capture
Replaces Sql Trace Stored procedures, functions and catalog views

42. Resolve performance issues quickly—free trial
Try Database Performance Analyzer FREE for 14 days
Improve root cause of slow performance
Quickly identify root cause of issues that impact end-user response time
See historical trends over days, months, and years
Understand impact of VMware® performance
Agentless architecture with no dependence on Oracle Packs, installs in minutes

43. About Solarwinds

44. Thank You!

45. APPENDIX

46. Errors Collected
Memory-related errors
17803, 701, 802, 8645, 8651, 8657 and 8902
Non-yielding scheduler issues - error 17883

47. Non-yielding scheduler issues - error 17883
Errors Collected
Memory-related errors – “Cont.”
17803, 701, 802, 8645, 8651, 8657 and 8902
Non-yielding scheduler issues - error 17883

48. System Health Severity Levels
Severity level 10 messages are informational 
indicate problems caused by mistakes in the information you have entered.
Severity levels from 11 through 16 are generated by the user
can be corrected by the user.
Severity levels from 17 through 25 indicate software or hardware errors
when a level 17, 18, or 19 error occurs, you can continue working
might not be able to execute a particular statement.
. Severity levels from 20 through 25 indicate system problems
fatal errors, which means that the process is no longer running.
process freezes before it stops
client connection closes, and the client might not be able to reconnect.
Severity Level 20: SQL Server Fatal Error in Current Process
Severity Level 21: SQL Server Fatal Error in Database (dbid) Processes
Severity Level 22: SQL Server Fatal Error Table Integrity Suspect
Severity Level 23: SQL Server Fatal Error: Database Integrity Suspect
Severity Level 24: Hardware Error
Severity Level 25: Some type of system error

49. Scheduler_monitor Query
XML =
(select CAST(xet.target_data as xml)
from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'int') AS process_utilization
'int') as system_idle
'float') as page_faults
'float') as working_set_delta
'int') as memory_utilization
AS t(e)
WHERE 'varchar(50)') = 'scheduler_monitor_system_health_ring_buffer_record'

50. Wait_info Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'int') AS Duration
AS Session_id
'varchar(max)') AS wait_type
'varchar(max)') AS wait_type_no
'varchar(max)') AS ErrMsg
'varchar(max)') AS sql_text
AS t(e)
WHERE 'varchar(50)') = 'wait_info'
ORDER BY Duration desc

51. Error_reported Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
SELECT 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'int') AS ErrNo
'int') AS Severity
'varchar(max)') AS ErrMsg
'varchar(max)') AS sql_text
AS t(e)
WHERE 'varchar(50)') = 'error_reported' '

52. Security Error Query DECLARE @x XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
select 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'varchar(max)') AS calling_api_name
'int') AS error_code
'int') AS session_i
AS t(e)
WHERE 'varchar(50)') = 'security_error_ring_buffer_recorded'

53. Connectivity Error Query
XML =
(select CAST(xet.target_data as xml) from sys.dm_xe_session_targets xet
join sys.dm_xe_sessions xe
on (xe.address = xet.event_session_address)
where xe.name = 'system_health' and xet.target_name='ring_buffer')
select 'varchar(50)') AS EventName
'datetime') AS DateAndTime
'varchar(max)') AS type
'varchar(max)') AS tds_flags
'int') AS session_id
'varchar(max)') AS source_type
'varchar(max)') AS source_text
'varchar(max)') AS local_host
'varchar(max)') AS remote_host
'int') AS tds_input_buffer_bytes
AS t(e)
WHERE 'varchar(50)') = 'connectivity_ring_buffer_recorded'

54. Xml_dead_lock Query DECLARE @XMLData xml SELECT TOP 1
@XMLData = CAST(target_data AS xml)
FROM
sys.dm_xe_session_targets xst
JOIN sys.dm_xe_sessions xs
ON xs.address = xst.event_session_address
WHERE
xs.name = 'system_health'
and xst.target_name = 'ring_buffer'
SELECT
COUNT(1) AS cnt
AS X (Col)