1. Troubleshooting SQL IO issues with Xperf
Joe McTaggart
Microsoft
Premier Field Engineer
© 2012 Microsoft Corporation Microsoft Confidential

2. Conditions and Terms of Use
Microsoft Confidential
This training package is proprietary and confidential, and is intended only for uses described in the training materials. Content and software is provided to you under a Non-Disclosure Agreement and cannot be distributed. Copying or disclosing all or any portion of the content and/or software included in such packages is strictly prohibited.
The contents of this package are for informational and training purposes only and are provided "as is" without warranty of any kind, whether express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non-infringement.
Training package content, including URLs and other Internet Web site references, is subject to change without notice. Because Microsoft must respond to changing market conditions, the content should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Unless otherwise noted, the companies, organizations, products, domain names, addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, address, logo, person, place, or event is intended or should be inferred.
Copyright and Trademarks
© 2012 Microsoft Corporation. All rights reserved.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
For more information, see Use of Microsoft Copyrighted Content at
Microsoft®, Internet Explorer®, Outlook®, SkyDrive®, Windows Vista®, Zune®, Xbox 360®, DirectX®, Windows Server® and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.
© 2012 Microsoft Corporation Microsoft Confidential

3. Joe McTaggart joemct@Microsoft.com
Joined Microsoft in 2001
SQL CTS (customer technical support) through 2006
Premier Field Engineer since 2009
Reactive Troubleshooting
Proactive engagements
Training

4. Event Tracing for Windows

5. Event Tracing for Windows (ETW)
General purpose, high speed tracing Kernel and Application events Built in to the OS Can be turned on and off on the fly
Added in Windows 2000
Has since become the key instrumentation technology for windows.
Provides the foundation for xperf/wpa, netmon, procmon, processesplorer, SQL XEvents, etc
© 2012 Microsoft Corporation Microsoft Confidential

6. ETW Architecture Four main components --Event Provider
--logical entity the writes events to ETW Sessions
--Controller
--Starts and Stops ETW Sessions
--Consumer
--application that reads trace files or listens for realtime events
--Tracing Session
--buffering and logging of events
© 2012 Microsoft Corporation Microsoft Confidential

7. Xperf

8. Xperf Overview Consists of a controller and a consumer
Xperf – trace controller
Xperfviewer – consumer - reads etl traces
Moving away from Xperf to WPA and WPR
Windows Performance Recorder (WPR) for collecting traces
Windows Performance Analyzer (WPA) for view traces
Very similar to xperf/xperfviewer but with UI enhancements
Part of the Windows Performance ToolKit
Downloaded as part of the Windows SDK
Can install just the Windows Performance Toolkit to get Xperf binaries
© 2012 Microsoft Corporation Microsoft Confidential

9. When to use xperf?
Kernel and application level tracing. Provides end to end tracing. Can capture both kernel flags and application providers at the same time High CPU CPU Sampling Stackwalk can show thread stack of high cpu consumers Slow IO MiniFilter tracing Device Driver tracing (storport tracing requires a hotfix in Windows pre-2012) Memory leak Can track memory allocations Can track Handle allocations (starting in Windows 8/Windows Server 2012)

10. Windows IO Stack or or NIC NIC iSCSI HBA iSCSI HBA FC HBA FC HBA
Application
Port driver
iSCSIprt.sys
I/O Subsystem
NIC
NIC
File System
NTFS.sys or
Software Initiator
MSiSCSI.sys or
iSCSI HBA
Volume Snapshot
VOLSNAP.sys
Multi-path
MPIO.sys
iSCSI HBA
Port driver
Storport.sys
Volume Manager
DMIO.sys
Miniport
Vendor specific
FC HBA
Cluster
Clusdisk.sys
FC HBA
Time measured by Performance Monitor counters such as Avg. Disk Sec/read includes the time spent in these drivers
Class driver
DISK.sys
diskperf.sys

11. Xperf Gotcha’s
Windows 8 SDK shows it requires .NET Framework 4.5 Only necessary for other component and is not required for the Windows Performance Toolkit DisablePagingExecutive Regkey must be enabled to allow for proper Stackwalk information Trace files can get large quickly Precise tracing necessary Use circular files Lost Events If the target drive for the etl logs is not fast enough events can be lost. Increase memory buffer to reduce lost events Whenever possible, test your xperf cmd before running in production

12. Xperf Commands
Xperf –providers KF --lists kernel flags available Xperf –providers KG --lists kernel groups available Xperf –on <KF or KG> -stackwalk <KF> -MaxBuffers <##> -BufferSize <KB> -file circular --start an xperf trace Xperf –d <filename> --stop an xperf trace

13. Xperf Commands – Detailed
High CPU Issues: xperf.exe –on Latency –stackWalk Profile CPU with Wait Analysis: Xperf -on Latency+DISPATCHER –stackWalk CSwitch+ReadyThread+ThreadCreate+Profile Disk Issues with Mini-Filter tracing xperf -on PROC_THREAD+LOADER+CSWITCH+FILENAME+FILE_IO+FILE_IO_INIT+DR IVERS+ FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE -f kernel.etl -stackwalk CSwitch+DiskReadInit+DiskWriteInit+DiskFlushInit+FileCreate+FileCleanup +FileClose+FileRead+FileWrite+MiniFilterPreOpInit+MiniFilterPostOpInit

14. Xperf Demo © 2012 Microsoft Corporation Microsoft Confidential SQL IO
--usbdrive.etl
--Open sqldiag
--pick out 15 sec error message
--start with FILEIO graphs
--reorder to process on left
--duration, size, filename, offset on the right
--order by duration
--convert error message offset to decimal
--sort by offset and find decimal value.
--Point out all we can see event type, size of IO, time of IO, flags, etc
New Window
--Device IO
--Sort by duration
--Zoom in and cut off the very end (bad data)
--Show how one driver passes to another.
--looking for big drops
--left should have module, process
--show sql server and times spent in ntfs.sys
--show how it is compressed.ndf
--FILEIO split into smaller IO’s by ntfs. Not a 1:1 mapping
--show storeport.sys
--not a disk issue
New windows
--Mini Filter delay by process
--sort by duration.
--add process
--mpfilter.sys = forefront endpoint protection
--show total time of 31 sec matches diff between filtermgr.sys and ntfs.sys
High CPU:
--Sqlhighcpu.etl
--show lining up symbols
--dig into CPU Sampling graph
--follow the cpu time (parallel, sort, hash)
© 2012 Microsoft Corporation Microsoft Confidential

15. SQL PFE Blog
SQL PFE Team Blog General Xperf and looking at SQL CPU: g-sql-server-high-cpu-usage-using-xperf.aspx Xperf and SQL IO: cause-of-sql-server-io-bottleneck-using-xperf.aspx

16. Premier Field Engineer (317) 319-4943
Contact
Joe McTaggart
Premier Field Engineer
(317)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION
© 2012 Microsoft Corporation Microsoft Confidential