1. Hardcore Production Debugging of.NET Applications Ingo Rammer ingo.rammer@thinktecture.com

2. Ingo Rammer and thinktecture Support and consulting for software architects and developers Prototyping and Architectural Consulting Performance and Scalability Optimization Architecture and Code Reviews Developer-Coaching and -Mentoring http://www.thinktecture.com ingo.rammer@thinktecture.com

3. Toolchest WinDbg/CDB + SOS Mdbg + MdbgSample ADPlus GFlags LogViewer SOS Assist & DumpGen Hawkeye

4. Possibilities Processes Tracing.NET memory leaks API Call Logging KD and WinDbg to trace sync elements (mutex) Programmatically creating memory dumps Remote debugging with WinDbg Service debugging with WinDbg and Gflags Catching CLR Exceptions in WinDbg And more … Scenarios

5. XCOPY Tools XCOPY – you don‘t have to install anything on a server Definitely not Visual Studio All tools are free Debuggers are from Microsoft, rest is optional  Easy to put it on most machines even in restrictive environments I carry them on a USB stick Note: Rest of the slides are mainly for your reference. We‘ll now switch to the tools!

6. MDbg Managed Code Debugger Successor of cordbg (Common Object Runtime Debugger) Two different version! MDBG in the SDK Supported „MDBG Sample“ as C# Source IL Disassembly GUI Unsupported, but that’s the MDbg you want!

7. MDbg Commands ru -> Run Application at -> Attach to running process g -> Go (Like F5 in Visual Studio) lo -> Load Module (“LO ILDASM”) ca ex -> Catch Exceptions w -> Where am I? (Stack Trace) t -> Thread-List t -> Thread selection (“t 3”) p -> Print (local variable)

8. MDbg with swallowed Exceptions Drag and Drop of the EXE to MDbg Or: start Mdbg and use Run or Attach ca ex (Catch Exceptions) g (Go) When the exception is caught: w (Stack Trace) g(If needed: Go to continue)

9. WinDbg Low Level Debugger (also Kernel Debugging) Doesn‘t know about.NET, only unmanaged code Extension module SOS (Son Of Strike) gives access to internal.NET data structures Quite complex, but extremely powerful: Memory leak analysis (all existing objects) Analysis of memory dumps Remote debugging over TCP/IP (also with TCP forwarder as relay)

10. WinDbg Fundamentals.loadby sos mscorlib -> load SOS from the directory, from which mscorlib has been loaded. Only works *after* CLR has been initialized in the debuggee. G -> go (F5) CTRL+BREAK -> Break ~ -> Thread list ~#s-> Thread selection (z.B. ~4s) !clrstack -> Stack trace !help [cmd]-> List of SOS commands.hh [cmd]-> Help for WinDbg commands q-> Quit qd-> Quit, but detach first

11. WinDbg and Exceptions WinDbg by default breaks at a few unmanaged exceptions (like Access Violations) SXE -> Break SXI -> Ignore SXN -> Output SXE clr -> Break on all CLR Exceptions Important: NullReferenceException, DivideByZero are not CLR Exceptions, but regular unmanaged Exceptions SXE av-> Break at Access Violation (null ref, default) SXE dz-> Break at Divide by Zero (default)

12. WinDbg and Memory Leaks WinDbg doesn‘t know about.NET and can therefore see more than any managed code debugger !dumpheap -> All existing objects (reachable or not!) !gcroot -> garbage collection paths to an object !do -> Dump object !da -> Dump array !dso-> Dump stack objects (not precise, but very important) !dumpheap –stat !dumpheap –type !gcroot Use together with HawkEye, to identify the class name of GUI elements.dump /ma c:\dumps\mydump.dmp -> Dump (/ma: mini dump with “All” option)

13. Memory Dump Analysis ADPlus (Auto Dump Plus) Creates memory dumps immediately or when a certain event (unmanaged) happens Dumps can be multiple 100 MBs in size, but can usually be ZIPped quite nicely Dumps contain the complete process memory and more (handle information) Also allow reconstruction of all loaded DLLs and EXEs

14. ADPlus C:\> adplus –hang –p C:\> adplus –hang –pn C:\> adplus –crash –p

15. Global Flags (Gflags.exe) Tool of Debugging Tools for Windows Allows the immediate start of a debugger when an application starts For services: Interact with Desktop has to be activated if debugger is running as frontend You can use local remote debugging if this is not desirable

16. KD.EXE Kernel debugging for.NET applications … well? Find out about blocking situation regarding Kernel sync elements Important: current symbols! C:\> mkdir c:\symbols C:\> set _NT_SYMBOL_PATH=SRV*C:\symbols*http://msdl.microsoft.com/download/symbolshttp://msdl.microsoft.com/download/symbols

17. KD for kernel sync constructs Command: !process (  not PID!) C:\> KD /kl (  local Kernel debugging) lkd> !process 0 0 (  basic information about all processes of this name) PROCESS 8900e020 SessionId: 0 Cid: 1cd0 Peb: 7ffde000 ParentCid: 12a0 lkd> !process 8900e020 2 (  2: thread information with locks) THREAD 89d86da8 Cid 1cd0.1ba0 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode Alertable 89d86e98 NotificationTimer THREAD 87ba5020 Cid 1cd0.17e0 Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable 89d48fc0 Mutant - owning thread 89d86da8 THREAD 88827360 Cid 1cd0.1d88 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable 89d48fc0 Mutant - owning thread 89d86da8 In WinDbg/SOS: use Cid from KD to match !threads command‘s OSID column to find the offending thread

18. Creating memory dumps DbgHelp.dll of the Debugging Tools for Windows is redistributable [DllImport("DbgHelp.dll", SetLastError=true)] private static extern bool MiniDumpWriteDump( IntPtr hProcess, int processId, IntPtr fileHandle, int dumpType, IntPtr excepInfo, IntPtr userInfo, IntPtr extInfo); public static void CreateMiniDump(int pid, string outputFileName) { using (FileStream stream = new FileStream(outputFileName, FileMode.Create, FileAccess.ReadWrite)) { using (Process proc = Process.GetProcessById(pid)) { int dumpType = (int)0x00000306; MiniDumpWriteDump(proc.Handle, proc.Id, stream.Handle, dumpType, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero); }

19. Remote Debugging with WinDbg WinDbg/CDB can be started to act as a server and will listen on a TCP endpoint for commands CDB -server tcp:port=1234 WinDbg –remote tcp:server=localhost,port=1234 Or: WinDbg menu File  Connect to Remote Session Can also be used with Gflags! Warning: there is no built-in security

20. Stopping on CLR Exceptions Note: the currently public versions of SOS.DLL expose critical memory leaks which consume a few MB of RAM for each CLR exception which is running through a filter like the following (even if the filter is NOT triggered) !StopOnException –create System.IO.DirectoryNotFoundException 1

21. MDbg SDK: http://www.microsoft.com/downloads/details.aspx?familyid=38449 a42-6b7a-4e28-80ce-c55645ab1310&displaylang=en http://www.microsoft.com/downloads/details.aspx?familyid=38449 a42-6b7a-4e28-80ce-c55645ab1310&displaylang=en WinDbg: http://www.microsoft.com/whdc/devtools/debugging/default.mspx http://www.microsoft.com/whdc/devtools/debugging/default.mspx HawkEye: http://www.acorns.com.au/Projects/Hawkeye/http://www.acorns.com.au/Projects/Hawkeye/ SOSAssist: http://www.thinktecture.com/sosassisthttp://www.thinktecture.com/sosassist Related Content

22. Resources Technical Communities, Webcasts, Blogs, Chats & User Groups http://www.microsoft.com/communities/default.mspx http://www.microsoft.com/communities/default.mspx Microsoft Learning and Certification http://www.microsoft.com/learning/default.mspx http://www.microsoft.com/learning/default.mspx Microsoft Developer Network (MSDN) & TechNet http://microsoft.com/msdn http://microsoft.com/technet http://microsoft.com/msdn http://microsoft.com/technet Trial Software and Virtual Labs http://www.microsoft.com/technet/downloads/trials/defa ult.mspx http://www.microsoft.com/technet/downloads/trials/defa ult.mspx New, as a pilot for 2007, the Breakout sessions will be available post event, in the TechEd Video Library, via the My Event page of the website Required slide: Please customize this slide with the resources relevant to your session MSDN Library Knowledge Base Forums MSDN Magazine User Groups Newsgroups E-learning Product Evaluations Videos Webcasts V-labs Blogs MVPs Certification Chats learn support connect subscribe Visit MSDN in the ATE Pavilion and get a FREE 180-day trial of MS Visual Studio Team System!

23. Complete your evaluation on the My Event pages of the website at the CommNet or the Feedback Terminals to win!

24. © 2007 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.