1. David Solomon David Solomon Expert Seminars
ADM391 Effective Microsoft® Windows® Troubleshooting with the Sysinternals Tools
David Solomon David Solomon Expert Seminars
Mark Russinovich Winternals Software
TechEd 2002

2. Agenda Introduction Identifying Processes Application Failures
TechEd 2002

3. About The Speakers Authors of: David Solomon: Mark Russinovich:
Inside Windows 2000, 3rd Edition (Microsoft Press)
Inside Windows 2000/XP/2003 Interactive Internals Video Tutorial
Used by Microsoft for worldwide internal training
David Solomon:
Teaches Windows internals classes (
Writes books and articles on Windows internals
Mark Russinovich:
Author of tools on
Co-founder and Chief Software Architect for Winternals Software (
Teaches Windows internals classes
TechEd 2002

4. Troubleshooting Topics
Things we will address:
Identifying processes and unexplained CPU activity
Application failures
Registry issues
Permission problems
DLL version mismatches
Things we aren’t covering:
Crash dump (blue screen) analysis
Performance analysis
Network problem analysis
TechEd 2002

5. Sysinternals Tools All freeware Tools we will use:
Downloadable from
Some tools include source
Tools updated frequently
Check the site regularly or subscribe to Sysinternals newsletter
Tools we will use:
Process Explorer – view process details
Filemon – monitors file I/O
Regmon – monitors registry I/O
Strings – dumps printable strings in files
~50 other Sysinternals tools on web site
TechEd 2002

6. Agenda Introduction Understanding Process Activity
Application Failures
TechEd 2002

7. The CPU Is Busy – Why? System is busy (may be slow) What is running?
A user or system process?
Interrupt activity?
What’s it doing?
File I/O? Network I/O? Registry calls?
Application code?
TechEd 2002

8. Which Process Is Running?
Determine which process’ threads are consuming the most CPU time
Quick method:
Open Task Manager ->Processes
Sort processes by “CPU” usage column
Other tools
Qslice.exe (Resource Kit)
Performance Monitor (monitor %Processor Time counter in process object for all processes)
TechEd 2002

9. Task Manager: Applications vs. Processes
Applications tab: List of top level visible windows
Windows are owned by threads
Right-click on a window and select “Go to process”
Processes tab: List of processes
Can configure with View->Select columns
“Running” means waiting for window messages
TechEd 2002

10. Identify The Image
Once you’ve found the process of interest, what is it?
Sometimes name of .EXE identifies clearly (e.g., Winword.exe)
Often, it doesn’t since Task Manager doesn’t show the full path of the image
We need more information!
TechEd 2002

11. Process Explorer “Super Task Manager”
Shows full image path, command line, environment variables, parent process, security access token, open handles, loaded DLLs & mapped files
TechEd 2002

12. Process Explorer Process tree Additional details in process list
If left justified, parent has exited
Disappears if you sort by any column
Bring back with View->Show Process Tree
Additional details in process list
Icon and description (from .EXE)
User Name shows which security database account is from (e.g. which domain)
Highlight Own, Services Processes
Differences highighting
Green: new, Red: gone
View->Update speed->Paused
TechEd 2002

13. Process Properties Image tab: Performance tab: Security tab:
Description, company name, version (from .EXE)
Full image path
Command line used to start process
Current directory
Parent process
User name
Start time
Performance tab:
Basic process CPU/memory usage
Security tab:
Access token (groups list, privilege list)
Environment tab: environment variables
Services tab (only for service processes):
List of services hosted by process
Compare XP “tasklist /svc” with details Process Explorer shows
TechEd 2002

14. Process Explorer Demo
Double click on date/time in task bar (lower right of screen)
In Process Explorer, hit F5 to refresh
Find new process created (RUNDLL32.EXE)
Examine command line arguments
TechEd 2002

15. Handle View Lower half of display shows either: Handle View
Open handles
Loaded DLLs & mapped files
Handle View
Suggestion: sort by type or path column
Objects of type “File” and “Key” are most interesting for general troubleshooting
Can view the state of synchronization objects (mutexes, semaphores, events)
By default, shows named objects
Click on Options->Show Unnamed Objects
TechEd 2002

16. Uses of Handle View Solve file locked errors
Use the search feature to determine what process is holding a file or directory open
Can even close an open files (be careful!)
Detect handle leaks using refresh difference highlighting
Understand resources used by an application
Files
Registry keys
TechEd 2002

17. Using Handle Table to Identify a Process
What if image properties say nothing?
Examine open handles
Open files or registry keys may give a clue
TechEd 2002

18. DLL View Click on View->DLL View Uses: Show Relocated DLLs option
Shows more than just loaded DLLs
Includes .EXE and any “memory mapped files”
High speed file access mechanism
Makes file appear as virtual memory
Uses:
Detect DLL versioning problems
Compare the output from a working process with that of a failing one (use File->Save As)
Find which processes are using a specific DLL (search for it)
Show Relocated DLLs option
Highlights relocated DLLs in yellow
TechEd 2002

19. Process Explorer: DLL Lab
Run Word XP
In ProcExp, click File->Save As and save the DLL view of WinWord in a text file
Exit Word
In Explorer, navigate to c:\Program Files\Common Files\Microsoft Shared\Office10\1033
Rename msointl.dll as msointl.dll.good
Copy dwintl.dll to msointl.dll
Try and start Word XP
With the message box from Word still showing, go to ProcExp & save DLL view of failing Word process
Edit both DLL view .txt files to eliminate process list (just leave DLL list)
Run Windiff and compare the two DLL views to determine the problem
TechEd 2002

20. DLL to Change to Break Various Word Versions
Office 97
\Program Files\Microsoft Office\Office
Replace mso97.dll with msain800.dll
Office 2000
\Program Files\Microsoft Office\Office\1033
Replace MSO9INTL.DLL with OUTLLIBR.DLL
Office XP
\Program Files\Common Files\Microsoft Shared\Office10\1033
Replace msointl.dll with dwintl.dll
TechEd 2002

21. Access Denied on Mapped Files
Attempting to delete a DLL or EXE that is in use gets “access denied”, not “file locked”
Example: try and delete Notepad.exe while you’re running it
Can be misleading
If this occurs, and it’s not due to permissions problems, simply search DLL list with Process Explorer for file
TechEd 2002

22. Agenda Introduction Identifying Processes Application Failures
TechEd 2002

23. Troubleshooting Application Failures
Most applications do a poor job of reporting file-related or registry-related errors
E.g. permissions problems
Missing files
Missing or corrupt registry data
TechEd 2002

24. Troubleshooting Application Failures
When in doubt, run Filemon and Regmon!
Filemon monitors File I/O; Regmon monitors registry I/O
Ideal for troubleshooting a wide variety of application failures
Also useful for to understand and tune file system access
E.g. understanding hard drive activity
TechEd 2002

25. Using Regmon/Filemon Two basic techniques:
Go to end of log and look backwards to where problem occurred or is evident and focused on the last things done
Compare a good log with a bad log
Often comparing the I/O and Registry activity of a failing process with one that works may point to the problem
Have to first massage log file to remove data that differs run to run
Delete first 3 columns (they are always different: line #, time, process id)
Easy to do with Excel by deleting columns
Then compare with FC (built in tool) or Windiff (Resource Kit)
TechEd 2002

26. File I/O Activity Sometimes, applications perform needless file I/O
Run Filemon to see how “quiet” your system is when you think nothing is going on
Some applications perform inefficient file I/O
May be due to libraries used in application that indirectly cause needless I/O
TechEd 2002

27. Filemon Demo Run Filemon Set filter to only include Notepad.exe
Run Notepad
Type some text
Save file as “test.txt”
Go back to Filemon
Stop logging
Set highlight to “test.txt”
Find line representing creation of new file
Hint: look for create operation
TechEd 2002

28. Example Problem
While typing in the document Word XP closes without any prompts
See filemon-lab2.log and try and determine why
Go to end of Filemon log and search backwards for Winword.exe
Stop at first unexplainable activity
TechEd 2002

29. Solution: Filemon Example
Working backwards, the first “strange” or unexplainable behavior are the constant reads past end of file to MSSP3ES.LEX
User looked up what .LEX file was
Related to Word proofing tools
Uninstalled and reinstalled proofing tools & problem went away
TechEd 2002

30. Example Problem Excel reports an error "File Not Found" when starting
Sometimes when clicking OK, Excel would work fine. Other times Excel would close when OK was clicked.
See filemon-lab3.log
TechEd 2002

31. Solution
Go to the last thing Excel did (search from bottom for Excel.exe)
Excel is opening a file in the \XLstart folder
Files in this folder get opened by Excel automatically upon startup
The file F:\OFF97PRO\Office\XLStart\59403E20 turned out to be some random file (not an XLS) that caused Excel to die
Removing the file eliminated the problem
TechEd 2002

32. Permission Problems
Many applications don’t report access denied errors well
Example:
In Explorer, create a folder c:\noaccess
Remove all rights to the folder
Run Notepad & type some text
Run Filemon – set filter to Notepad.exe
In Notepad, File->Save As to c:\noaccess\test.txt
Note error reported
Look at Filemon trace and find Access Denied
TechEd 2002

33. Example: Access Denied
For example, an Outlook application failed with this error:
Ran Filemon and found it was getting Access Denied
Someone had misread a request to remove EDIT rights and removed all rights
TechEd 2002

34. Example: Access Denied
AOL reported this error:
Filemon showed this:
User did not have admin rights to AOL directory
waol.exe OPEN C:\PROGRA~1\AMERIC~1.0\IDB\main.ind ACCESS DENIED
TechEd 2002

35. Understanding Disk Activity
Performance counters show which disks are being hit, but not which files
Filemon pinpoints which file(s) are being accessed and how frequently
Example: used Filemon on a server to determine which file(s) were being accessed most frequently
Imported into Excel and make a pie chart by file name or operation type
Moved these files to a different disk on a different controller
TechEd 2002

36. DLL Problems
Process Explorer may solve a DLL versioning issue, but may not if:
A DLL is missing
The order of DLL loads is relevant
So, use Filemon!
Look at the last DLL opened before the application died
Compare the startup of a working with a failing application
Missing or inaccessible DLLs often not reported correctly
Look for “NOTFOUND” or “ACCESS DENIED”
May be opening wrong versions due to wrong versions being in folders in PATH
TechEd 2002

37. Example Problem: Word Dies
Word97 starts and a few seconds later gets a Dr. Watson (access violation)
Customer tried re-installing Office – still failed
Solution:
Ran Filemon, looked at last DLL loaded before Dr. Watson
It was a printer DLL
Uninstalled printer – problem went away
TechEd 2002

38. Example Problem: Help Fails
The Help command in an application failed on Win95, but worked fine on Win98/ME/NT4/Win2000/WinXP
Failed with meaningless error message
TechEd 2002

39. Solution Ran Filemon on failing system and working system
Reduced log to file opens
Compared logs
At the point they diverged, looked backwards to last common thing done
An OLE system DLL was loaded
Noticed this OLE DLL was loaded from a directory in the user’s PATH on Win95, but from \Windows\System on other versions
Conclusion:
DLL loaded on Win95 system was not for Win95
Got proper version for Win95, problem went away
TechEd 2002

40. Filemon Example: Access Hangs
Problem: Access 2000 would hang when trying to import an Excel file
Worked fine on other users’ workstations
User had Access 97 and Access 2000 installed
Compare c:\lab\filemon-lab4-good.log with filemon-lab4-bad.log
TechEd 2002

41. Solution: Access Hangs
Failing system was loading an old Access DLL from \winnt\system32 due to having installed Access 97 previously
First unexplainable difference was that Accwiz.dll was being loaded from two different directories
Removed DLL in \winnt\system32 and problem went away
TechEd 2002

42. Configuration Problems
Missing, corrupted or overly-secure Registry settings often lead to application crashes and errors
Some applications don’t completely remove registry data at uninstall
Regmon may yield the answer…
TechEd 2002

43. Registry Activity
Normally, registry activity should be only at application/system startup and exit
But, sadly, lots of processes perform needless registry querying…
Try running Regmon to see how “quiet” your registry is
TechEd 2002

44. Using Regmon
Sometimes queries to what is not there is more interesting than what is there
Identify missing Registry keys
Search for status “NOTFOUND”
Find incorrect or corrupt data
Examine values read and/or written (in Other column)
Troubleshoot permission problems
Search for status “ACCESS DENIED”
TechEd 2002

45. Regmon Applications
If you suspect registry data is causing problems, rename the key and re-run the application
Most applications re-create user settings when run
In this way, the data won’t be seen by the application
Can always rename the key back
TechEd 2002

46. Example Problem Internet Explorer failed to start with this error:
See c:\lab\regmon-lab2.log
TechEd 2002

47. Solution Looked backwards from end of Regmon log
Last queries were to: HKCU\Software\Microsoft\Internet Connection Wizard
Looked here and found a single value “Completed” set to 0
Compared to other users—theirs was 1
Set this manually to 1 and problem went away
TechEd 2002

48. Example Problem Problem: Solution:
User somehow disabled all toolbars and menus in Word
No way to open files, change settings etc.
Solution:
With Regmon, captured startup of Word
Found location of user-specific settings for Word
Deleted this Registry key
Re-ran Word, which re-created user settings from scratch
TechEd 2002

49. Example Problem Internet Explorer hung when started
Default internet connection was set, but wasn’t being dialed
Dialing the connection first manually and then running IE worked
Background information:
User had previously installed the AT&T Dialer program, but had uninstalled it and created dial up connection manually
TechEd 2002

50. Solution
Ran Regmon
Looked backwards from end (at the point IE was hung)
Found references to ATT under a PhoneBook key
Renamed ATT key and problem went away
Conclusion: registry junk was left from uninstall
TechEd 2002

51. Filemon/Regmon as a Service
Sometimes need to capture I/O or registry activity during the logon or logoff process
E.g. errors occurring during logon/logoff
Solutions:
Install and run Filemon/Regmon as a service using Srvany tool in Resource Kit
Can configure to start at system boot
For a quick, one-time execution, run Filemon/Regmon with “psexec –s –i”
In either case, but tools remain running after logoff
TechEd 2002

52. For More Info...
Visit the Sysinternals web site for frequent updates and new tools
Subscribe to the Sysinternals newsletter to get information on Windows internals and Sysinternals tool updates
Take our advanced internals and troubleshooting class or check out our videos (see
Get the next edition of our book (to be called Windows Internals 4th edition)
Will cover advanced troubleshooting
To release end of calendar 2003
TechEd 2002

53. Community Resources Community Resources
Most Valuable Professional (MVP)
Newsgroups
Converse online with Microsoft Newsgroups, including Worldwide
User Groups
Meet and learn with your peers
TechEd 2002

54. evaluations
TechEd 2002

55. TechEd 2002 © 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.
TechEd 2002