1. Windows NT® Internals David Solomon David Solomon Expert Seminars Microsoft Corporation

2. Agenda Introduction Tools System Architecture Processes and Threads
Memory Management

3. About The Speaker David Solomon
14 years at Digital - the last 10 as a developer in the VMS operating system development group
Started Windows NT developer training company in 1992
Author of Inside Windows NT, 2nd edition (Microsoft Press) and Windows NT for OpenVMS Professionals (Digital Press)
Regular speaker at industry conferences (WinDev, Tech•Ed, Software Development, DECUS...)
Recipient of past Microsoft MVP award for MSWIN32 technical support

4. About The Company
David Solomon Expert Seminars offers high-quality Windows developer training
Taught by well known industry experts and authors who develop and teach their own courses
Instructors include:
Doug Boling, Brian Catlin, Jamie Hanrahan, Jeff Prosise, Jeffrey Richter, and David Solomon
Topics include:
Windows CE
Windows NT Internals
Windows NT and WDM Device Drivers
Windows NT® Server Applications
Win32® Programming
Visual C++® and MFC
COM/ActiveX® Programming
To be notified of new classes and other developments, join our interest list

5. Session Goals Goals Audience assumptions Acknowledgements
Explain internal architecture and operation of core Windows NT components
Use various tools that demonstration internal Windows NT behavior
Audience assumptions
Familiar with basic 32-bit OS concepts
Familiar with Win32 API (processes, threads, memory management)
Acknowledgements
Jamie Hanrahan - co-author of the Windows NT internals seminar from which these slides were taken
Dave Cutler, Helen Custer, John Balciunas, Lou Perazzoli, Mark Lucovsky, Steve Wood, Tom Miller, Gary Kimura, and Landy Wang for their support and assistance in understanding Windows NT internals
1-17e: Changed to “goals and prereqs” (seemed more friendly)

6. Windows NT Architecture
Environment Subsystems
System Processes
Services
Applications
POSIX
Replicator
Service Controller
Alerter
OS/2
WinLogon
User
Application
RPC
Session
Manager
Event
Logger
Win32
User
Mode
Subsystem DLLs
System
Threads
NTDLL.DLL
Kernel
Mode
Executive API
I/O Manager
Cache
Manager
Processes
& Threads
Security
Virtual
Memory
Win32
User,GDI
File
systems
Object management / Executive RTL
Device drivers
Kernel
Hardware Abstraction Layer (HAL)
Hardware interfaces (buses, I/O, interrupts, timers, clocks, DMA, cache control, etc.)
Copyright by Microsoft Corporation. Used by permission.

7. Windows NT 5.0 Internal changes
In one sense, much is the same
Basic architecture of many components unchanged:
Win32 subsystem, memory manager, process model, thread scheduling, security model, file system
But lots of additions of major new functionality:
Active Directory, distributed security, Kerberos, Microsoft management console, IntelliMirror™, NTFS extensions (content indexing, quotas, reparse points, sparse files, link tracking)

8. Windows NT 5.0 Internal changes
Kernel/core changes include:
I/O system (plug and play and power management)
64-bit Very Large Memory support for Alpha
Job object
Integration of Terminal Server
Comparable to level of change from 3.51 to 4.0
Also many incremental performance improvements:
Object Manager, Memory manager (e.g., working set management algorithms), SMP scalability…

9. Agenda Introduction Tools System Architecture Processes and Threads
Memory Management

10. Tools Preview tool executable origin
Performance Monitor PerfMon Windows NT
Registry Editor RegEdt32 Windows NT
Windows NT Diagnostics WinMSD Windows NT
Kernel Debugger i386kd, Widows NT CD \support\debug
alphakd
Pool Monitor poolmon Windows NT CD \support\debug
Global Flags gflags Windows NT Resource Kit
Open Handles oh Windows NT Resource Kit
QuickSlice qslice Windows NT Resource Kit
Process Viewer pviewer, Windows NT Resource Kit pview Platform SDK, VC++
Process Exploder pview Windows NT Resource Kit 4.0
Process Status pstat Windows NT Resource Kit
Pmon pmon Windows NT Resource Kit
Object Viewer WinObj Platform SDK
Process Walker PWalk Platform SDK
Page Fault Monitor PFMon Platform SDK
Spy++ Visual C++

11. Windows NT Resource Kits
Full “Windows NT 5.0 Resource Kit”
250+ utilities
Combines what was in the 4.0 Server and Workstation resource kits
Subset “Windows NT 5.0 Resource Kit Support Tools”
50 utilities
Ships in \support\reskit on Windows NT CD

12. www.sysinternals.com Windows NT internals articles and tools
Some generated using reverse engineering (e.g., no source access)
Some examples:
winobj - view object manager namespace and objects
nthandlex - show open handles by process
ntfilmon - log all file I/O operations
ntregmon - log all registry accesses
cpufrob - change thread quantum
Caveat: Most include a device driver, hence you’re added “trusted code”
No warranty on using these on your system!

13. GFLAGS (Global Flags)
Changes system-wide or image-wide debugging flags
Poolmon requires “enable pool taggin”
Oh (open handles) requires “maintain a list of objects for each type”

14. Windows NT Kernel Debugger (1 Of 4)
Two versions:
Command line: I386KD.EXE, ALPHAKD, etc., shipped with Windows NT
In NTcdrom:\support\debug\i386, … \debug\alpha, etc.
Select directory to match host system (where you will run the debugger executable); select executable to match target system (system being debugged)
Also need many DLLs from this directory
Also need symbol files from NTcdrom:\support\debug\targetarch\symbols\ …
Extended via WinDbg shipped with Platform SDK (part of MSDN Professional)
Provides GUI, fully-symbolic, source-level debugging
Needs same DLLs and symbol files
1-17 picked up your slide

15. Windows NT Kernel Debugger (2 Of 4)
Documentation:
Windows NT Workstation Resource Guide (see “Windows NT Debugger”)
Windows NT Device Driver Kit (DDK)
See i386kd -?
Help within debugger: commands “?” and “!?” and “!help”

16. Windows NT Kernel Debugger (3 Of 4)
Two modes of operation:
Open a crash dump file: C:\> set _NT_SYMBOL_PATH= ntcdrom:\support\debug\i386\symbols C:\> i386kd -Z dumpfilename
Connect to a live system via null modem cable (must boot target system with /DEBUG/DEBUGPORT=COMn in boot.ini) C:\> set _NT_SYMBOL_PATH=ntcdrom:\support\debug\i386\symbols C:\> set _NT_DEBUG_PORT=COMn default COM1 C:\> set _NT_DEBUG_BAUD_RATE=nnnnn default C:\> i386kd
serial “null modem” cable (for debugger)
host
target

17. Windows NT Kernel Debuggers (4 Of 4)
Third-party product: SoftICE for Windows NT (NuMega)
Runs on same system - e.g., doesn’t require second system for live debugging
x86 only
See

18. Agenda Introduction Tools System Architecture Processes and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processes and Threads
Memory Management

19. Kernel Mode Versus User Mode
A processor state
Controls access to memory
Each memory page is tagged to show the required mode for reading and for writing
Protects the system from the users
Protects the user (process) from themselves
System is not protected from system
Code regions are tagged “no write in any mode”
Controls ability to execute privileged instructions
A Windows NT abstraction
Intel: Ring 0, Ring 3
PerfMon, Processor: “Privileged Time” and “User Time”
Associated with threads
Threads can change from user to kernel and back
Part of saved context, along with registers, etc.
Does not affect scheduling
Components Access mode
Applications User
Subsystem processes User
Executive Kernel
Kernel Kernel
Drivers Kernel
HAL Kernel

20. Getting Into Kernel Mode
Code is run in kernel mode for one of three reasons:
1. Requests from user mode
Via the system service dispatch mechanism
Kernel-mode code runs in the context of the requesting thread
2. Interrupts from external devices
Windows NT-supplied interrupt dispatcher invokes the interrupt service routine
ISR runs in the context of the interrupted thread (so-called “arbitrary thread context”)
ISR often requests the execution of a “DPC routine,” which also runs in kernel mode
Time not charged to interrupted thread
3. Dedicated kernel-mode system threads
Some threads in the system stay in kernel mode at all times (mostly in the “System” process)
Scheduled, preempted, etc., like any other threads
13d: new
13g-1: rearranged major bullets

21. Interrupt Dispatching
user or kernel mode
code
kernel mode
Note, no thread or process context switch!
Interrupt dispatch routine
interrupt !
Disable interrupts
Record machine state (trap frame) to allow resume
Mask equal- and lower-IRQL interrupts
Find and call appropriate ISR
Dismiss interrupt
Restore machine state (including mode and enabled interrupts)
Interrupt service routine
Tell the device to stop interrupting
Interrogate device state, start next operation on device, etc.
Request a DPC
Return to caller

22. Interrupt Precedence Via IRQLs
IRQL = Interrupt Request Level
The “precedence” of the interrupt with respect to other interrupts
Different interrupt sources have different IRQLs
Not the same as IRQ
IRQL is also a state of the processor
Servicing an interrupt raises processor IRQL to that interrupt’s IRQL
This masks subsequent interrupts at equal and lower IRQLs
User mode is limited to IRQL 0 31
Low
APC
Dispatch/DPC
Device 1 .
Device n
Clock
Interprocessor Interrupt
Power fail
High 30 29 28
Hardware interrupts
Deferrable software interrupts 2 1
normal thread execution

23. Interprocessor Interrupt
Alpha IRQLs
IRQL on Alpha implemented in PAL code 7
High 6
Interprocessor Interrupt 5
Clock 4
Device High 3
Device 2
Dispatch/DPC 1
APC
Low

24. DPCs (Deferred Procedure Calls)
A list of “work requests”
One queue per processor (but processors can run each others’ DPCs)
Implicitly ordered by time of request (FIFO)
Used to defer processing from higher (device) interrupt level to a lower (dispatch) level
Used heavily for driver “after interrupt” functions
Used for quantum end and timer expiration
queue head
DPC object
DPC object
DPC object
DfrdCtx
SysArg1
SysArg2
XydriverDpcRtn(DpcObj,
DfrdCtx,SysArg1,SysArg2) {
// ... }

25. Accounting For Kernel-Mode Time
“Processor Time” = total busy time of processor (equal to elapsed real time - idle time)
“Processor Time” = “User Time” + “Privileged Time”
“Privileged Time” = time spent in kernel mode
“Privileged Time” includes:
Interrupt Time
DPC Time
Again note: interrupts and DPCs are not charged to any process or thread
Screen snapshot from: Programs | Administrative Tools | Performance Monitor
click on “+” button, or select Edit | Add to chart…

26. Agenda Introduction Tools System Architecture Processes and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processes and Threads
Memory Management

27. Windows NT Executive Upper layers of operating system
Provides “generic OS” services
Processes, threads, memory management, I/O, interprocess communication, synchronization, security
Almost completely portable C code
Exports functions (“services”) which may be invoked via user-mode APIs
Interface is NTDLL.DLL
E.g., Win32 ReadFile -> executive NtReadFile
Most interfaces to executive services not documented
Used by subsystem writers
TBD: demo/dump/trace of SS call

28. Windows NT Kernel
Abstracts differences between processor architectures
x86 vs. Alpha vs., etc.
Main services
Thread scheduling and context switching
Generic wait operations
Exception and interrupt dispatching
Operating system synchronization primitives (MP and UP)
Not a classic “microkernel”
shares address space withrest of kernel-mode components
Machine Independent C
Assembler
Machine Dep. C

29. HAL - Hardware Abstraction Layer
A separate loaded binary (c:\winnt\system32\hal.dll)
Several different versions for different motherboards, UP vs. MP, etc.
Installation procedure selects appropriate HAL for platform and copies to Hal.Dll on system disk
Purpose:
Isolate (abstract) Kernel and Executive from platform-specific details
Present uniform model for ease of driver development
HAL abstracts:
I/O system specifics (bus interfaces, DMA…)
System timers, Cache coherency and flushing
SMP support, Hardware interrupt priorities
OEM Development Kit needed to buildHALs
HAL contains some Executive and Kernel subroutines
Sample HAL routines:
HalGetBusData
HalGetBusDataByOffset
HalAssignSlotResources
HalSetBusData
HalSetBusDataByOffset
HalTranslateBusAddress
HalGetInterruptVector
HalGetAdapter
READ_REGISTER_ULONG
WRITE_PORT_UCHAR

30. Kernel-Mode Device Drivers
Separate loadable modules (drivername.SYS)
Linked like .EXEs
Linked against NTOSKRNL.EXE and HAL.DLL
Only way to add “kernel extensions” or to access kernel mode system routines
Defined in registry
Same area as Win32 services (t.b.d.)
Differentiated by Type value
View loaded drivers with pstat.exe, drivers.exe
Several types:
“Ordinary” hardware drivers
File system
NDIS miniport, SCSI miniport (linked against port drivers)
Win32K.Sys - Windowing system

31. WDM (Win32 Driver Model)
Extension to Windows NT driver model to support for Plug and Play and Power Management
Allows source/(x86) binary-compatible drivers across Windows 98 and Windows NT 5.0
Non trivial additions to existing drivers:
3 new major IRP types
36 new minor IRPs added
6 new miniport driver types
Supporting WDM affects every area of a driver

32. WDM Drivers What’s covered in WDM: Not covered by WDM:
IEEE 1394 (Firewire)
Universal Serial Bus (USB)
Audio: Speakers, microphone, CODEC
Human Interface Devices: mouse, keyboard, monitor controls, game devices
Still Imaging: Cameras, scanners
Video Devices: Video capture, DVD
Advanced Power and Configuration Interface (ACPI) BIOS support
Not covered by WDM:
Network
Storage
File System
Video

33. Agenda Introduction Tools System Architecture Processes and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processes and Threads
Memory Management

34. Object management / Executive RTL Hardware Abstraction Layer (HAL)
NTOSKRNL.EXE
Environment
Subsystems
Services
System Processes
Applications
POSIX
Replicator
Service Controller
Alerter
OS/2
WinLogon
User
Application
RPC
User
Mode
Session
Manager
Event
Logger
Win32
Subsystem DLLs
System
Threads
NTDLL.DLL
NtosKrnl.Exe
Kernel
Mode
Executive API
I/O Manager
Cache
Manager
Processes
& Threads
Security
Virtual
Memory
Win32
User,GDI
File
systems
Object management / Executive RTL
Device drivers
Kernel
Hardware Abstraction Layer (HAL)
Hardware interfaces (buses, I/O, interrupts, timers, clocks, DMA, cache control, etc.)
Copyright by Microsoft Corporation. Used by permission.

35. NTOSKRNL.EXE NTOSKRNL.EXE HAL.DLL BOOTVID.DLL 13d: new slide
Windows NT executive and kernel
HAL.DLL
Hardware Abstraction Layer - interface to hardware platform
BOOTVID.DLL
Boot video driver
13d: new slide
TBD: 13f: I don’t like it here; it’s too early.

36. Naming Convention For Internal Windows NT Routines
Two- or three-letter component code in beginning of function name
Ex - General executive routine Ob - Object management Exp - Executive private (not exported) Io - I/O subsystem Cc - Cache manager Se - Security Mm - Memory management Ps - Process structure Rtl - Run-Time Library Lsa - Security Authentication FsRtl - File System Run-Time Lib Zw - File access, etc. Ke - Kernel Ki - Kernel internal (not available outside the kernel) Hal - Hardware Abstraction Layer READ_, WRITE_ - I/O port and register access
Executive
Kernel
HAL

37. Multiprocessor Support
Code comprising NTOSKRNL compiled twice: Once for uniprocessor, once for multiprocessor
Avoids penalizing uniprocessor systems for added MP complexity
Two files on Windows NT media:
UP version: NTOSKRNL.EXE
MP version: NTKRNLMP.EXE
Selected at installation time, but copied to NTOSKRNL
All drivers, DLLs, EXEs are built to run on on MP
Upgrading from Uniprocessor vs Multiprocessor
See uptomp.exe (in Resource Kit)
2 files replaced with different code
NTKRNLMP.EXE replaces NTOSKRNL.EXE
new HAL replaces HAL.DLL
4 files replaced with same code, but modified image header
KERNEL32.DLL, NTDLL.DLL, WINSRV.DLL, WIN32K.SYS

38. Identifying Your NTOSKRNL
Build numbers
Incremented each time Windows NT is built from sources (i.e., different for beta releases)
Service packs
Replaces .EXEs (including usually NTOSKRNL), .DLLs, etc.
Do not change Windows NT build number
Free versus Checked build
Free = retail version; Checked = debug version
Used primarily in driver testing
Build number is the same
Recompilation of system with DEBUG flag true
Therefore a different NTOSKRNL.EXE
Note: MP only (NTOSKRNL and NTKRNLMP.EXE identical)
Screen snapshot from:
Programs | Administrative Tools | Windows NT Diagnostics

39. Workstation Vs Server Core operating system executables are identical
NTOSKRNL.EXE, HAL.DLL, xxxDRIVER.SYS, etc., (t.b.d.)
Windows NT Server a superset of Workstation
domains, host-based RAID 5, NetWare gateway, DHCP server, WINS, DNS, full Internet Information Server…
Enterprise Server adds yet more functionality (Clusters, 3GB address space)
Terminal Server enables multi-user thin client support
MP limits: Workstation: 2 CPUs, Server: 4 CPUs, Server Enterprise: 8 CPUs
15a: new slide. This shouldn’t take any extra time

40. Workstation Vs Server Registry indicates system type
HKLM\CurrentControlSet\Control\ProductOptions
ProductType: WinNT=Workstation, ServerNT=Server not a domain controller, LanManNT=Server that is a Domain Controller
ProductSuite: Indicates Enterprise Edition, Terminal Server…
Code in the operating system tests these values and behaves slightly differently in a few places
Licensing limits (number of processors, number of inbound network connections, etc.)
Boot-time calculations (memory manager)
Default length of time slice
See DDK: MmIsThisAnNtasSystem

41. Agenda Introduction Tools System Architecture Processe and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processe and Threads
Memory Management

42. System Threads Internal worker routines that need thread context
Drivers or Executive can create system threads
Always run in kernel mode
Usually associated with the “System” process by default
But can be tied to any process
Not non-preemptible (unless they raise IRQL to 2 or above)
Kernel mode APIs:
PsCreateSystemThread
PsTerminateSystemThread
KeSetBasePriorityThread
KeSetPriorityThread

43. Threads In The “System” Process
Note CPU time is 100% kernel mode
“Start address” is address of thread function
On Intel (at least):
Addresses 8xxxxxxx will correspond to symbols in NtosKrnl.Exe
Addresses Axxxxxxx are routines in Win32K.Sys
Addresses Fxxxxxxx are routines in loaded device drivers
Screen snapshot from: Programs | Resource Kit |
Diagnostics | Process Viewer select “System” process

44. Threads In The “System” Process
Memory Management
Modified Page Writer for mapped files
Modified Page Writer for paging files
Balance Set Manager
Swapper (kernel stack, working sets)
Zero page thread (thread 0, priority 0)
Security Reference Monitor
Command Server Thread
Network
Redirector and Server Worker Threads
Threads created by drivers for their exclusive use
Examples: Floppy driver, parallel port driver
Pool of Executive Worker Threads
Used by drivers, file systems…
Accessed via ExQueueWorkItem
TBD - add list of all thread entry points from a sample system
LAB: find system process with pview; note thread addresses; identify a few by mapping to symbols
LAB: Perfmon, select thread object, priv time, all threads in system process
1-17: Added zero page thread
1-17f: Promoted driver threads to top-level bullet (was formerly under Network)

45. Threads In System Process (Observed on Intel Windows NT Workstation 4
Routine Name
Priority
Notes
Phase1Initialization
First thread in life of system; becomes zero
page thread
ExpWorkerThread
9-16
Pool of worker threads
MiDereferenceSegmentThread 18
Dereferences segments; also expands
paging file
MiModifiedPageWriter 17
Writes modifed pages to paging file
KeBalanceSetManager 16
Reclaims memory from processes, with aid
of . . .
KeSwapProcessOrStack 23
Scheduled by balance set manager
FsRtlWorkerThread
16, 17
Dedicated worker threads for FSDs
SepRmCommandServerThread 15
Security Reference Monitor Command
Server
MiMappedPageWriter
Writes modified pages to mapped files
(Win32 threads)
routines in Win32K.Sys (0xA )
(driver threads)
various
routines in *driver.Sys (0xF )

46. Agenda Introduction Tools System Architecture Processes and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processes and Threads
Memory Management

47. Environment Subsystems
Expose “native API”
“Wrap” and extend Windows NT native functionality
Interfaces to write subsystems not documented
Two main components
Subsystem DLLs - convert documented API to native API
Environment Subsystem Process - maintain state of client processes; implement some subsystem APIs
Three provided with Windows NT:
Win32
Posix
Bare minimum Posix standards, no optional components
OS/2
Support for 1.x character-mode applications only
drv 15a: new slide
13g-1: minor changes

48. Subsystem Extensions OS/2 Posix
Microsoft sells an add-on to the OS/2 subsystem
Supports 1.x Presentation Manager
Posix
OpenNT from SoftWay
More-featured replacement for Posix subsystem

49. Environment Subsystems
Subsystem for each .exe specified in image header
See winnt.h
See Explorer / QuickView (right-click on .exe or .dll file)
Or \reskit\exetype image.exe
IMAGE_SUBSYSTEM_UNKNOWN // Unknown subsystem
IMAGE_SUBSYSTEM_NATIVE // Image doesn't require a subsystem
IMAGE_SUBSYSTEM_WINDOWS_GUI 2 // Win32 subsystem (graphical app)
IMAGE_SUBSYSTEM_WINDOWS_CUI 3 // Win32 subsystem (character cell)
IMAGE_SUBSYSTEM_OS2_CUI // OS/2 subsystem
IMAGE_SUBSYSTEM_POSIX_CUI // Posix subsystem

50. Showing .exe Type With QuickView
In Explorer:
Right-click on an executable file or .DLL
“Context menu” appears
Select Quick View

51. Environment Subsystems Loading
Subsystems to load specified in registry:
\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems
Values:
Required - list of value names for subsystems to load at boot time
Optional - list of value names for subsystems to load when needed
Windows - value giving filespec of Win32 subsystem (csrss.exe)
csrss.exe Win32 APIs required (Client Server Runtime SubSystem)
os2ss.exe OS/2 APIs optional
psxss.exe Posix APIs optional
Kmode - value giving filespec of Win32K.Sys (kernel-mode component of Win32)
Some Win32 API DLLs are in “known DLLs” registry entry:
\SYSTEM\CurrenctControlSet\Control\Session Manager\KnownDLLs
LAB: regedt32: examine above; identify process name for Win32 subsys
1-17F: Added info on exetypes for subsystems.

52. Environment Subsystems Components
Subsystem process
For Win32: CSRSS.EXE
API DLLs
For Win32: Kernel32.DLL, Gdi32.DLL, User32.DLL, etc.
Kernel-mode extension to executive
Win32 only: Win32K.SYS
Environment Subsystems
System
and Server Processes
User
Application
OS/2
POSIX
Win32
Subsystem DLL
User
Mode
NTDLL.DLL
Kernel
Mode
Executive
Win32
User/GDI
Device Drivers
Kernel
Hardware Abstraction Layer (HAL)

53. Windows NT Simplified Architecture (3.51 and earlier)
Environment Subsystems
System
and Server Processes
User
Application
OS/2
Win32
POSIX
User
Mode
Subsystem DLL 1 2
NTDLL.DLL
Kernel
Mode
Executive
LPC
Device Drivers
Kernel
Hardware Abstraction Layer (HAL)
Most Win32 Kernel APIs
All other Win32 APIs, including User and GDI APIs 1 2

54. Windows NT Simplified Architecture (4.0 and later)
Environment Subsystems
System
and Server Processes
User
Application
OS/2
POSIX
Win32
User
Mode
Subsystem DLL 1 2 3
NTDLL.DLL
Kernel
Mode
Win32
User/GDI
Executive
LPC
Device Drivers
Kernel
Hardware Abstraction Layer (HAL)
Most Win32 Kernel APIs
Most Win32 User and GDI APIs
A few Win32 APIs 1 2 3

55. (Reduced) Role Of Win32 Subsystem Process
Process creation and deletion
Thread creation and deletion
Get temporary file name
Drive letters
Security checks for file system redirector
Window management for console (character cell) applications
Some support for 16-bit DOS support (NTVDM.EXE)
1-17: and also “NLS local stuff”. What is NLS? Ans National Locale Services

56. Agenda Introduction Tools System Architecture Processes and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processes and Threads
Memory Management

57. Invoking System Functions From User Mode
Kernel-mode functions (“services”) are invoked from user mode via a protected mechanism
x86: INT 2E; Alpha: SYSCALL (PALcode)
I.e., on a call to an OS service from user mode, the last thing that happens in user mode is this “change mode to kernel” instruction
Causes an interrupt, handled by the system service dispatcher (KiSystemService) in kernel mode
Return to user mode is done by dismissing the interrupt or exception
The desired system function is selected by the “system service number”
Every Windows NT function exported to user mode has a unique number
Push this number on the stack just before the “change mode” instruction (after pushing the arguments to the service)
This number is an index into the system service dispatch table
Table gives kernel-mode entry point address and argument list length for each exported function
LAB: do commands
1-17: moved the ntoskrnl.exe line here from next slide, makes a good lead-in
TODO: repro output from these commands in handout
13d: included Alpha SYSCALL
13g: Moved this here. Transition from previous sectiion: “Now the question comes up, how do we get from user to kernel mode? In other words, how does the system allow applications running in user mode, to request functions which must be performed in kernel mode, without giving away the system’s security?”
13g-1: minor changes
15a: several more details added here and subsequent slides. This is one of those “we say this stuff anyway, so why not have it in the slides” poitns. Or at least I say it…
15a: note, this is a “bottom-up” approach. We talked about the exec kernel and HAL first. Then we describe the INT 2e. then we talk about what “wraps” the INT 2e.

58. Invoking System Functions From User Mode
All validity checks are done after the user to kernel transition
KiSystemService probes argument list, copies it to kernel-mode stack, and calls the executive or kernel routine pointed to by the table
Service-specific routine checks argument values, probes pointed-to buffers, etc.
Once past that point, everything is “trusted”
This is safe, because:
The system service table is in kernel-protected memory; and
The kernel mode routines pointed to by the system service table are in kernel-protected memory; therefore:
User mode code can’t supply the code to be run in kernel mode; it can only select from among a predefined list
Arguments are copied to the kernel mode stack before validation; therefore:
Other threads in the process can’t corrupt the arguments “out from under” the service

59. NTDLL.DLL
PUSH of service # and INT 2E are “wrapped” by small “jacket” procedures in NTDLL.DLL
These user-mode routines have the same function names and arguments as the kernel mode routines they call
E.g., NtWriteFile in NtDll.Dll invokes NtWriteFile in NtosKrnl.Exe
Therefore exports of NTDLL are the “NT native API”
Entry points in NtDll.Dll are not supported or documented for use from user mode apps
A few are documented in the DDK for call from kernel mode
A few images that come with Windows NT are written to the “native API” exposed by NtDll.Dll (“Windows NT native images”)
See article on
NTDLL also contains image loader and other support functions
What about getting to USER and GDI functions in Win32K.SYS?
System service wrapper exists in USER32.DLL, GDI32.DLL
Does not go through NTDLL.DLL

60. Tracing An Example Win32 Call
call WriteFile(…)
Win32 application
WriteFile
in Kernel32.Dll
call NtWriteFile
return to caller
Win32-
specific
NtWriteFile
in NtDll.Dll
Int 2E
return to caller
used by all
subsystems U K
software interrupt
call NtWriteFile
dismiss interrupt
KiSystemService
in NtosKrnl.Exe
Source: MSJ, August 1996, page 21
(by Matt Pietrek)
NtWriteFile
in NtosKrnl.Exe
do the operation
return to caller

61. Tracing An Example Win32 Call
Depends.Exe in Resource Kit and Platform SDK
Allows viewing of image->DLL relationships, imports, and exports

62. Examining Symbols In Key Images
Examine imports and exports of an .EXE down to the OS
In Explorer, right mouse click on EXE or DLL, then “quick view” (built in) or “View Dependencies” (Dependency Walker tool in ResKit and Platform SDK)
Or use LINK /DUMP /EXPORTS, /IMPORTS
1. Look at imports of \winnt\system32\notepad.exe
2. Look at exports and imports of kernel32.dll
Most of the exports are documented Win32 calls
3. Look at exports and imports of ntdll.dll
None of the exports are documented
Some are the same as exports from ntoskrnl.exe, documented in DDK, with identical
13h: added reference to Dependency Walker
LAB: do the link/dump/exports command
(V4: right-click, quick view on ntoskrnlexe does this)
Link /dump /imports advapi32.dll
LAB: set up to write a dump file (cpnl)
also add a boot.ini line for /maxmem=8
boot with 8MB
crash NT (tlist, kill <pid> for csrss, system crashes
set _NT_SYMBOL_PATH to where symbols are
i386kd -Z <dumpfilespec>
.logopen filename
x *
LABEND
ntdll.dll - exposes executive routines to e.g. native subsystems and Win32 DLLs (its entry points run in user mode)
TBD: Clean this up, add some quickview screen snapshots

63. Examining Symbols In Key Images
4. Look at exports and imports of ntoskrnl.exe
About 1000 total exported symbols
About 300 of the exported routine names are documented in DDK
Callable only from kernel mode
5. Look at all global symbols in ntoskrnl.exe
Defined in \support\symbols\xxx\debug\exe\ntoskrnl.dbg
Quick viewer won’t display - use Kernel Debugger “x *” with just this .dbg file loaded
About 4000 total symbols (Includes executive data cells in addition to routines)
Exports of ntoskrnl.exe are a subset of this list

64. Agenda Introduction Tools System Architecture Processes and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processes and Threads
Memory Management

65. Process-Based Windows NT Code
Pieces of Windows NT that run in separate executables (.exe’s), in separate processes
Started by system
Not tied to a user logon
Have full process context
Three types:
Environment Subsystems (already described)
Win32 Services
System startup processes
Note: “system startup processes” is not an official MS-defined name
LAB; Tlist/t - process “origin” hierarchy
13b: Moved tlist output to a later slide
13d: changed “system processes” to “system startup processes”

66. Process Creation Hierarchy
tlist.exe (from resource kit)
tlist /t shows creation hierarchy
Creating process can exit, leaving created process running - hence this display does not show all creators
Explorer.exe is actually started by userinit.exe, which then exits

67. Process-Based Windows NT Code Win32 services
Win32 .EXEs (applications) that run independently of a logged on user
Start at boot or logon time, survive logoff
Defined by CreateService API - view through Control Panel
See srvany.exe, sc.exe, srvinstw.exe, instsrv.exe in Resource Kit
Typically do not interact with the desktop
Get startup configuration parameters from Registry
Log errors to Windows NT Event Log
Use some form of IPC mechanism for client communication and control
Services will likely make use of Windows NT security impersonation
Remotely manageable (start, stop, user-defined codes)
Server Manager allows remote control of services
Code is the same to control services locally vs. remotely
Examples of built-in Windows NT Services
Schedule service (at command), Event Log, Remote Access Server, etc.
16e: new slides

68. Life Of A Service Install time Management / maintenance Registry
Setup application tells Service Controller about the service
System boot / initialization
SCM reads registry, starts services as directed
Management / maintenance
Control panel can start and stop services and change startup parameters
Service
Controller
Registry
Setup
Application
CreateService
Service
Processes
Control
Panel

69. Where Are Services Defined?
Maintained in Windows NT Registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
One key per installed service
Mandatory information kept on each service:
Type of service (Win32, Driver…)
Imagename of service .EXE
NOTE: Some service .EXEs contain more than one service
Start type (automatic, manual, or disabled)
Optional information:
Display Name
Dependencies
Account and password to run under
Can store application-specific configuration parameters
“Parameters” under service key
16e: new slides
LAB: identify a few running services (which process they are in);
- find both a Win32 “own” and Win32 “share” service
- find services inside services.exe process (dump registry services key into a file; search for services.exe)
1-17f: Moved

70. Process-Based Windows NT Code System startup processes
Separate processes loaded or started at boot time (not as services or environment subsystems)
Names of images are not in registry
“Hardwired” in the source code
Most are Win32 executables, one (smss) is a “native image”
(Idle) Process id 0 Part of the loaded system image Home for idle thread(s) (not a real process nor real threads) Called “System Process” in many displays
(System) Process id 2 Part of the loaded system image Home for kernel-defined threads (not a real process) Thread 0 (routine name Phase1Initialization) launches the first “real” process, running smss.exe… …and then becomes the zero page thread
1-17F see previous slide notes. Did this so we can meaningfully describe “system processes” by exclusion. Also, removed references to “native images” (that’s been moved to the environment subsystems section).
13d; More details, split into a second slide

71. Process-Based Windows NT Code System startup processes
smss.exe Session Manager The first “created” process Takes parameters from \Registry\Machine\System\CurrentControlSet\Control\Session Manager Launches required subsystems (csrss) and winlogon
winlogon.exe Logon process Presents first login prompt Presents “enter username and password” dialog Launches services.exe, lsass.exe, and nddeagnt.exe When someone logs in, launches userinit.exe
services.exe Service Controller; also, home for many NT-supplied services Starts processes for services not part of services.exe (driven by \Registry\Machine\System\CurrentControlSet\Services )
lsass.exe Local Security Authentication Server
userinit.exe Started after logon; starts desktop (Explorer.Exe) and exits (hence does not show up in tlist output; Explorer appears to be an orphan)
explorer.exe and its children are the creators of all interactive apps

72. Agenda Introduction Tools System Architecture Processes and Threads
Kernel Mode Environment
Executive, Kernel, HAL, Drivers
Product Packaging
System Threads
Environment Subsystems
System Service Dispatching
Process-based Windows NT code
Summary
Processes and Threads
Memory Management

73. Four Contexts For Executing Code
Full process and thread context:
User applications
Win32 Services
Environment subsystem processes
System startup processes
Have thread context but no “real” process:
Threads in “System” process
Routines called by other threads / processes:
Subsystem DLLs
Executive system services (NtReadFile, etc.)
GDI routines in Win32K.Sys (and graphics drivers)
No process or thread context
(“Arbitrary thread context”)
Interrupt dispatching
Device drivers

74. Where Is The Code? TBD: What is ntdll.dll? 1-17g new slide
Kernel32.Dll, Gdi32.Dll, User32.Dll
Export Win32 entry points
NtDll.Dll
Provides user-mode access to system-space routines
Also contains heap manager, image loader, thread startup routine
Ntoskrnl.Exe (or Ntkrnlmp.exe)
Executive and kernel
Includes most routines that run as threads in “system” process
Win32K.Sys
The loadable module that includes the now-kernel-mode Win32 code (formerly in csrss.exe)
Hal.Dll
Hardware Abstraction Library
drivername.Sys
Loadable kernel drivers
TBD: What is ntdll.dll?
1-17g new slide
13a: added second bullet on contents of NTDLL.DLL

75. Agenda Introduction Tools System Architecture Processes and Threads
Memory Management

76. Systemwide Address Space
Processes And Threads
Per-process
address space
Thread
What is a process?
Represents an instance of a running program
You create a process to run a program
Starting an application creates a process
Primary argument to CreateProcess is image file name (or command line)
What is a thread?
An execution context within a process
Primary argument to CreateThread is a function entry point address
All threads in a process share the same per-process address space
Every process starts with one thread
Running the program’s “main” function
Can create other threads in the same process
Can create additional processes
Thread
Thread
Systemwide Address Space

77. Tools To Examine Processes
Task Manager
Performance Monitor
pviewer.exe (pview in Platform SDK): shows processes, threads within processes, memory details
pview.exe (process explode): thread and process ACLs and tokens
tlist.exe - tlist /t shows parent/child relationships
QuickSlice
qlice.exe
CPU usage by process, and by thread within each process
Pulist - process user list
Vadump - dump virtual address space of a process
1-17F: new slide

78. Tools To Examine Processes
Page fault monitor (pfmon.exe)
Shows page fault type and origin of subject application
Can provide data to working set tuner (part of Platform SDK)
Pstat
pstat.exe (char mode, no icon)
One-time snapshot of system
Shows state of threads within all processes, with wait reasons
Kernel debugger
Shows various internal structures
See Windows NT® Workstation Resource Kit documentation
oh.exe (ResKit), nthandleex ( - show open handles
Ntpmon (

79. Windows NT 5.0 Job Object
New kernel object to collect a group of related processes
CreateJobObject/OpenJobObject
System enforces job quotas and security context
Limits: Total and current CPU time, total and active processes, per-process and per-job CPU time, min and max working set, CPU affinity, priority class
Security limits: No administrators token, only restricted token, only specific token, filter token, no accessing windows outside the job, no reading/writing the clipboard
To examine: See new performance counters + new !job command in kernel debugger

80. Processes And Threads Internal Structures
Access token
VAD
VAD
VAD
Process object
Virtual address space descriptors
See kernel debugger commands:
!processfields
!threadfields
!process
!thread
!tokenfields
!token
!handle
!object
Handle table
object
object
TBD: Description, picture of VAD
LAB: kdbg, !processfields
!threadfields
!process 0
Thread
Thread
Thread …
Access token

81. !processfields Pcb: 0x0 ForkInProgress: 0x158 ExitStatus: 0x68
LockEvent: x6c
LockCount: x7c
CreateTime: x80
ExitTime: x88
LockOwner: x90
UniqueProcessId: x94
ActiveProcessLinks: x98
QuotaPeakPoolUsage[0]: xa0
QuotaPoolUsage[0]: xa8
PagefileUsage: 0xb0
CommitCharge: xb4
PeakPagefileUsage: xb8
PeakVirtualSize: xbc
VirtualSize: xc0
Vm: xc8
LastProtoPteFault: xf8
DebugPort: xfc
ExceptionPort: x100
ObjectTable: x104
Token: x108
WorkingSetLock: x10c
WorkingSetPage: x12c
ProcessOutswapEnabled: x130
ProcessOutswapped: x131
AddressSpaceInitialized: 0x132
AddressSpaceDeleted: x133
AddressCreationLock: x134
ForkInProgress: x158
VmOperation: x15c
VmOperationEvent: x160
PageDirectoryPte: x164
LastFaultCount: x168
VadRoot: x170
VadHint: x174
CloneRoot: x178
NumberOfPrivatePages: x17c
NumberOfLockedPages: x180
ForkWasSuccessful: x184
ExitProcessCalled: x186
CreateProcessReported: x187
SectionHandle: x188
Peb: x18c
SectionBaseAddress: x190
QuotaBlock: x194
LastThreadExitStatus: x198
WorkingSetWatch: x19c
LpcPort: x1a0
InheritedFromUniqueProcessId: 0x1a4
GrantedAccess: x1a8
DefaultHardErrorProcessing x1ac
LdtInformation: x1b0
VadFreeHint: x1b4
VdmObjects: x1b8
ProcessMutant: x1bc
ImageFileName[0]: x1dc
VmTrimFaultValue: x1ec

82. !threadfields
Tcb: x0
CreateTime: x1b0
ExitTime: x1b8
ExitStatus: x1c0
PostBlockList: x1c4
TerminationPortList: x1cc
ActiveTimerListLock: x1d4
ActiveTimerListHead: x1d8
Cid: x1e0
LpcReplySemaphore: x1e8
LpcReplyMessage: x1fc
LpcReplyMessageId: x200
Client: x208
IrpList: x20c
TopLevelIrp: x214
ReadClusterSize: x21c
ForwardClusterOnly: x220
DisablePageFaultClustering: 0x221
DeadThread: x222
HasTerminated: x223
EventPair: x224
GrantedAccess: x228
ThreadsProcess: x22c
StartAddress: x230
Win32StartAddress: x234
LpcExitThreadCalled: x238
HardErrorsAreDisabled: x239
1-17: someone is sure to ask why there’s no token field here when there is a token field in the processfields output.
TBD: Screen dumps of !process, !thread, !handle (!handle maybe not here)
1-17f: If we want to keep the listings as part of the handout, we can drop these two slides.

83. Looking At Waiting Threads
13d: new slide
pstat.exe (Resource Kit)
Shows state of every thread in every process
But for threads that are waiting, that’s all we know…

84. Looking At Waiting Threads
!thread command in kernel debugger shows what a thread is waiting on
13d: new slide

85. Object-type-specific data
Dispatcher Objects
Any kernel object you can wait for is a “dispatcher object”
Some exclusively for synchronization
E.g., events, mutexes (“mutants”), semaphores, queues, timers
Others can be waited for as a side effect of their prime function
E.g., processes, threads, file objects
Non-waitable kernel objects are called “control objects”
All dispatcher objects have a common header
All dispatcher objects are in one of two states
“Signalled” versus “nonsignalled”
When signalled, a wait on the object is satisfied
Different object types differ in terms of what changes their state
Wait and unwait implementation is common to all types of dispatcher objects
Dispatcher object
12B: new
Size
Type
State
Wait listhead
Object-type-specific data
(see \ddk\inc\nttddk.h)

86. Object-type-specific data Object-type-specific data
Wait Blocks
Thread Objects
WaitBlockList
Key
Type
Next link
List entry
Object
Thread
Key
Type
Next link
List entry
Object
Thread
Key
Type
Next link
List entry
Object
Thread
Represent a thread’s reference to something it’s waiting for (one per handle passed to WaitFor…)
All wait blocks from a given wait call are chained to the waiting thread
Type indicates wait for “any” or “all”
Key denotes argument list position for WaitForMultipleObjects
WaitBlockList
Dispatcher
Objects
Wait blocks
Size
Type
State
Wait listhead
Object-type-specific data
1-17: ta-da! (also see next slide)]
13d: I know we talked about hiding this - but I don’t see how we can not talk about this stuff in an “internals” class. You did fine on it in Seattle…
Size
Type
State
Wait listhead
Object-type-specific data

87. Agenda Introduction Tools System Architecture Processes and Threads
Memory Management
Virtual Address Space Layout
Process Memory Usage
Global System Cache
System Memory Usage

88. 4GB Virtual Address Space
2 GB per-process
Address space of one process is not directly reachable from other processes
2 GB systemwide
The operating system is loaded here, and appears in every process’s address space
There is no process for “the operating system” (though there are processes that do things for the OS, more or less in “background”)
.EXE code
Globals
Per-thread user mode stacks
Process heaps
.DLL code
Unique per process, accessible in user or kernel mode
7FFFFFFF
Per process, accessible only in kernel mode
Exec, Kernel, HAL, drivers, per-thread kernel mode stacks, Win32K.Sys
File system cache
Paged pool
Non-paged pool
13H: combined 2 separate address space slides into one; indicated page tables here, rather than waiting until MM section
15a: added description C
Process page tables,
hyperspace
System wide,
accessible only in kernel mode
FFFFFFFF 7

89. System Space Layout x86 Alpha AXP 80000000 80000000
System code (NTOSKRNL, HAL, boot
drivers); initial nonpaged pool
System code (NTOSKRNL, HAL,
boot drivers) and initial nonpaged pool A
System Mapped Views (e.g. WIN32K.SYS)
or session space (Terminal Server only) A
Additional System PTEs (& big cache) C
Process Page Tables and Page Directory C
Process Page Tables and Page Directory C
Hyperspace and process working set list C
Hyperspace and process working set list C
Unused No Access C
Unused No Access C
System Working Set List
C0C00000
System Working Set List C
System Cache C
System Cache
DE000000
System Mapped Views (e.g. WIN32K.SYS) E
Paged Pool E
Paged Pool
EB (min)
System PTEs
EB (min)
System PTEs
Non-Paged Pool expansion
Non-Paged Pool expansion
FFBE0000
Crash dump information
FFC00000
FDFEC000
Crash dump information & HAL usage
HAL usage

90. 3GB Process Space Option
Only available on x86 Server Enterprise Edition
Boot with /3GB option in BOOT.INI
Chief “loser” in system space is file system cache
Expands per-process address space
But image must be marked as “large address space aware”
A stopgap while we wait for 64-bit Windows NT (Merced and Alpha; post-Windows NT 5.0)
.EXE code
Globals
Per-thread user mode stacks
.DLL code
Process heaps
Unique per process, accessible in user or kernel mode
Unique per process
(= per appl.),
user mode
Per process, accessible only in kernel mode
15a: added description
BFFFFFFF C
Process page tables,
hyperspace
System wide,
accessible only in kernel mode
Exec, kernel, HAL,
drivers, etc.
FFFFFFFF

91. 64-bit Very Large Memory In Windows NT 5.0
2GB user space
2GB user space
FFFFFFF
2GB process space
Alpha Windows NT Server Enterprise Edition only
Referenced by 64-bit pointers
Cannot be paged out - must be resident at all times
Cannot be used for code, only data file mapping
New APIs: VirtualAllocVlm, MapViewOfFileVlm, Read/WriteFileVlm, Read/WriteProcessMemoryVlm, etc.)
Yet another stopgap prior to 64-bit Windows NT
28GB Large Memory Area
13H: new slide
15a: cleaned up to look parallel to previous two slides
TBD: these ranges appear disjoint - fix
FFFFFFFF
Invalid (inaccesible)
(about 1.8x10^19 bytes; not to scale!)
FFFFFFFF 7FFFFFFF
FFFFFFFF
2GB system space
FFFFFFFF FFFFFFFF

92. Application Startup Maps V.A.S. To Code On Disk
7FFFFFFF
.exe
.dll
paging file
See link/dump/header, or QuickView for .exe’s and .dll’s
CreateFileMapping, MapViewOfFile simply make the mechanism available to application-level code
All of these files may simultaneously be mapped by other processes

93. Process Virtual Address Layout
LAB: Run process walker against something
\ntreskit\perftool\meastool\vadump <pid>
(can just attach to process)
Screen snapshot from: Programs | SDK Tools | Process Walker
Process | Load Process | notepad

94. Agenda Introduction Tools System Architecture Processes and Threads
Memory Management
Virtual Address Space Layout
Process Memory Usage
Global System Cache
System Memory Usage

95. Process Memory Usage
Working set: All the physical pages “owned” by a process
Essentially, all the pages the process can reference without incurring a page fault
Upper limit on size for each process
When limit is reached, a page must be released for every page that’s brought in (“working set replacement”)
Working set limit: The maximum pages the process can own
Maximum is calculated as (available pages pages)
Result stored in MmMaximumWorkingSetSize
LAB: Perfmon - process working sets
pmon - all working sets (text output)
An interesting thing about the working set is that it starts out nearly empty. There is no part of the executable or DLLs that are brought into the working set automatically as part of image startup; the first page of code is brought in because of the page fault that happens when the StartOfThread function (described previously) calls the application’s main() function.

96. Working Set List A FIFO list for each process
newer pages
older pages
PerfMon
Process “WorkingSet”

97. Working Set Replacement
PerfMon
Process “WorkingSet”
To standby or modified page list
add diagram showing many processes’ working set pages scattered through physical memory
When working set “count” = working set size, must give up pages to make room for new pages
Page replacement is ”modified FIFO”
MP x86 and Alpha: no regard to accessed bit
Windows NT 5.0 on uniprocessor x86 takes into account age

98. Locking Pages
Pages may be locked into the process working set
Locked pages are guarenteed in physical memory (“resident”) when any thread in process is executing
Win32:
status = VirtualLock(baseAddress, size);
status = VirtualUnlock(baseAddress, size);
Number of lockable pages is a fraction of the maximum working set size
Changed by SetProcessWorkingSetSize
Pages can be locked into physical memory (by drivers only)
Pages are then immune from outswapping as well as paging MmProbeAndLockPages
13b: moved here. “If you don’t like the page replacement strategy, you can remove some of your VAS from consideration for paging…”

99. Memory Management Information Task manager processes tab2 4 1 3
“Mem Usage” = physical memory used by process (working set size, not working set limit)
“VM Size” = private (not shared) committed virtual space in processes
“Mem Usage” in status bar is total of “VM Size” column/maximum allowed - i.e., same as “commit charge” in “Performance” tab (see next slide) - not same as “Mem Usage” column here! 1 2 3
13b: new slide
13f: added new subsection with all of these “memory displays” at the end. Added new comment on bogus-named “mem usage” in status bar. Added callouts (consistent among these slides) to show relationships among displays and link displays to text. Oddly, “mem usage” in status bar is not total of “VM size” in processes list. 4
Screen snapshot from : Task Manager | Processes tab

100. Memory Management Information PerfMon - process object
“Working Set” = working set size (not limit)
“Private Bytes” = same as “VM Size” from Task Manager Processes list
“Virtual Bytes” = committed virtual space, including shared pages 1 2 6
13b: new slide
13d: Is “virtual bytes” really just the committed virtual space? Or does it include reserved? 2 6 1
Screen snapshot from: Performance Monitor counters from Process object

101. Memory Management Information Task manager performance tab
“Commit charge total” = total of private (not shared) committed virtual space in all processes (i.e. total of “VM Size” from processes display)
“Commit charge limit” = sum of available physical memory + free space in paging file 3 4 3 4
13b: new slide
13d: added info on “file cache”
13d: fixed info on “file cache” to be physical vs. virtual
Screen snapshot from: Task Manager | Performance tab

102. Agenda Introduction Tools System Architecture Processes and Threads
Memory Management
Virtual Address Space Layout
Process Memory Usage
Global System Cache
System Memory Usage

103. File System Virtual Block Cache
Shared by all file systems (local or remote)
Caches all files
Including file system metadata files
Virtual block cache (not logical block)
Managed in terms of blocks within files, not blocks within partition
Uses standard Windows NT virtual memory mechanisms
Coherency maintained between mapped files and read/write access
Virtual size: mb (960MB if large cache size set)
In system virtual address space, so visible to all
Divided into 256kb “views”

104. Cached File Operations
Open a file:
Find an available view
Map the first 256kb of the file into the view
Read from or write to a cached file:
Remap as necessary to map referenced section of file into the cache
Copy data between application buffer and cache’s virtual address space
Actual I/O is due to paging
Process
address
space
System
address
space
File

105. Fast I/O Fast I/O path 13d: moved “fast I/O” info here Cache Manager
I/O Subsystem API (Ntxxx)
Fast I/O
path
I/O Manager (Ioxxx)
Driver
Support
Routines
(Io, Ex,
Ke, Mm,
Hal, FsRtl,
...)
File System drivers
(e.g. NTFS)
Fast I/O path
Allows executive I/O APIs to access cache directly
Bypasses file system driver
Bypasses IRP generation, probe-and-lock of user buffer, etc.
13d: moved “fast I/O” info here
Disk device driver
HAL I/O access routines
I/O ports and registers

106. Cache Size Physical size: Depends on available memory
Competes for physical memory with processes, paged pool, pageable system code
Part of “system working set”
Automatically expanded / shrunk by system
Normal working set adjustment mechanisms
Relies on Memory Manager for global memory policy
Performance Monitor: Memory object | System cache resident bytes shows current physical space occupied by cache
See \SYSTEM\CurrentControlSet\Control\Session Manager\ Memory Management\LargeSystemCache
Default is 0 for both Workstation and Server
1 = favor system working set vs. process working set
also allows cache to be >512MB virtual size
Can modify with Control Panel->Network->Services-> Server properties
13d: separated this

107. Cache Functions And Control
Automatic asynchronous readahead
Done by separate “Readahead” system thread
64kb readaheads by default
Predicts next read location based on history of last 3 reads
Readahead hints can be provided to CreateFile:
FILE_FLAG_SEQUENTIAL does 192kb read ahead
FILE_FLAG_RANDOM_ACCESS disables read ahead
Write-back, not write-through
Dirty page threshold forces writing
Small system: Physical Pages / 8; medium system: Physical Pages / 4
Large system: add above 2 together
“Lazy writer” thread queues 1/4 of dirty pages every second to separate “Write Behind” system thread (note, does not flush mapped files)
Can override via CreateFile with FILE_FLAG_WRITE_THROUGH
Or explicitly call FlushFileBuffers when you care (does flush mapped files)
13d: added note on mapped files

108. Cache Functions And Control
Can disable cache completely on a per-file basis
CreateFile with FILE_FLAG_NO_BUFFERING
Requires reads/writes to be done on sector boundaries
Buffers must be aligned in memory on sector boundaries
13d: Moved this here

109. Agenda Introduction Tools System Architecture Processes and Threads
Memory Management
Virtual Address Space Layout
Process Memory Usage
Global System Cache
System Memory Usage

110. System Paged Memory
Just as processes have working sets, Windows NT’s pageable system-space code and data lives in the “system working set”
Cache is one of 4 components of “system working set”
Pageable components of system working set:
Paged pool
Pageable code and data in the exec
Pageable code and data in kernel-mode drivers, Win32K.Sys, graphics drivers, etc.
Global file system data cache
To get physical (resident) size of these with PerfMon, look at:
Memory | Pool Paged Resident Bytes
Memory | System Code Resident Bytes
Memory | System Driver Resident Bytes
Memory | System Cache Resident Bytes
Memory | Cache bytes counter is total of these four “resident” (physical) counters (not just the cache; same as “File Cache” on Task Manager / Performance tab
TBD: Doesn’t the cache have its own working set? Can’t believe that it’s limited to 450 pages
13d: fixed to reflect that “cache bytes” is physical

111. Sessions
New memory management object to support Windows NT® Server 5.0
All processes in an interactive session share a:
Session-specific copy of Win32K.Sys
Instance of Winlogon
Session working set
System code (NTOSKRNL, HAL, boot
drivers); initial nonpaged pool A
Win32k.sys *8MB) A
Session Working Set Lists
x86
Mapped Views for Session
Paged Pool for Session
A0C00000 A

112. System Nonpaged Memory
Nonpageable components:
Nonpageable parts of NtosKrnl.Exe, drivers
Nonpaged pool (see PerfMon, Memory object: Pool nonpaged bytes)
To get size of nonpageable system code, run \ntreskit\pstat.exe & add columns 1 & 2
non-paged code
non-paged data
pageable code+data
output of “drivers” (\ntreskit\drivers.exe) is similar
Win32K.Sys is paged, even though it shows up as nonpaged 7 9 8 7
TBD: maybe make this later?
TBD: show how to get sizes
TBD: add some stuff on perfmon paging counters - page faults vs. page read IOs vs. pages read, etc. 8 9

113. Monitoring Pool Usage Poolmon.exe in \support\debug
Must first turn on pool tagging with gflags
“p” to toggle between nonpaged, paged pool, or both
Sorting:
“b” to sort by total # of bytes
“a” to sort by # of allocations
“t” to sort by structure tag

114. “Free” Memory
System keeps unassigned physical pages (those not part of any working set) on five lists
Free page list
Modified page list
Standby page list
Zero page list
Bad page list - pages that failed memory test at system startup
13f: slight rearrangement of text in this section

115. Managing Physical Pages
demand zero page faults
pages read from disk
zero
page
thread
Standby
Page List
Free
Page List
Zero
Page
List
“soft”
page
faults
Bad
Page
List
Process
Working
Sets
working set replacement
modified
page
writer
LAB: Perfmon - available bytes
TBD: change build order to match slides - or change slides
Modified
Page List

116. Memory Management Information Task manager performance tab1
“Available” memory = total of free, zero, and standby lists (majority usually are standby pages)
“File cache” is really total physical size of pageable portions of: paged pool, NtosKrnl.Exe code and data, drivers code and data, and file system cache (same as PerfMon “cache bytes” counter)
“Kernel Memory Paged” is resident size of paged pool
“Kernel Memory Nonpaged” is actual size of nonpaged pool 3 2 1 4 2
13b: new slide
13d: added info on “file cache”
13d: fixed info on “file cache” to be physical vs. virtual 3 4
Screen snapshot from: Task Manager | Performance tab

117. Summary: Accounting For Physical Memory Usage
Process working sets
Perfmon: Process / Working set
Note, shared resident pages are counted the process working set of every process that’s faulted them in
Hence, the total of all of these may be greater than physical memory
Nonpageable system code (NTOSKRNL + drivers, including win32k.sys &graphics drivers)
See total displayed by DRIVERS utility in Windows NT Resource Kit
Nonpageable pool
Perfmon: Memory / Pool nonpaged bytes
Free, zero, and standby page lists
Perfmon: Memory / Available bytes
Or: Task Manager / Performance tab: Physical memory: Available
Pageable, but currently-resident, system-space memory
Perfmon: Memory / Pool paged resident bytes
Perfmon: Memory / System cache resident bytes
Perfmon: Memory / System code resident bytes
Perfmon: Memory / System driver resident bytes
Memory | Cache bytes counter is really total of these four “resident” (physical) counters
Modified, Bad page lists
can only see size with !memusage command in Kernel Debugger
1-17: new slide
LAB: add up NT physical memory usage of system space:
TBD: eventually add a “loadable mdules” slide showing pieces of the exec and kernel; move this last bullet there
13d: Reformatted, added info

118. Windows NT Internals Information Sources
Books
Inside Windows NT (Solomon, MS Press)
Advanced Windows (Richter, MS Press)
Windows NT Workstation Resource Guide (MS Press)
MSDN Library
Platform SDK API documentation
Windows NT Device Driver Kit (DDK) documentation
Win32 Knowledge Base - has some Windows NT internals articles
Past
Windows NT conferences audio/video tapes (
- Windows NT internals articles and tools
- hardware developers and driver writers
- Installable File System Developers Kit
comp.os.ms-windows.programmer.nt.kernel-mode - drivers newsgroup
- Windows NT device driver FAQ