Windows NT® Internals David Solomon David Solomon Expert Seminars Microsoft Corporation

Agenda Introduction Tools System Architecture Processes and Threads Memory Management

About The Speaker David Solomon 14 years at Digital - the last 10 as a developer in the VMS operating system development group Started Windows NT developer training company in 1992 Author of Inside Windows NT, 2nd edition (Microsoft Press) and Windows NT for OpenVMS Professionals (Digital Press) Regular speaker at industry conferences (WinDev, Tech•Ed, Software Development, DECUS...) Recipient of past Microsoft MVP award for MSWIN32 technical support

About The Company David Solomon Expert Seminars offers high-quality Windows developer training Taught by well known industry experts and authors who develop and teach their own courses Instructors include: Doug Boling, Brian Catlin, Jamie Hanrahan, Jeff Prosise, Jeffrey Richter, and David Solomon Topics include: Windows CE Windows NT Internals Windows NT and WDM Device Drivers Windows NT® Server Applications Win32® Programming Visual C++® and MFC COM/ActiveX® Programming To be notified of new classes and other developments, join our e-mail interest list

Session Goals Goals Audience assumptions Acknowledgements Explain internal architecture and operation of core Windows NT components Use various tools that demonstration internal Windows NT behavior Audience assumptions Familiar with basic 32-bit OS concepts Familiar with Win32 API (processes, threads, memory management) Acknowledgements Jamie Hanrahan (jeh@cmkrnl.com - www.cmkrnl.com), co-author of the Windows NT internals seminar from which these slides were taken Dave Cutler, Helen Custer, John Balciunas, Lou Perazzoli, Mark Lucovsky, Steve Wood, Tom Miller, Gary Kimura, and Landy Wang for their support and assistance in understanding Windows NT internals 1-17e: Changed to “goals and prereqs” (seemed more friendly)

Windows NT Architecture Environment Subsystems System Processes Services Applications POSIX Replicator Service Controller Alerter OS/2 WinLogon User Application RPC Session Manager Event Logger Win32 User Mode Subsystem DLLs System Threads NTDLL.DLL Kernel Mode Executive API I/O Manager Cache Manager Processes & Threads Security Virtual Memory Win32 User,GDI File systems Object management / Executive RTL Device drivers Kernel Hardware Abstraction Layer (HAL) Hardware interfaces (buses, I/O, interrupts, timers, clocks, DMA, cache control, etc.) Copyright by Microsoft Corporation. Used by permission.

Windows NT 5.0 Internal changes In one sense, much is the same Basic architecture of many components unchanged: Win32 subsystem, memory manager, process model, thread scheduling, security model, file system But lots of additions of major new functionality: Active Directory, distributed security, Kerberos, Microsoft management console, IntelliMirror™, NTFS extensions (content indexing, quotas, reparse points, sparse files, link tracking)

Windows NT 5.0 Internal changes Kernel/core changes include: I/O system (plug and play and power management) 64-bit Very Large Memory support for Alpha Job object Integration of Terminal Server Comparable to level of change from 3.51 to 4.0 Also many incremental performance improvements: Object Manager, Memory manager (e.g., working set management algorithms), SMP scalability…

Agenda Introduction Tools System Architecture Processes and Threads Memory Management

Tools Preview tool executable origin Performance Monitor PerfMon Windows NT Registry Editor RegEdt32 Windows NT Windows NT Diagnostics WinMSD Windows NT Kernel Debugger i386kd, Widows NT CD \support\debug alphakd Pool Monitor poolmon Windows NT CD \support\debug Global Flags gflags Windows NT Resource Kit Open Handles oh Windows NT Resource Kit QuickSlice qslice Windows NT Resource Kit Process Viewer pviewer, Windows NT Resource Kit pview Platform SDK, VC++ Process Exploder pview Windows NT Resource Kit 4.0 Process Status pstat Windows NT Resource Kit Pmon pmon Windows NT Resource Kit Object Viewer WinObj Platform SDK Process Walker PWalk Platform SDK Page Fault Monitor PFMon Platform SDK Spy++ Visual C++

Windows NT Resource Kits Full “Windows NT 5.0 Resource Kit” 250+ utilities Combines what was in the 4.0 Server and Workstation resource kits Subset “Windows NT 5.0 Resource Kit Support Tools” 50 utilities Ships in \support\reskit on Windows NT CD

www.sysinternals.com Windows NT internals articles and tools Some generated using reverse engineering (e.g., no source access) Some examples: winobj - view object manager namespace and objects nthandlex - show open handles by process ntfilmon - log all file I/O operations ntregmon - log all registry accesses cpufrob - change thread quantum Caveat: Most include a device driver, hence you’re added “trusted code” No warranty on using these on your system!

GFLAGS (Global Flags) Changes system-wide or image-wide debugging flags Poolmon requires “enable pool taggin” Oh (open handles) requires “maintain a list of objects for each type”

Windows NT Kernel Debugger (1 Of 4) Two versions: Command line: I386KD.EXE, ALPHAKD, etc., shipped with Windows NT In NTcdrom:\support\debug\i386, … \debug\alpha, etc. Select directory to match host system (where you will run the debugger executable); select executable to match target system (system being debugged) Also need many DLLs from this directory Also need symbol files from NTcdrom:\support\debug\targetarch\symbols\ … Extended via WinDbg shipped with Platform SDK (part of MSDN Professional) Provides GUI, fully-symbolic, source-level debugging Needs same DLLs and symbol files 1-17 picked up your slide

Windows NT Kernel Debugger (2 Of 4) Documentation: Windows NT Workstation Resource Guide (see “Windows NT Debugger”) Windows NT Device Driver Kit (DDK) See i386kd -? Help within debugger: commands “?” and “!?” and “!help”

Windows NT Kernel Debugger (3 Of 4) Two modes of operation: Open a crash dump file: C:\> set _NT_SYMBOL_PATH= ntcdrom:\support\debug\i386\symbols C:\> i386kd -Z dumpfilename Connect to a live system via null modem cable (must boot target system with /DEBUG/DEBUGPORT=COMn in boot.ini) C:\> set _NT_SYMBOL_PATH=ntcdrom:\support\debug\i386\symbols C:\> set _NT_DEBUG_PORT=COMn default COM1 C:\> set _NT_DEBUG_BAUD_RATE=nnnnn default 19200 C:\> i386kd serial “null modem” cable (for debugger) host target

Windows NT Kernel Debuggers (4 Of 4) Third-party product: SoftICE for Windows NT (NuMega) Runs on same system - e.g., doesn’t require second system for live debugging x86 only See www.numega.com

Agenda Introduction Tools System Architecture Processes and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processes and Threads Memory Management

Kernel Mode Versus User Mode A processor state Controls access to memory Each memory page is tagged to show the required mode for reading and for writing Protects the system from the users Protects the user (process) from themselves System is not protected from system Code regions are tagged “no write in any mode” Controls ability to execute privileged instructions A Windows NT abstraction Intel: Ring 0, Ring 3 PerfMon, Processor: “Privileged Time” and “User Time” Associated with threads Threads can change from user to kernel and back Part of saved context, along with registers, etc. Does not affect scheduling Components Access mode Applications User Subsystem processes User Executive Kernel Kernel Kernel Drivers Kernel HAL Kernel

Getting Into Kernel Mode Code is run in kernel mode for one of three reasons: 1. Requests from user mode Via the system service dispatch mechanism Kernel-mode code runs in the context of the requesting thread 2. Interrupts from external devices Windows NT-supplied interrupt dispatcher invokes the interrupt service routine ISR runs in the context of the interrupted thread (so-called “arbitrary thread context”) ISR often requests the execution of a “DPC routine,” which also runs in kernel mode Time not charged to interrupted thread 3. Dedicated kernel-mode system threads Some threads in the system stay in kernel mode at all times (mostly in the “System” process) Scheduled, preempted, etc., like any other threads 13d: new 13g-1: rearranged major bullets

Interrupt Dispatching user or kernel mode code kernel mode Note, no thread or process context switch! Interrupt dispatch routine interrupt ! Disable interrupts Record machine state (trap frame) to allow resume Mask equal- and lower-IRQL interrupts Find and call appropriate ISR Dismiss interrupt Restore machine state (including mode and enabled interrupts) Interrupt service routine Tell the device to stop interrupting Interrogate device state, start next operation on device, etc. Request a DPC Return to caller

Interrupt Precedence Via IRQLs IRQL = Interrupt Request Level The “precedence” of the interrupt with respect to other interrupts Different interrupt sources have different IRQLs Not the same as IRQ IRQL is also a state of the processor Servicing an interrupt raises processor IRQL to that interrupt’s IRQL This masks subsequent interrupts at equal and lower IRQLs User mode is limited to IRQL 0 31 Low APC Dispatch/DPC Device 1 . Device n Clock Interprocessor Interrupt Power fail High 30 29 28 Hardware interrupts Deferrable software interrupts 2 1 normal thread execution

Interprocessor Interrupt Alpha IRQLs IRQL on Alpha implemented in PAL code 7 High 6 Interprocessor Interrupt 5 Clock 4 Device High 3 Device 2 Dispatch/DPC 1 APC Low

DPCs (Deferred Procedure Calls) A list of “work requests” One queue per processor (but processors can run each others’ DPCs) Implicitly ordered by time of request (FIFO) Used to defer processing from higher (device) interrupt level to a lower (dispatch) level Used heavily for driver “after interrupt” functions Used for quantum end and timer expiration queue head DPC object DPC object DPC object DfrdCtx SysArg1 SysArg2 XydriverDpcRtn(DpcObj, DfrdCtx,SysArg1,SysArg2) { // ... }

Accounting For Kernel-Mode Time “Processor Time” = total busy time of processor (equal to elapsed real time - idle time) “Processor Time” = “User Time” + “Privileged Time” “Privileged Time” = time spent in kernel mode “Privileged Time” includes: Interrupt Time DPC Time Again note: interrupts and DPCs are not charged to any process or thread Screen snapshot from: Programs | Administrative Tools | Performance Monitor click on “+” button, or select Edit | Add to chart…

Agenda Introduction Tools System Architecture Processes and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processes and Threads Memory Management

Windows NT Executive Upper layers of operating system Provides “generic OS” services Processes, threads, memory management, I/O, interprocess communication, synchronization, security Almost completely portable C code Exports functions (“services”) which may be invoked via user-mode APIs Interface is NTDLL.DLL E.g., Win32 ReadFile -> executive NtReadFile Most interfaces to executive services not documented Used by subsystem writers TBD: demo/dump/trace of SS call

Windows NT Kernel Abstracts differences between processor architectures x86 vs. Alpha vs., etc. Main services Thread scheduling and context switching Generic wait operations Exception and interrupt dispatching Operating system synchronization primitives (MP and UP) Not a classic “microkernel” shares address space withrest of kernel-mode components Machine Independent C Assembler Machine Dep. C

HAL - Hardware Abstraction Layer A separate loaded binary (c:\winnt\system32\hal.dll) Several different versions for different motherboards, UP vs. MP, etc. Installation procedure selects appropriate HAL for platform and copies to Hal.Dll on system disk Purpose: Isolate (abstract) Kernel and Executive from platform-specific details Present uniform model for ease of driver development HAL abstracts: I/O system specifics (bus interfaces, DMA…) System timers, Cache coherency and flushing SMP support, Hardware interrupt priorities OEM Development Kit needed to buildHALs HAL contains some Executive and Kernel subroutines Sample HAL routines: HalGetBusData HalGetBusDataByOffset HalAssignSlotResources HalSetBusData HalSetBusDataByOffset HalTranslateBusAddress HalGetInterruptVector HalGetAdapter READ_REGISTER_ULONG WRITE_PORT_UCHAR

Kernel-Mode Device Drivers Separate loadable modules (drivername.SYS) Linked like .EXEs Linked against NTOSKRNL.EXE and HAL.DLL Only way to add “kernel extensions” or to access kernel mode system routines Defined in registry Same area as Win32 services (t.b.d.) Differentiated by Type value View loaded drivers with pstat.exe, drivers.exe Several types: “Ordinary” hardware drivers File system NDIS miniport, SCSI miniport (linked against port drivers) Win32K.Sys - Windowing system

WDM (Win32 Driver Model) Extension to Windows NT driver model to support for Plug and Play and Power Management Allows source/(x86) binary-compatible drivers across Windows 98 and Windows NT 5.0 Non trivial additions to existing drivers: 3 new major IRP types 36 new minor IRPs added 6 new miniport driver types Supporting WDM affects every area of a driver

WDM Drivers What’s covered in WDM: Not covered by WDM: IEEE 1394 (Firewire) Universal Serial Bus (USB) Audio: Speakers, microphone, CODEC Human Interface Devices: mouse, keyboard, monitor controls, game devices Still Imaging: Cameras, scanners Video Devices: Video capture, DVD Advanced Power and Configuration Interface (ACPI) BIOS support Not covered by WDM: Network Storage File System Video

Agenda Introduction Tools System Architecture Processes and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processes and Threads Memory Management

Object management / Executive RTL Hardware Abstraction Layer (HAL) NTOSKRNL.EXE Environment Subsystems Services System Processes Applications POSIX Replicator Service Controller Alerter OS/2 WinLogon User Application RPC User Mode Session Manager Event Logger Win32 Subsystem DLLs System Threads NTDLL.DLL NtosKrnl.Exe Kernel Mode Executive API I/O Manager Cache Manager Processes & Threads Security Virtual Memory Win32 User,GDI File systems Object management / Executive RTL Device drivers Kernel Hardware Abstraction Layer (HAL) Hardware interfaces (buses, I/O, interrupts, timers, clocks, DMA, cache control, etc.) Copyright by Microsoft Corporation. Used by permission.

NTOSKRNL.EXE NTOSKRNL.EXE HAL.DLL BOOTVID.DLL 13d: new slide Windows NT executive and kernel HAL.DLL Hardware Abstraction Layer - interface to hardware platform BOOTVID.DLL Boot video driver 13d: new slide TBD: 13f: I don’t like it here; it’s too early.

Naming Convention For Internal Windows NT Routines Two- or three-letter component code in beginning of function name Ex - General executive routine Ob - Object management Exp - Executive private (not exported) Io - I/O subsystem Cc - Cache manager Se - Security Mm - Memory management Ps - Process structure Rtl - Run-Time Library Lsa - Security Authentication FsRtl - File System Run-Time Lib Zw - File access, etc. Ke - Kernel Ki - Kernel internal (not available outside the kernel) Hal - Hardware Abstraction Layer READ_, WRITE_ - I/O port and register access Executive Kernel HAL

Multiprocessor Support Code comprising NTOSKRNL compiled twice: Once for uniprocessor, once for multiprocessor Avoids penalizing uniprocessor systems for added MP complexity Two files on Windows NT media: UP version: NTOSKRNL.EXE MP version: NTKRNLMP.EXE Selected at installation time, but copied to NTOSKRNL All drivers, DLLs, EXEs are built to run on on MP Upgrading from Uniprocessor vs Multiprocessor See uptomp.exe (in Resource Kit) 2 files replaced with different code NTKRNLMP.EXE replaces NTOSKRNL.EXE new HAL replaces HAL.DLL 4 files replaced with same code, but modified image header KERNEL32.DLL, NTDLL.DLL, WINSRV.DLL, WIN32K.SYS

Identifying Your NTOSKRNL Build numbers Incremented each time Windows NT is built from sources (i.e., different for beta releases) Service packs Replaces .EXEs (including usually NTOSKRNL), .DLLs, etc. Do not change Windows NT build number Free versus Checked build Free = retail version; Checked = debug version Used primarily in driver testing Build number is the same Recompilation of system with DEBUG flag true Therefore a different NTOSKRNL.EXE Note: MP only (NTOSKRNL and NTKRNLMP.EXE identical) Screen snapshot from: Programs | Administrative Tools | Windows NT Diagnostics

Workstation Vs Server Core operating system executables are identical NTOSKRNL.EXE, HAL.DLL, xxxDRIVER.SYS, etc., (t.b.d.) Windows NT Server a superset of Workstation domains, host-based RAID 5, NetWare gateway, DHCP server, WINS, DNS, full Internet Information Server… Enterprise Server adds yet more functionality (Clusters, 3GB address space) Terminal Server enables multi-user thin client support MP limits: Workstation: 2 CPUs, Server: 4 CPUs, Server Enterprise: 8 CPUs 15a: new slide. This shouldn’t take any extra time

Workstation Vs Server Registry indicates system type HKLM\CurrentControlSet\Control\ProductOptions ProductType: WinNT=Workstation, ServerNT=Server not a domain controller, LanManNT=Server that is a Domain Controller ProductSuite: Indicates Enterprise Edition, Terminal Server… Code in the operating system tests these values and behaves slightly differently in a few places Licensing limits (number of processors, number of inbound network connections, etc.) Boot-time calculations (memory manager) Default length of time slice See DDK: MmIsThisAnNtasSystem

Agenda Introduction Tools System Architecture Processe and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processe and Threads Memory Management

System Threads Internal worker routines that need thread context Drivers or Executive can create system threads Always run in kernel mode Usually associated with the “System” process by default But can be tied to any process Not non-preemptible (unless they raise IRQL to 2 or above) Kernel mode APIs: PsCreateSystemThread PsTerminateSystemThread KeSetBasePriorityThread KeSetPriorityThread

Threads In The “System” Process Note CPU time is 100% kernel mode “Start address” is address of thread function On Intel (at least): Addresses 8xxxxxxx will correspond to symbols in NtosKrnl.Exe Addresses Axxxxxxx are routines in Win32K.Sys Addresses Fxxxxxxx are routines in loaded device drivers Screen snapshot from: Programs | Resource Kit | Diagnostics | Process Viewer select “System” process

Threads In The “System” Process Memory Management Modified Page Writer for mapped files Modified Page Writer for paging files Balance Set Manager Swapper (kernel stack, working sets) Zero page thread (thread 0, priority 0) Security Reference Monitor Command Server Thread Network Redirector and Server Worker Threads Threads created by drivers for their exclusive use Examples: Floppy driver, parallel port driver Pool of Executive Worker Threads Used by drivers, file systems… Accessed via ExQueueWorkItem TBD - add list of all thread entry points from a sample system LAB: find system process with pview; note thread addresses; identify a few by mapping to symbols LAB: Perfmon, select thread object, priv time, all threads in system process 1-17: Added zero page thread 1-17f: Promoted driver threads to top-level bullet (was formerly under Network)

Threads In System Process (Observed on Intel Windows NT Workstation 4 Routine Name Priority Notes Phase1Initialization First thread in life of system; becomes zero page thread ExpWorkerThread 9-16 Pool of worker threads MiDereferenceSegmentThread 18 Dereferences segments; also expands paging file MiModifiedPageWriter 17 Writes modifed pages to paging file KeBalanceSetManager 16 Reclaims memory from processes, with aid of . . . KeSwapProcessOrStack 23 Scheduled by balance set manager FsRtlWorkerThread 16, 17 Dedicated worker threads for FSDs SepRmCommandServerThread 15 Security Reference Monitor Command Server MiMappedPageWriter Writes modified pages to mapped files (Win32 threads) routines in Win32K.Sys (0xA0000000) (driver threads) various routines in *driver.Sys (0xF0000000)

Agenda Introduction Tools System Architecture Processes and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processes and Threads Memory Management

Environment Subsystems Expose “native API” “Wrap” and extend Windows NT native functionality Interfaces to write subsystems not documented Two main components Subsystem DLLs - convert documented API to native API Environment Subsystem Process - maintain state of client processes; implement some subsystem APIs Three provided with Windows NT: Win32 Posix Bare minimum Posix standards, no optional components OS/2 Support for 1.x character-mode applications only drv 15a: new slide 13g-1: minor changes

Subsystem Extensions OS/2 Posix Microsoft sells an add-on to the OS/2 subsystem Supports 1.x Presentation Manager Posix OpenNT from SoftWay More-featured replacement for Posix subsystem www.opennt.com

Environment Subsystems Subsystem for each .exe specified in image header See winnt.h See Explorer / QuickView (right-click on .exe or .dll file) Or \reskit\exetype image.exe IMAGE_SUBSYSTEM_UNKNOWN 0 // Unknown subsystem IMAGE_SUBSYSTEM_NATIVE 1 // Image doesn't require a subsystem IMAGE_SUBSYSTEM_WINDOWS_GUI 2 // Win32 subsystem (graphical app) IMAGE_SUBSYSTEM_WINDOWS_CUI 3 // Win32 subsystem (character cell) IMAGE_SUBSYSTEM_OS2_CUI 5 // OS/2 subsystem IMAGE_SUBSYSTEM_POSIX_CUI 7 // Posix subsystem

Showing .exe Type With QuickView In Explorer: Right-click on an executable file or .DLL “Context menu” appears Select Quick View

Environment Subsystems Loading Subsystems to load specified in registry: \SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems Values: Required - list of value names for subsystems to load at boot time Optional - list of value names for subsystems to load when needed Windows - value giving filespec of Win32 subsystem (csrss.exe) csrss.exe Win32 APIs required (Client Server Runtime SubSystem) os2ss.exe OS/2 APIs optional psxss.exe Posix APIs optional Kmode - value giving filespec of Win32K.Sys (kernel-mode component of Win32) Some Win32 API DLLs are in “known DLLs” registry entry: \SYSTEM\CurrenctControlSet\Control\Session Manager\KnownDLLs LAB: regedt32: examine above; identify process name for Win32 subsys 1-17F: Added info on exetypes for subsystems.

Environment Subsystems Components Subsystem process For Win32: CSRSS.EXE API DLLs For Win32: Kernel32.DLL, Gdi32.DLL, User32.DLL, etc. Kernel-mode extension to executive Win32 only: Win32K.SYS Environment Subsystems System and Server Processes User Application OS/2 POSIX Win32 Subsystem DLL User Mode NTDLL.DLL Kernel Mode Executive Win32 User/GDI Device Drivers Kernel Hardware Abstraction Layer (HAL)

Windows NT Simplified Architecture (3.51 and earlier) Environment Subsystems System and Server Processes User Application OS/2 Win32 POSIX User Mode Subsystem DLL 1 2 NTDLL.DLL Kernel Mode Executive LPC Device Drivers Kernel Hardware Abstraction Layer (HAL) Most Win32 Kernel APIs All other Win32 APIs, including User and GDI APIs 1 2

Windows NT Simplified Architecture (4.0 and later) Environment Subsystems System and Server Processes User Application OS/2 POSIX Win32 User Mode Subsystem DLL 1 2 3 NTDLL.DLL Kernel Mode Win32 User/GDI Executive LPC Device Drivers Kernel Hardware Abstraction Layer (HAL) Most Win32 Kernel APIs Most Win32 User and GDI APIs A few Win32 APIs 1 2 3

(Reduced) Role Of Win32 Subsystem Process Process creation and deletion Thread creation and deletion Get temporary file name Drive letters Security checks for file system redirector Window management for console (character cell) applications Some support for 16-bit DOS support (NTVDM.EXE) 1-17: and also “NLS local stuff”. What is NLS? Ans National Locale Services

Agenda Introduction Tools System Architecture Processes and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processes and Threads Memory Management

Invoking System Functions From User Mode Kernel-mode functions (“services”) are invoked from user mode via a protected mechanism x86: INT 2E; Alpha: SYSCALL (PALcode) I.e., on a call to an OS service from user mode, the last thing that happens in user mode is this “change mode to kernel” instruction Causes an interrupt, handled by the system service dispatcher (KiSystemService) in kernel mode Return to user mode is done by dismissing the interrupt or exception The desired system function is selected by the “system service number” Every Windows NT function exported to user mode has a unique number Push this number on the stack just before the “change mode” instruction (after pushing the arguments to the service) This number is an index into the system service dispatch table Table gives kernel-mode entry point address and argument list length for each exported function LAB: do commands 1-17: moved the ntoskrnl.exe line here from next slide, makes a good lead-in TODO: repro output from these commands in handout 13d: included Alpha SYSCALL 13g: Moved this here. Transition from previous sectiion: “Now the question comes up, how do we get from user to kernel mode? In other words, how does the system allow applications running in user mode, to request functions which must be performed in kernel mode, without giving away the system’s security?” 13g-1: minor changes 15a: several more details added here and subsequent slides. This is one of those “we say this stuff anyway, so why not have it in the slides” poitns. Or at least I say it… 15a: note, this is a “bottom-up” approach. We talked about the exec kernel and HAL first. Then we describe the INT 2e. then we talk about what “wraps” the INT 2e.

Invoking System Functions From User Mode All validity checks are done after the user to kernel transition KiSystemService probes argument list, copies it to kernel-mode stack, and calls the executive or kernel routine pointed to by the table Service-specific routine checks argument values, probes pointed-to buffers, etc. Once past that point, everything is “trusted” This is safe, because: The system service table is in kernel-protected memory; and The kernel mode routines pointed to by the system service table are in kernel-protected memory; therefore: User mode code can’t supply the code to be run in kernel mode; it can only select from among a predefined list Arguments are copied to the kernel mode stack before validation; therefore: Other threads in the process can’t corrupt the arguments “out from under” the service

NTDLL.DLL PUSH of service # and INT 2E are “wrapped” by small “jacket” procedures in NTDLL.DLL These user-mode routines have the same function names and arguments as the kernel mode routines they call E.g., NtWriteFile in NtDll.Dll invokes NtWriteFile in NtosKrnl.Exe Therefore exports of NTDLL are the “NT native API” Entry points in NtDll.Dll are not supported or documented for use from user mode apps A few are documented in the DDK for call from kernel mode A few images that come with Windows NT are written to the “native API” exposed by NtDll.Dll (“Windows NT native images”) See article on www.sysinternals.com NTDLL also contains image loader and other support functions What about getting to USER and GDI functions in Win32K.SYS? System service wrapper exists in USER32.DLL, GDI32.DLL Does not go through NTDLL.DLL

Tracing An Example Win32 Call call WriteFile(…) Win32 application WriteFile in Kernel32.Dll call NtWriteFile return to caller Win32- specific NtWriteFile in NtDll.Dll Int 2E return to caller used by all subsystems U K software interrupt call NtWriteFile dismiss interrupt KiSystemService in NtosKrnl.Exe Source: MSJ, August 1996, page 21 (by Matt Pietrek) NtWriteFile in NtosKrnl.Exe do the operation return to caller

Tracing An Example Win32 Call Depends.Exe in Resource Kit and Platform SDK Allows viewing of image->DLL relationships, imports, and exports

Examining Symbols In Key Images Examine imports and exports of an .EXE down to the OS In Explorer, right mouse click on EXE or DLL, then “quick view” (built in) or “View Dependencies” (Dependency Walker tool in ResKit and Platform SDK) Or use LINK /DUMP /EXPORTS, /IMPORTS 1. Look at imports of \winnt\system32\notepad.exe 2. Look at exports and imports of kernel32.dll Most of the exports are documented Win32 calls 3. Look at exports and imports of ntdll.dll None of the exports are documented Some are the same as exports from ntoskrnl.exe, documented in DDK, with identical 13h: added reference to Dependency Walker LAB: do the link/dump/exports command (V4: right-click, quick view on ntoskrnlexe does this) Link /dump /imports advapi32.dll LAB: set up to write a dump file (cpnl) also add a boot.ini line for /maxmem=8 boot with 8MB crash NT (tlist, kill <pid> for csrss, system crashes set _NT_SYMBOL_PATH to where symbols are i386kd -Z <dumpfilespec> .logopen filename x * LABEND ntdll.dll - exposes executive routines to e.g. native subsystems and Win32 DLLs (its entry points run in user mode) TBD: Clean this up, add some quickview screen snapshots

Examining Symbols In Key Images 4. Look at exports and imports of ntoskrnl.exe About 1000 total exported symbols About 300 of the exported routine names are documented in DDK Callable only from kernel mode 5. Look at all global symbols in ntoskrnl.exe Defined in \support\symbols\xxx\debug\exe\ntoskrnl.dbg Quick viewer won’t display - use Kernel Debugger “x *” with just this .dbg file loaded About 4000 total symbols (Includes executive data cells in addition to routines) Exports of ntoskrnl.exe are a subset of this list

Agenda Introduction Tools System Architecture Processes and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processes and Threads Memory Management

Process-Based Windows NT Code Pieces of Windows NT that run in separate executables (.exe’s), in separate processes Started by system Not tied to a user logon Have full process context Three types: Environment Subsystems (already described) Win32 Services System startup processes Note: “system startup processes” is not an official MS-defined name LAB; Tlist/t - process “origin” hierarchy 13b: Moved tlist output to a later slide 13d: changed “system processes” to “system startup processes”

Process Creation Hierarchy tlist.exe (from resource kit) tlist /t shows creation hierarchy Creating process can exit, leaving created process running - hence this display does not show all creators Explorer.exe is actually started by userinit.exe, which then exits

Process-Based Windows NT Code Win32 services Win32 .EXEs (applications) that run independently of a logged on user Start at boot or logon time, survive logoff Defined by CreateService API - view through Control Panel See srvany.exe, sc.exe, srvinstw.exe, instsrv.exe in Resource Kit Typically do not interact with the desktop Get startup configuration parameters from Registry Log errors to Windows NT Event Log Use some form of IPC mechanism for client communication and control Services will likely make use of Windows NT security impersonation Remotely manageable (start, stop, user-defined codes) Server Manager allows remote control of services Code is the same to control services locally vs. remotely Examples of built-in Windows NT Services Schedule service (at command), Event Log, Remote Access Server, etc. 16e: new slides

Life Of A Service Install time Management / maintenance Registry Setup application tells Service Controller about the service System boot / initialization SCM reads registry, starts services as directed Management / maintenance Control panel can start and stop services and change startup parameters Service Controller Registry Setup Application CreateService Service Processes Control Panel

Where Are Services Defined? Maintained in Windows NT Registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services One key per installed service Mandatory information kept on each service: Type of service (Win32, Driver…) Imagename of service .EXE NOTE: Some service .EXEs contain more than one service Start type (automatic, manual, or disabled) Optional information: Display Name Dependencies Account and password to run under Can store application-specific configuration parameters “Parameters” under service key 16e: new slides LAB: identify a few running services (which process they are in); - find both a Win32 “own” and Win32 “share” service - find services inside services.exe process (dump registry services key into a file; search for services.exe) 1-17f: Moved

Process-Based Windows NT Code System startup processes Separate processes loaded or started at boot time (not as services or environment subsystems) Names of images are not in registry “Hardwired” in the source code Most are Win32 executables, one (smss) is a “native image” (Idle) Process id 0 Part of the loaded system image Home for idle thread(s) (not a real process nor real threads) Called “System Process” in many displays (System) Process id 2 Part of the loaded system image Home for kernel-defined threads (not a real process) Thread 0 (routine name Phase1Initialization) launches the first “real” process, running smss.exe… …and then becomes the zero page thread 1-17F see previous slide notes. Did this so we can meaningfully describe “system processes” by exclusion. Also, removed references to “native images” (that’s been moved to the environment subsystems section). 13d; More details, split into a second slide

Process-Based Windows NT Code System startup processes smss.exe Session Manager The first “created” process Takes parameters from \Registry\Machine\System\CurrentControlSet\Control\Session Manager Launches required subsystems (csrss) and winlogon winlogon.exe Logon process Presents first login prompt Presents “enter username and password” dialog Launches services.exe, lsass.exe, and nddeagnt.exe When someone logs in, launches userinit.exe services.exe Service Controller; also, home for many NT-supplied services Starts processes for services not part of services.exe (driven by \Registry\Machine\System\CurrentControlSet\Services ) lsass.exe Local Security Authentication Server userinit.exe Started after logon; starts desktop (Explorer.Exe) and exits (hence does not show up in tlist output; Explorer appears to be an orphan) explorer.exe and its children are the creators of all interactive apps

Agenda Introduction Tools System Architecture Processes and Threads Kernel Mode Environment Executive, Kernel, HAL, Drivers Product Packaging System Threads Environment Subsystems System Service Dispatching Process-based Windows NT code Summary Processes and Threads Memory Management

Four Contexts For Executing Code Full process and thread context: User applications Win32 Services Environment subsystem processes System startup processes Have thread context but no “real” process: Threads in “System” process Routines called by other threads / processes: Subsystem DLLs Executive system services (NtReadFile, etc.) GDI routines in Win32K.Sys (and graphics drivers) No process or thread context (“Arbitrary thread context”) Interrupt dispatching Device drivers

Where Is The Code? TBD: What is ntdll.dll? 1-17g new slide Kernel32.Dll, Gdi32.Dll, User32.Dll Export Win32 entry points NtDll.Dll Provides user-mode access to system-space routines Also contains heap manager, image loader, thread startup routine Ntoskrnl.Exe (or Ntkrnlmp.exe) Executive and kernel Includes most routines that run as threads in “system” process Win32K.Sys The loadable module that includes the now-kernel-mode Win32 code (formerly in csrss.exe) Hal.Dll Hardware Abstraction Library drivername.Sys Loadable kernel drivers TBD: What is ntdll.dll? 1-17g new slide 13a: added second bullet on contents of NTDLL.DLL

Agenda Introduction Tools System Architecture Processes and Threads Memory Management

Systemwide Address Space Processes And Threads Per-process address space Thread What is a process? Represents an instance of a running program You create a process to run a program Starting an application creates a process Primary argument to CreateProcess is image file name (or command line) What is a thread? An execution context within a process Primary argument to CreateThread is a function entry point address All threads in a process share the same per-process address space Every process starts with one thread Running the program’s “main” function Can create other threads in the same process Can create additional processes Thread Thread Systemwide Address Space

Tools To Examine Processes Task Manager Performance Monitor pviewer.exe (pview in Platform SDK): shows processes, threads within processes, memory details pview.exe (process explode): thread and process ACLs and tokens tlist.exe - tlist /t shows parent/child relationships QuickSlice qlice.exe CPU usage by process, and by thread within each process Pulist - process user list Vadump - dump virtual address space of a process 1-17F: new slide

Tools To Examine Processes Page fault monitor (pfmon.exe) Shows page fault type and origin of subject application Can provide data to working set tuner (part of Platform SDK) Pstat pstat.exe (char mode, no icon) One-time snapshot of system Shows state of threads within all processes, with wait reasons Kernel debugger Shows various internal structures See Windows NT® Workstation Resource Kit documentation oh.exe (ResKit), nthandleex (www.sysinternals.com) - show open handles Ntpmon (www.sysinternals.com)

Windows NT 5.0 Job Object New kernel object to collect a group of related processes CreateJobObject/OpenJobObject System enforces job quotas and security context Limits: Total and current CPU time, total and active processes, per-process and per-job CPU time, min and max working set, CPU affinity, priority class Security limits: No administrators token, only restricted token, only specific token, filter token, no accessing windows outside the job, no reading/writing the clipboard To examine: See new performance counters + new !job command in kernel debugger

Processes And Threads Internal Structures Access token VAD VAD VAD Process object Virtual address space descriptors See kernel debugger commands: !processfields !threadfields !process !thread !tokenfields !token !handle !object Handle table object object TBD: Description, picture of VAD LAB: kdbg, !processfields !threadfields !process 0 Thread Thread Thread … Access token

!processfields Pcb: 0x0 ForkInProgress: 0x158 ExitStatus: 0x68 LockEvent: 0x6c LockCount: 0x7c CreateTime: 0x80 ExitTime: 0x88 LockOwner: 0x90 UniqueProcessId: 0x94 ActiveProcessLinks: 0x98 QuotaPeakPoolUsage[0]: 0xa0 QuotaPoolUsage[0]: 0xa8 PagefileUsage: 0xb0 CommitCharge: 0xb4 PeakPagefileUsage: 0xb8 PeakVirtualSize: 0xbc VirtualSize: 0xc0 Vm: 0xc8 LastProtoPteFault: 0xf8 DebugPort: 0xfc ExceptionPort: 0x100 ObjectTable: 0x104 Token: 0x108 WorkingSetLock: 0x10c WorkingSetPage: 0x12c ProcessOutswapEnabled: 0x130 ProcessOutswapped: 0x131 AddressSpaceInitialized: 0x132 AddressSpaceDeleted: 0x133 AddressCreationLock: 0x134 ForkInProgress: 0x158 VmOperation: 0x15c VmOperationEvent: 0x160 PageDirectoryPte: 0x164 LastFaultCount: 0x168 VadRoot: 0x170 VadHint: 0x174 CloneRoot: 0x178 NumberOfPrivatePages: 0x17c NumberOfLockedPages: 0x180 ForkWasSuccessful: 0x184 ExitProcessCalled: 0x186 CreateProcessReported: 0x187 SectionHandle: 0x188 Peb: 0x18c SectionBaseAddress: 0x190 QuotaBlock: 0x194 LastThreadExitStatus: 0x198 WorkingSetWatch: 0x19c LpcPort: 0x1a0 InheritedFromUniqueProcessId: 0x1a4 GrantedAccess: 0x1a8 DefaultHardErrorProcessing 0x1ac LdtInformation: 0x1b0 VadFreeHint: 0x1b4 VdmObjects: 0x1b8 ProcessMutant: 0x1bc ImageFileName[0]: 0x1dc VmTrimFaultValue: 0x1ec

!threadfields Tcb: 0x0 CreateTime: 0x1b0 ExitTime: 0x1b8 ExitStatus: 0x1c0 PostBlockList: 0x1c4 TerminationPortList: 0x1cc ActiveTimerListLock: 0x1d4 ActiveTimerListHead: 0x1d8 Cid: 0x1e0 LpcReplySemaphore: 0x1e8 LpcReplyMessage: 0x1fc LpcReplyMessageId: 0x200 Client: 0x208 IrpList: 0x20c TopLevelIrp: 0x214 ReadClusterSize: 0x21c ForwardClusterOnly: 0x220 DisablePageFaultClustering: 0x221 DeadThread: 0x222 HasTerminated: 0x223 EventPair: 0x224 GrantedAccess: 0x228 ThreadsProcess: 0x22c StartAddress: 0x230 Win32StartAddress: 0x234 LpcExitThreadCalled: 0x238 HardErrorsAreDisabled: 0x239 1-17: someone is sure to ask why there’s no token field here when there is a token field in the processfields output. TBD: Screen dumps of !process, !thread, !handle (!handle maybe not here) 1-17f: If we want to keep the listings as part of the handout, we can drop these two slides.

Looking At Waiting Threads 13d: new slide pstat.exe (Resource Kit) Shows state of every thread in every process But for threads that are waiting, that’s all we know…

Looking At Waiting Threads !thread command in kernel debugger shows what a thread is waiting on 13d: new slide

Object-type-specific data Dispatcher Objects Any kernel object you can wait for is a “dispatcher object” Some exclusively for synchronization E.g., events, mutexes (“mutants”), semaphores, queues, timers Others can be waited for as a side effect of their prime function E.g., processes, threads, file objects Non-waitable kernel objects are called “control objects” All dispatcher objects have a common header All dispatcher objects are in one of two states “Signalled” versus “nonsignalled” When signalled, a wait on the object is satisfied Different object types differ in terms of what changes their state Wait and unwait implementation is common to all types of dispatcher objects Dispatcher object 12B: new Size Type State Wait listhead Object-type-specific data (see \ddk\inc\nttddk.h)

Object-type-specific data Object-type-specific data Wait Blocks Thread Objects WaitBlockList Key Type Next link List entry Object Thread Key Type Next link List entry Object Thread Key Type Next link List entry Object Thread Represent a thread’s reference to something it’s waiting for (one per handle passed to WaitFor…) All wait blocks from a given wait call are chained to the waiting thread Type indicates wait for “any” or “all” Key denotes argument list position for WaitForMultipleObjects WaitBlockList Dispatcher Objects Wait blocks Size Type State Wait listhead Object-type-specific data 1-17: ta-da! (also see next slide)] 13d: I know we talked about hiding this - but I don’t see how we can not talk about this stuff in an “internals” class. You did fine on it in Seattle… Size Type State Wait listhead Object-type-specific data

Agenda Introduction Tools System Architecture Processes and Threads Memory Management Virtual Address Space Layout Process Memory Usage Global System Cache System Memory Usage

4GB Virtual Address Space 2 GB per-process Address space of one process is not directly reachable from other processes 2 GB systemwide The operating system is loaded here, and appears in every process’s address space There is no process for “the operating system” (though there are processes that do things for the OS, more or less in “background”) 00000000 .EXE code Globals Per-thread user mode stacks Process heaps .DLL code Unique per process, accessible in user or kernel mode 7FFFFFFF 80000000 Per process, accessible only in kernel mode Exec, Kernel, HAL, drivers, per-thread kernel mode stacks, Win32K.Sys File system cache Paged pool Non-paged pool 13H: combined 2 separate address space slides into one; indicated page tables here, rather than waiting until MM section 15a: added description C0000000 Process page tables, hyperspace System wide, accessible only in kernel mode FFFFFFFF 7

System Space Layout x86 Alpha AXP 80000000 80000000 System code (NTOSKRNL, HAL, boot drivers); initial nonpaged pool System code (NTOSKRNL, HAL, boot drivers) and initial nonpaged pool A0000000 System Mapped Views (e.g. WIN32K.SYS) or session space (Terminal Server only) A4000000 Additional System PTEs (& big cache) C0000000 Process Page Tables and Page Directory C0000000 Process Page Tables and Page Directory C1000000 Hyperspace and process working set list C0400000 Hyperspace and process working set list C2000000 Unused No Access C0800000 Unused No Access C3000000 System Working Set List C0C00000 System Working Set List C4000000 System Cache C1000000 System Cache DE000000 System Mapped Views (e.g. WIN32K.SYS) E1000000 Paged Pool E1000000 Paged Pool EB000000 (min) System PTEs EB000000 (min) System PTEs Non-Paged Pool expansion Non-Paged Pool expansion FFBE0000 Crash dump information FFC00000 FDFEC000 Crash dump information & HAL usage HAL usage

3GB Process Space Option 00000000 Only available on x86 Server Enterprise Edition Boot with /3GB option in BOOT.INI Chief “loser” in system space is file system cache Expands per-process address space But image must be marked as “large address space aware” A stopgap while we wait for 64-bit Windows NT (Merced and Alpha; post-Windows NT 5.0) .EXE code Globals Per-thread user mode stacks .DLL code Process heaps Unique per process, accessible in user or kernel mode Unique per process (= per appl.), user mode Per process, accessible only in kernel mode 15a: added description BFFFFFFF C0000000 Process page tables, hyperspace System wide, accessible only in kernel mode Exec, kernel, HAL, drivers, etc. FFFFFFFF

64-bit Very Large Memory In Windows NT 5.0 00000000 00000000 2GB user space 2GB user space 00000000 7FFFFFFF 2GB process space Alpha Windows NT Server Enterprise Edition only Referenced by 64-bit pointers Cannot be paged out - must be resident at all times Cannot be used for code, only data file mapping New APIs: VirtualAllocVlm, MapViewOfFileVlm, Read/WriteFileVlm, Read/WriteProcessMemoryVlm, etc.) Yet another stopgap prior to 64-bit Windows NT 00000001 00000000 28GB Large Memory Area 13H: new slide 15a: cleaned up to look parallel to previous two slides TBD: these ranges appear disjoint - fix 00000007 FFFFFFFF 00000008 00000000 Invalid (inaccesible) (about 1.8x10^19 bytes; not to scale!) FFFFFFFF 7FFFFFFF FFFFFFFF 80000000 2GB system space FFFFFFFF FFFFFFFF

Application Startup Maps V.A.S. To Code On Disk 00000000 7FFFFFFF .exe .dll paging file See link/dump/header, or QuickView for .exe’s and .dll’s CreateFileMapping, MapViewOfFile simply make the mechanism available to application-level code All of these files may simultaneously be mapped by other processes

Process Virtual Address Layout LAB: Run process walker against something \ntreskit\perftool\meastool\vadump <pid> (can just attach to process) Screen snapshot from: Programs | SDK Tools | Process Walker Process | Load Process | notepad

Agenda Introduction Tools System Architecture Processes and Threads Memory Management Virtual Address Space Layout Process Memory Usage Global System Cache System Memory Usage

Process Memory Usage Working set: All the physical pages “owned” by a process Essentially, all the pages the process can reference without incurring a page fault Upper limit on size for each process When limit is reached, a page must be released for every page that’s brought in (“working set replacement”) Working set limit: The maximum pages the process can own Maximum is calculated as (available pages - 512 pages) Result stored in MmMaximumWorkingSetSize LAB: Perfmon - process working sets pmon - all working sets (text output) An interesting thing about the working set is that it starts out nearly empty. There is no part of the executable or DLLs that are brought into the working set automatically as part of image startup; the first page of code is brought in because of the page fault that happens when the StartOfThread function (described previously) calls the application’s main() function.

Working Set List A FIFO list for each process newer pages older pages PerfMon Process “WorkingSet”

Working Set Replacement PerfMon Process “WorkingSet” To standby or modified page list add diagram showing many processes’ working set pages scattered through physical memory When working set “count” = working set size, must give up pages to make room for new pages Page replacement is ”modified FIFO” MP x86 and Alpha: no regard to accessed bit Windows NT 5.0 on uniprocessor x86 takes into account age

Locking Pages Pages may be locked into the process working set Locked pages are guarenteed in physical memory (“resident”) when any thread in process is executing Win32: status = VirtualLock(baseAddress, size); status = VirtualUnlock(baseAddress, size); Number of lockable pages is a fraction of the maximum working set size Changed by SetProcessWorkingSetSize Pages can be locked into physical memory (by drivers only) Pages are then immune from outswapping as well as paging MmProbeAndLockPages 13b: moved here. “If you don’t like the page replacement strategy, you can remove some of your VAS from consideration for paging…”

Memory Management Information Task manager processes tab 2 4 1 3 “Mem Usage” = physical memory used by process (working set size, not working set limit) “VM Size” = private (not shared) committed virtual space in processes “Mem Usage” in status bar is total of “VM Size” column/maximum allowed - i.e., same as “commit charge” in “Performance” tab (see next slide) - not same as “Mem Usage” column here! 1 2 3 13b: new slide 13f: added new subsection with all of these “memory displays” at the end. Added new comment on bogus-named “mem usage” in status bar. Added callouts (consistent among these slides) to show relationships among displays and link displays to text. Oddly, “mem usage” in status bar is not total of “VM size” in processes list. 4 Screen snapshot from : Task Manager | Processes tab

Memory Management Information PerfMon - process object “Working Set” = working set size (not limit) “Private Bytes” = same as “VM Size” from Task Manager Processes list “Virtual Bytes” = committed virtual space, including shared pages 1 2 6 13b: new slide 13d: Is “virtual bytes” really just the committed virtual space? Or does it include reserved? 2 6 1 Screen snapshot from: Performance Monitor counters from Process object

Memory Management Information Task manager performance tab “Commit charge total” = total of private (not shared) committed virtual space in all processes (i.e. total of “VM Size” from processes display) “Commit charge limit” = sum of available physical memory + free space in paging file 3 4 3 4 13b: new slide 13d: added info on “file cache” 13d: fixed info on “file cache” to be physical vs. virtual Screen snapshot from: Task Manager | Performance tab

Agenda Introduction Tools System Architecture Processes and Threads Memory Management Virtual Address Space Layout Process Memory Usage Global System Cache System Memory Usage

File System Virtual Block Cache Shared by all file systems (local or remote) Caches all files Including file system metadata files Virtual block cache (not logical block) Managed in terms of blocks within files, not blocks within partition Uses standard Windows NT virtual memory mechanisms Coherency maintained between mapped files and read/write access Virtual size: 64-512mb (960MB if large cache size set) In system virtual address space, so visible to all Divided into 256kb “views”

Cached File Operations Open a file: Find an available view Map the first 256kb of the file into the view Read from or write to a cached file: Remap as necessary to map referenced section of file into the cache Copy data between application buffer and cache’s virtual address space Actual I/O is due to paging Process address space System address space File

Fast I/O Fast I/O path 13d: moved “fast I/O” info here Cache Manager I/O Subsystem API (Ntxxx) Fast I/O path I/O Manager (Ioxxx) Driver Support Routines (Io, Ex, Ke, Mm, Hal, FsRtl, ...) File System drivers (e.g. NTFS) Fast I/O path Allows executive I/O APIs to access cache directly Bypasses file system driver Bypasses IRP generation, probe-and-lock of user buffer, etc. 13d: moved “fast I/O” info here Disk device driver HAL I/O access routines I/O ports and registers

Cache Size Physical size: Depends on available memory Competes for physical memory with processes, paged pool, pageable system code Part of “system working set” Automatically expanded / shrunk by system Normal working set adjustment mechanisms Relies on Memory Manager for global memory policy Performance Monitor: Memory object | System cache resident bytes shows current physical space occupied by cache See \SYSTEM\CurrentControlSet\Control\Session Manager\ Memory Management\LargeSystemCache Default is 0 for both Workstation and Server 1 = favor system working set vs. process working set also allows cache to be >512MB virtual size Can modify with Control Panel->Network->Services-> Server properties 13d: separated this

Cache Functions And Control Automatic asynchronous readahead Done by separate “Readahead” system thread 64kb readaheads by default Predicts next read location based on history of last 3 reads Readahead hints can be provided to CreateFile: FILE_FLAG_SEQUENTIAL does 192kb read ahead FILE_FLAG_RANDOM_ACCESS disables read ahead Write-back, not write-through Dirty page threshold forces writing Small system: Physical Pages / 8; medium system: Physical Pages / 4 Large system: add above 2 together “Lazy writer” thread queues 1/4 of dirty pages every second to separate “Write Behind” system thread (note, does not flush mapped files) Can override via CreateFile with FILE_FLAG_WRITE_THROUGH Or explicitly call FlushFileBuffers when you care (does flush mapped files) 13d: added note on mapped files

Cache Functions And Control Can disable cache completely on a per-file basis CreateFile with FILE_FLAG_NO_BUFFERING Requires reads/writes to be done on sector boundaries Buffers must be aligned in memory on sector boundaries 13d: Moved this here

Agenda Introduction Tools System Architecture Processes and Threads Memory Management Virtual Address Space Layout Process Memory Usage Global System Cache System Memory Usage

System Paged Memory Just as processes have working sets, Windows NT’s pageable system-space code and data lives in the “system working set” Cache is one of 4 components of “system working set” Pageable components of system working set: Paged pool Pageable code and data in the exec Pageable code and data in kernel-mode drivers, Win32K.Sys, graphics drivers, etc. Global file system data cache To get physical (resident) size of these with PerfMon, look at: Memory | Pool Paged Resident Bytes Memory | System Code Resident Bytes Memory | System Driver Resident Bytes Memory | System Cache Resident Bytes Memory | Cache bytes counter is total of these four “resident” (physical) counters (not just the cache; same as “File Cache” on Task Manager / Performance tab TBD: Doesn’t the cache have its own working set? Can’t believe that it’s limited to 450 pages 13d: fixed to reflect that “cache bytes” is physical

Sessions New memory management object to support Windows NT® Server 5.0 All processes in an interactive session share a: Session-specific copy of Win32K.Sys Instance of Winlogon Session working set 80000000 System code (NTOSKRNL, HAL, boot drivers); initial nonpaged pool A0000000 Win32k.sys *8MB) A0800000 Session Working Set Lists x86 Mapped Views for Session Paged Pool for Session A0C00000 A2000000

System Nonpaged Memory Nonpageable components: Nonpageable parts of NtosKrnl.Exe, drivers Nonpaged pool (see PerfMon, Memory object: Pool nonpaged bytes) To get size of nonpageable system code, run \ntreskit\pstat.exe & add columns 1 & 2 non-paged code non-paged data pageable code+data output of “drivers” (\ntreskit\drivers.exe) is similar Win32K.Sys is paged, even though it shows up as nonpaged 7 9 8 7 TBD: maybe make this later? TBD: show how to get sizes TBD: add some stuff on perfmon paging counters - page faults vs. page read IOs vs. pages read, etc. 8 9

Monitoring Pool Usage Poolmon.exe in \support\debug Must first turn on pool tagging with gflags “p” to toggle between nonpaged, paged pool, or both Sorting: “b” to sort by total # of bytes “a” to sort by # of allocations “t” to sort by structure tag

“Free” Memory System keeps unassigned physical pages (those not part of any working set) on five lists Free page list Modified page list Standby page list Zero page list Bad page list - pages that failed memory test at system startup 13f: slight rearrangement of text in this section

Managing Physical Pages demand zero page faults pages read from disk zero page thread Standby Page List Free Page List Zero Page List “soft” page faults Bad Page List Process Working Sets working set replacement modified page writer LAB: Perfmon - available bytes TBD: change build order to match slides - or change slides Modified Page List

Memory Management Information Task manager performance tab 1 “Available” memory = total of free, zero, and standby lists (majority usually are standby pages) “File cache” is really total physical size of pageable portions of: paged pool, NtosKrnl.Exe code and data, drivers code and data, and file system cache (same as PerfMon “cache bytes” counter) “Kernel Memory Paged” is resident size of paged pool “Kernel Memory Nonpaged” is actual size of nonpaged pool 3 2 1 4 2 13b: new slide 13d: added info on “file cache” 13d: fixed info on “file cache” to be physical vs. virtual 3 4 Screen snapshot from: Task Manager | Performance tab

Summary: Accounting For Physical Memory Usage Process working sets Perfmon: Process / Working set Note, shared resident pages are counted the process working set of every process that’s faulted them in Hence, the total of all of these may be greater than physical memory Nonpageable system code (NTOSKRNL + drivers, including win32k.sys &graphics drivers) See total displayed by DRIVERS utility in Windows NT Resource Kit Nonpageable pool Perfmon: Memory / Pool nonpaged bytes Free, zero, and standby page lists Perfmon: Memory / Available bytes Or: Task Manager / Performance tab: Physical memory: Available Pageable, but currently-resident, system-space memory Perfmon: Memory / Pool paged resident bytes Perfmon: Memory / System cache resident bytes Perfmon: Memory / System code resident bytes Perfmon: Memory / System driver resident bytes Memory | Cache bytes counter is really total of these four “resident” (physical) counters Modified, Bad page lists can only see size with !memusage command in Kernel Debugger 1-17: new slide LAB: add up NT physical memory usage of system space: TBD: eventually add a “loadable mdules” slide showing pieces of the exec and kernel; move this last bullet there 13d: Reformatted, added info

Windows NT Internals Information Sources Books Inside Windows NT (Solomon, MS Press) Advanced Windows (Richter, MS Press) Windows NT Workstation Resource Guide (MS Press) MSDN Library Platform SDK API documentation Windows NT Device Driver Kit (DDK) documentation Win32 Knowledge Base - has some Windows NT internals articles Past Windows NT conferences audio/video tapes (www.mobiletape.com) www.sysinternals.com - Windows NT internals articles and tools www.microsoft.com/hwdev - hardware developers and driver writers www.microsoft.com/hwdev/ntifskit - Installable File System Developers Kit comp.os.ms-windows.programmer.nt.kernel-mode - drivers newsgroup www.cmkrnl.com - Windows NT device driver FAQ