Pushing the Limits of Windows Mark Russinovich Technical Fellow Microsoft Corporation Session Code: CLI402 *Portions derived from David Solomon’s Windows Internals Seminar

About Me Technical Fellow, Microsoft Co-founder and Chief Software Architect of Winternals Software Co-author of Windows Internals 4th and 5th Edition and Inside Windows 2000 3rd Edition with David Solomon Author of TechNet Sysinternals Home of blog and forums Contributing Editor TechNet Magazine, Windows IT Pro Magazine Ph.D. in Computer Engineering

Pushing the Limits Resource exhaustion is obviously not a good thing At the minimum it causes service outage At worst it can cause data loss or even corruption As a Windows systems administrator you need to know the limits For capacity planning For systems monitoring This session explains core Windows kernel limits, where they come from, and how to monitor them I’ll use Sysinternals Testlimit and Notmyfault to stress resources: http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

Outline Virtual Memory Physical Memory Paged and Nonpaged Pool Processes and Threads Handles

Address Space Limits One virtual memory limit is address space Process can run out of space to allocate virtual memory Typically only an issue for 32-bit processes Use Testlimit -r (reserve memory) to see address space exhaustion: Process Explorer Virtual Size shows amount of address space used

(boot.ini: /3GB or /USERVA) 32-bit x86 Address Space 32-bits = 2^32 = 4 GB 4GT Address Space (boot.ini: /3GB or /USERVA) (BCD: increasuserva) Default 2 GB User process space 3 GB User process space 2 GB System Space 1 GB System Space

64-bit Address Spaces 64-bits = 2^64 = 17,179,869,184 GB x64 IA-64 32-bit process on 64-bit Windows = 4 GB x64 (AMD64 & Intel 64) IA-64 8192 GB (8 TB) User process space 7152 GB (7 TB) User process space ~8192 GB (8 TB) System Space ~8192 GB (8 TB) System Space

Virtual Memory Types Address space breakdown Shareable (e.g. EXE, DLL, shared memory, other memory mapped files) Private (e.g. process heap) Reserved (not yet committed) Free (not yet defined) Performance counters available: Private Bytes – private virtual memory Virtual Bytes – total of shareable+private+reserved No separate counters for Shareable or Reserved or Free

Looking at a Process Address Space Sysinternals VMMap utility shows address-space usage

Tracking Process Commit Usage Most virtual memory problems are due to a process leaking private committed memory Heap, GC heap, language heaps (CRT) Testlimit -m Viewing process commit usage: Private Bytes perf counter Process Explorer’s Private Bytes: Before Vista: Task Manager’s Virtual Size Vista and later: Task Manager’s Commit Size

The System Commit Limit Another virtual memory limit is total virtual memory: system commit limit Committed virtual memory must be backed either by physical memory or stored in the paging file Sum of (most of) physical memory and paging files The system commit limit can change Default paging file (System Managed): Minimum: 1.5x RAM if RAM < 1 GB; RAM otherwise Maximum: 3x RAM or 4 GB, whichever is larger

Viewing the System Commit Limit Process Explorer tracks the current commit limit in the System Information window: Prior to Vista, Task Manager shows it as “PF Usage”: Task Manager on Vista and higher doesn’t show a graph:

Exhausting the System Commit Limit 64-bit Testlimit -m (committed memory) will exhaust the system commit limit before its address space: You can increase the System commit limit by adding RAM or increasing the pagefile size

Sizing the Paging File Many recommendations use a formula based on RAM (1.5x, 2x, etc.) Actually, the more RAM, the smaller the paging file needed Should be based on workload usage of committed virtual memory Look at commit after workload has run Pre-Vista: Task Manager Vista+: Process Explorer Apply a formula to that to give buffer (1.5x or 2x) Make sure it’s big enough to hold a crash dump

Outline Virtual Memory Physical Memory Paged and Nonpaged Pool Processes and Threads Handles

Windows Physical Memory Limits The amount of physical memory Windows will use depends on: SKU licensing 32-bit vs 64-bit Tested systems Kernel address space See “Memory Limits for Windows Releases” for a complete list: http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx

Physical Memory on 64-bit Systems Kernel address space limits it to 2 PB (Petabytes) x64 page table structures limit physical memory to 48-bits physical addressing (256 TB) Current x64 hardware supports up to 1 TB

64-Bit Server Current Windows Server limits: Windows Server 2003: 16 GB (Standard) – 2 TB (Datacenter) Windows Server 2008: 32 GB (Standard) – 2 TB (Datacenter) Windows Server 2008R2: 8 GB (Foundation) – 2 TB (Datacenter) For Datacenter Edition, the limit is based on SKU and hardware testing We won’t support what we can’t test Limits have gone up even for a particular release of Windows because of access to larger machines Datacenter Edition maxes out all licensing, including memory

2 Terabytes The Windows architecture can support much more, but as of now, the largest system we’ve tested is 2 TB (Itanium):

32-Bit Server Windows 32-bit Server limits: Windows Server 2003: 2 GB (Web) – 128 GB (Datacenter) Windows Server 2008: 4 GB (Standard) – 64 GB (Datacenter) Windows Server 2008 R2: n/a The 32-bit Server limit is based on the kernel address space PFN entry is 28-bytes so 128 GB requires 890 MB Limits other system code and data Also why limit is lower when booted 4GT User Address Space System PFN Database

Client Systems 64-bit client limits solely based on licensing: 512 MB in Windows XP Starter 128 GB in Windows Vista Ultimate 192 GB in Windows 7 Ultimate 32-bit client limit is 4 GB Limited by potential driver incompatibility And usable amount might be less than 4 GB…

Drivers and Physical Memory To be safe, client limits physical memory addresses to 4 GB Some client device drivers still truncate addresses > 4 GB Client systems with > 4 GB are recent Servers are safe with > 4 GB Have had more than 4 GB for longer Devices are more mainstream Policy allows only WHQL-signed drivers 0.2 GB 0x00032000 DMA Destination Device Data 4 GB 4.2 GB Target 0x00000001 0x00032000 6 GB

Device Memory Device memory can push physical memory above 4 GB Device memory can push physical memory above 4 GB Firmware maps all device memory below 4GB For compatibility with older operating systems Biggest device memory consumer is video Result is that some physical memory may lie above 4 GB Watch out for Shared Video Memory (UMA) Video RAM 4 GB Inaccessible RAM

Physical Memory Management Working set: all the physical pages “owned” by a process Pre-Vista: Task Manager’s “Mem Usage” Vista+: Task Manager’s “Memory – Working Set” Process Explorer’s “Working Set” System has its own working set (broken into multiple in Windows 7) System keeps physical pages on one of several lists Working sets (process and system) Available: Free page list, Standby page list (file cache), Zero page list Modified page list: pages that must be saved before being repurposed Lists are implemented by entries in the “PFN database” Each page of physical memory has an entry in the database The Memory Manager watches paging activity to determine how to assign memory Notepad Word Explorer System Available

Working Set Lists newer pages older pages A process always starts with just enough working set to describe the address space It then incurs page faults when referencing a page that isn’t in its working set Many page faults may be resolved from memory (from other working sets, the standby or Modified page lists) The Memory Manager can take away pages and reassign them (trimming) Note: the amount of physical memory assigned to a process is invisible to the process It simply affects the performance of the process

Do You Have Enough Memory? There’s no sure-fire rule or counter to tell if you if you have enough memory The general rule: available memory remains generally low Use Perfmon to monitor available memory over time Use Process Explorer, or on Vista and later Task Manager, to monitor physical memory usage Use Process Explorer or Task Manager to see instantaneous value Watch in Process Monitor for excessive reads from paging file

Outline Virtual Memory Physical Memory Paged and Nonpaged Pool Processes and Threads Handles

Paged and Nonpaged Pool Backed by physical memory Kernel and device driver heap used by code paths that can’t take interrupts (ISRs and DPCs) or that hold spin locks Page fault in those paths results in IRQL_NOT_LESS_OR_EQUAL Paged pool: Backed by virtual memory (can be saved to paging file) Largest consumer is typically Registry Drivers use ExAllocatePoolWithTag to allocate both pool types Parameter specifies pool type Tag is 4-byte value that should be human readable and is used to help track pool usage

Viewing Pool Usage There are three performance counters that indicate pool usage: Pool nonpaged bytes Pool paged bytes (virtual size of paged pool – some may be paged out) Pool paged resident bytes (physical size of paged pool) No performance counter for viewing maximums Use Process Explorer to view current and maximums Must configure symbols to see maximum System Information dialog shows usage

Pool Limits Pool maximums are based on system address space and physical memory Nonpaged Pool 32-bit 64-bit XP, Server 2003 up to 1.2GB RAM: 32-256 MB  > 1.2GB RAM: 256MB min( ~400K/MB of RAM, 128GB) Vista, Server 2008, Windows 7, Server 2008 R2 min( ~75% of RAM, 2GB) min(~75% of RAM, 128GB) Paged Pool 32-bit 64-bit XP, Server 2003 XP: up to 491MB Server 2003: up to 650MB min( 4 * nonpaged pool limit, 128GB) Vista, Server 2008, Windows 7, Server 2008 R2 min( system commit limit, 2GB) min( system commit limit, 128GB)

Testing Pool Limits Sysinternals Notmyfault can leak paged and nonpaged pool Exhausting either pool type leads to erratic behavior and likely data loss or corruption

Tracking Pool Leaks Poolmon from the Windows Driver Kit shows pool usage by tag http://www.microsoft.com/whdc/devtools/WDK/ Search strings of drivers for pool tag: strings * | findstr <tag>

Pool Leaks and Crashes If a system crashes and you suspect a pool leak, use !vm to confirm: Use !poolused to view pool usage by tag Use 1 for nonpaged pool and 4 for paged

Outline Virtual Memory Physical Memory Paged and Nonpaged Pool Processes and Threads Handles

Thread Limits User-mode stack area is reserved and committed as needed Stacks grow down in memory “Guard” pages trigger stack expansion Committed Stack Grows Down Committed Thread Stack Guard Guard Reserved Reserved Before Expansion Before Expansion

32-Bit Thread Limits Default 2 GB address space and stack reserve: 2 GB / 1 MB = ~2,000 threads 32-bit thread on 64-bit Windows has additional overhead of 256 KB 64-bit stack

64-Bit Thread Limits On 64-bit Windows, the system commit limit will be hit before address space exhaustion Even with default 2 MB stacks

Process Limits Minimum process working set also charged against “resident available memory” Reserved physical memory so that process can run Default minimum is 200 KB, which Process Explorer shows Every active process has one thread, so process limit will be lower than thread limit

Outline Virtual Memory Physical Memory Paged and Nonpaged Pool Processes and Threads Handles

Handles and Objects Windows defines objects to represent operating system resources There are 42 in Windows 7 Use Sysinternals Winobj to view defined objects When a process wants to interact with a resource, it opens it That creates a handle in the process’ handle table Handle entry points at the object and records granted access Subsequent operations reference the handle Object Handle Table Entry Pointer Accesses

Handle Limits A process handle table can grow to about 16 million handles Handle tables are stored in paged pool, so that can also be a limiter 32-bit Windows: 8 bytes per entry 64-bit Windows: 16 bytes per entry

Tracking Handle Leaks A handle leaker will show a growing handle count over time Task Manager and Process Explorer show handle count columns Process Explorer also shows it in process properties dialog Process Explorer uses difference highlighting to show newly opened and closed handles

Tracking Handle Call Stacks Windows can capture call stacks at the time of handle open and close Use Application Verifier to track from process start Free download from Microsoft Use Windbg to enable and disable dynamically !htrace –enable, !htrace –disable

Viewing Handle Call Stacks Use Windbg to view stacks Use !htrace (no parameters) to show all recorded stacks Use !htrace –snapshot and !htrace –diff to see differences

Conclusion Knowing system limits can help you avoid resource exhaustion By knowing how to track resource usage you can identify potential problems and root cause them Visit my blog for posts on the limits I’ve covered and others Come to my other sessions: SIA301 Windows and Malware: Which Features Are Security and Which Aren't Tomorrow at 9am CLI301 Case of the Unexplained... Windows Troubleshooting Tomorrow at 1pm

Complete an evaluation on CommNet and enter to win an Xbox 360 Elite!

Required Slide © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.