2. Pushing the Limits of Windows
Mark Russinovich
Technical Fellow
Microsoft Corporation
Session Code: CLI402
*Portions derived from David Solomon’s Windows Internals Seminar

3. About Me Technical Fellow, Microsoft
Co-founder and Chief Software Architect of Winternals Software
Co-author of Windows Internals 4th and 5th Edition and Inside Windows rd Edition with David Solomon
Author of TechNet Sysinternals
Home of blog and forums
Contributing Editor TechNet Magazine, Windows IT Pro Magazine
Ph.D. in Computer Engineering

4. Pushing the Limits Resource exhaustion is obviously not a good thing
At the minimum it causes service outage
At worst it can cause data loss or even corruption
As a Windows systems administrator you need to know the limits
For capacity planning
For systems monitoring
This session explains core Windows kernel limits, where they come from, and how to monitor them
I’ll use Sysinternals Testlimit and Notmyfault to stress resources:

5. Outline Virtual Memory Physical Memory Paged and Nonpaged Pool
Processes and Threads
Handles

6. Address Space Limits One virtual memory limit is address space
Process can run out of space to allocate virtual memory
Typically only an issue for 32-bit processes
Use Testlimit -r (reserve memory) to see address space exhaustion:
Process Explorer Virtual Size shows amount of address space used

7. (boot.ini: /3GB or /USERVA)
32-bit x86 Address Space
32-bits = 2^32 = 4 GB
4GT Address Space
(boot.ini: /3GB or /USERVA)
(BCD: increasuserva)
Default
2 GB
User
process
space
3 GB
User
process
space
2 GB
System
Space
1 GB
System Space

8. 64-bit Address Spaces 64-bits = 2^64 = 17,179,869,184 GB x64 IA-64
32-bit process on 64-bit Windows = 4 GB
x64
(AMD64 & Intel 64)
IA-64
8192 GB
(8 TB)
User
process
space
7152 GB
(7 TB)
User
process
space
~8192 GB
(8 TB)
System
Space
~8192 GB
(8 TB)
System
Space

9. Virtual Memory Types Address space breakdown
Shareable (e.g. EXE, DLL, shared memory, other memory mapped files)
Private (e.g. process heap)
Reserved (not yet committed)
Free (not yet defined)
Performance counters available:
Private Bytes – private virtual memory
Virtual Bytes – total of shareable+private+reserved
No separate counters for Shareable or Reserved or Free

10. Looking at a Process Address Space
Sysinternals VMMap utility shows address-space usage

11. Tracking Process Commit Usage
Most virtual memory problems are due to a process leaking private committed memory
Heap, GC heap, language heaps (CRT)
Testlimit -m
Viewing process commit usage:
Private Bytes perf counter
Process Explorer’s Private Bytes:
Before Vista: Task Manager’s Virtual Size
Vista and later: Task Manager’s Commit Size

12. The System Commit Limit
Another virtual memory limit is total virtual memory: system commit limit
Committed virtual memory must be backed either by physical memory or stored in the paging file
Sum of (most of) physical memory and paging files
The system commit limit can change
Default paging file (System Managed):
Minimum: 1.5x RAM if RAM < 1 GB; RAM otherwise
Maximum: 3x RAM or 4 GB, whichever is larger

13. Viewing the System Commit Limit
Process Explorer tracks the current commit limit in the System Information window:
Prior to Vista, Task Manager shows it as “PF Usage”:
Task Manager on Vista and higher doesn’t show a graph:

14. Exhausting the System Commit Limit
64-bit Testlimit -m (committed memory) will exhaust the system commit limit before its address space:
You can increase the System commit limit by adding RAM or increasing the pagefile size

15. Sizing the Paging File
Many recommendations use a formula based on RAM (1.5x, 2x, etc.)
Actually, the more RAM, the smaller the paging file needed
Should be based on workload usage of committed virtual memory
Look at commit after workload has run
Pre-Vista: Task Manager
Vista+: Process Explorer
Apply a formula to that to give buffer (1.5x or 2x)
Make sure it’s big enough to hold a crash dump

16. Outline Virtual Memory Physical Memory Paged and Nonpaged Pool
Processes and Threads
Handles

17. Windows Physical Memory Limits
The amount of physical memory Windows will use depends on:
SKU licensing
32-bit vs 64-bit
Tested systems
Kernel address space
See “Memory Limits for Windows Releases” for a complete list:

18. Physical Memory on 64-bit Systems
Kernel address space limits it to 2 PB (Petabytes)
x64 page table structures limit physical memory to 48-bits physical addressing (256 TB)
Current x64 hardware supports up to 1 TB

19. 64-Bit Server Current Windows Server limits:
Windows Server 2003: 16 GB (Standard) – 2 TB (Datacenter)
Windows Server 2008: 32 GB (Standard) – 2 TB (Datacenter)
Windows Server 2008R2: 8 GB (Foundation) – 2 TB (Datacenter)
For Datacenter Edition, the limit is based on SKU and hardware testing
We won’t support what we can’t test
Limits have gone up even for a particular release of Windows because of access to larger machines
Datacenter Edition maxes out all licensing, including memory

20. 2 Terabytes
The Windows architecture can support much more, but as of now, the largest system we’ve tested is 2 TB (Itanium):

21. 32-Bit Server Windows 32-bit Server limits:
Windows Server 2003: 2 GB (Web) – 128 GB (Datacenter)
Windows Server 2008: 4 GB (Standard) – 64 GB (Datacenter)
Windows Server 2008 R2: n/a
The 32-bit Server limit is based on the kernel address space
PFN entry is 28-bytes so 128 GB requires 890 MB
Limits other system code and data
Also why limit is lower when booted 4GT
User Address Space
System
PFN Database

22. Client Systems 64-bit client limits solely based on licensing:
512 MB in Windows XP Starter
128 GB in Windows Vista Ultimate
192 GB in Windows 7 Ultimate
32-bit client limit is 4 GB
Limited by potential driver incompatibility
And usable amount might be less than 4 GB…

23. Drivers and Physical Memory
To be safe, client limits physical memory addresses to 4 GB
Some client device drivers still truncate addresses > 4 GB
Client systems with > 4 GB are recent
Servers are safe with > 4 GB
Have had more than 4 GB for longer
Devices are more mainstream
Policy allows only WHQL-signed drivers
0.2 GB 0x
DMA Destination
Device Data
4 GB
4.2 GB
Target
0x x
6 GB

24. Device Memory Device memory can push physical memory above 4 GB
Device memory can push physical memory above 4 GB
Firmware maps all device memory below 4GB
For compatibility with older operating systems
Biggest device memory consumer is video
Result is that some physical memory may lie above 4 GB
Watch out for Shared Video Memory (UMA)
Video RAM
4 GB
Inaccessible RAM

25. Physical Memory Management
Working set: all the physical pages “owned” by a process
Pre-Vista: Task Manager’s “Mem Usage”
Vista+: Task Manager’s “Memory – Working Set”
Process Explorer’s “Working Set”
System has its own working set (broken into multiple in Windows 7)
System keeps physical pages on one of several lists
Working sets (process and system)
Available: Free page list, Standby page list (file cache), Zero page list
Modified page list: pages that must be saved before being repurposed
Lists are implemented by entries in the “PFN database”
Each page of physical memory has an entry in the database
The Memory Manager watches paging activity to determine how to assign memory
Notepad
Word
Explorer
System
Available

26. Working Set Lists
newer pages
older pages
A process always starts with just enough working set to describe the address space
It then incurs page faults when referencing a page that isn’t in its working set
Many page faults may be resolved from memory (from other working sets, the standby or Modified page lists)
The Memory Manager can take away pages and reassign them (trimming)
Note: the amount of physical memory assigned to a process is invisible to the process
It simply affects the performance of the process

27. Do You Have Enough Memory?
There’s no sure-fire rule or counter to tell if you if you have enough memory
The general rule: available memory remains generally low
Use Perfmon to monitor available memory over time
Use Process Explorer, or on Vista and later Task Manager, to monitor physical memory usage
Use Process Explorer or Task Manager to see instantaneous value
Watch in Process Monitor for excessive reads from paging file

28. Outline Virtual Memory Physical Memory Paged and Nonpaged Pool
Processes and Threads
Handles

29. Paged and Nonpaged Pool
Backed by physical memory
Kernel and device driver heap used by code paths that can’t take interrupts (ISRs and DPCs) or that hold spin locks
Page fault in those paths results in IRQL_NOT_LESS_OR_EQUAL
Paged pool:
Backed by virtual memory (can be saved to paging file)
Largest consumer is typically Registry
Drivers use ExAllocatePoolWithTag to allocate both pool types
Parameter specifies pool type
Tag is 4-byte value that should be human readable and is used to help track pool usage

30. Viewing Pool Usage
There are three performance counters that indicate pool usage:
Pool nonpaged bytes
Pool paged bytes (virtual size of paged pool – some may be paged out)
Pool paged resident bytes (physical size of paged pool)
No performance counter for viewing maximums
Use Process Explorer to view current and maximums
Must configure symbols to see maximum
System Information dialog shows usage

31. Pool Limits
Pool maximums are based on system address space and physical memory
Nonpaged Pool
32-bit
64-bit
XP, Server 2003
up to 1.2GB RAM: MB  > 1.2GB RAM: 256MB
min( ~400K/MB of RAM, 128GB)
Vista, Server 2008, Windows 7, Server 2008 R2
min( ~75% of RAM, 2GB)
min(~75% of RAM, 128GB)
Paged Pool
32-bit
64-bit
XP, Server 2003
XP: up to 491MB Server 2003: up to 650MB
min( 4 * nonpaged pool limit, 128GB)
Vista, Server 2008, Windows 7, Server 2008 R2
min( system commit limit, 2GB)
min( system commit limit, 128GB)

32. Testing Pool Limits
Sysinternals Notmyfault can leak paged and nonpaged pool
Exhausting either pool type leads to erratic behavior and likely data loss or corruption

33. Tracking Pool Leaks
Poolmon from the Windows Driver Kit shows pool usage by tag
Search strings of drivers for pool tag: strings * | findstr <tag>

34. Pool Leaks and Crashes
If a system crashes and you suspect a pool leak, use !vm to confirm:
Use !poolused to view pool usage by tag
Use 1 for nonpaged pool and 4 for paged

35. Outline Virtual Memory Physical Memory Paged and Nonpaged Pool
Processes and Threads
Handles

36. Thread Limits User-mode stack area is reserved and committed as needed
Stacks grow down in memory
“Guard” pages trigger stack expansion
Committed
Stack
Grows
Down
Committed
Thread
Stack
Guard
Guard
Reserved
Reserved
Before Expansion
Before Expansion

37. 32-Bit Thread Limits
Default 2 GB address space and stack reserve: 2 GB / 1 MB = ~2,000 threads
32-bit thread on 64-bit Windows has additional overhead of 256 KB 64-bit stack

38. 64-Bit Thread Limits
On 64-bit Windows, the system commit limit will be hit before address space exhaustion
Even with default 2 MB stacks

39. Process Limits
Minimum process working set also charged against “resident available memory”
Reserved physical memory so that process can run
Default minimum is 200 KB, which Process Explorer shows
Every active process has one thread, so process limit will be lower than thread limit

40. Outline Virtual Memory Physical Memory Paged and Nonpaged Pool
Processes and Threads
Handles

41. Handles and Objects
Windows defines objects to represent operating system resources
There are 42 in Windows 7
Use Sysinternals Winobj to view defined objects
When a process wants to interact with a resource, it opens it
That creates a handle in the process’ handle table
Handle entry points at the object and records granted access
Subsequent operations reference the handle
Object
Handle
Table
Entry
Pointer
Accesses

42. Handle Limits
A process handle table can grow to about 16 million handles
Handle tables are stored in paged pool, so that can also be a limiter
32-bit Windows: 8 bytes per entry
64-bit Windows: 16 bytes per entry

43. Tracking Handle Leaks
A handle leaker will show a growing handle count over time
Task Manager and Process Explorer show handle count columns
Process Explorer also shows it in process properties dialog
Process Explorer uses difference highlighting to show newly opened and closed handles

44. Tracking Handle Call Stacks
Windows can capture call stacks at the time of handle open and close
Use Application Verifier to track from process start
Free download from Microsoft
Use Windbg to enable and disable dynamically
!htrace –enable, !htrace –disable

45. Viewing Handle Call Stacks
Use Windbg to view stacks
Use !htrace (no parameters) to show all recorded stacks
Use !htrace –snapshot and !htrace –diff to see differences

46. Conclusion
Knowing system limits can help you avoid resource exhaustion
By knowing how to track resource usage you can identify potential problems and root cause them
Visit my blog for posts on the limits I’ve covered and others
Come to my other sessions:
SIA301 Windows and Malware: Which Features Are Security and Which Aren't
Tomorrow at 9am
CLI301 Case of the Unexplained... Windows Troubleshooting
Tomorrow at 1pm

47. Complete an evaluation on CommNet and enter to win an Xbox 360 Elite!

48. Required Slide
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.