 Demand Technology Software, Inc. 32-bit Virtual Memory Constraints in Windows 2000 and 2003 Mark Friedman Demand Technology 1020 Eighth Avenue South, Suite 6, Naples, FL USA phone: (239) fax: (239)

 Demand Technology, Inc. 32-bit VM Constraints 2 Outline Topics  Virtual Memory concepts  Virtual Memory constraints in 32-bit Windows  Physical Address Extension (PAE)  Leaking processes  Page faults/sec

 Demand Technology, Inc. 32-bit VM Constraints 3 Virtual Memory Manager 4 GB virtual per process address space  Lower 2 GB - Private Page 0 reserved Code pages Heap  Upper 2 GB - System System code Shared dlls System cache

 Demand Technology, Inc. 32-bit VM Constraints 4 Virtual Memory Manager  All address spaces share the virtual addresses from the same upper 2 GB of the System area  All virtual addresses in the lower 2 GB User area are unique  Shared memory in the System range can be used for IPC

 Demand Technology, Inc. 32-bit VM Constraints 5 Virtual Memory Manager  Commit Limit Limit on the number of Virtual Memory pages that the system will allocate Sizeof (RAM) + paging file(s) Memory allocations start to fail as % Committed Bytes in Use  100%

 Demand Technology, Inc. 32-bit VM Constraints 6 Virtual Memory Manager  Commit Limit But page files are extendible! Should you or shouldn’t you?

 Demand Technology, Inc. 32-bit VM Constraints 7 Virtual Memory Manager

 Demand Technology, Inc. 32-bit VM Constraints 8 Extended Virtual Addressing Boot.ini /3 GB switch  Virtual storage constraint relief for some applications  Squeezes the System VM into 1 GB Physical Address Extension (PAE)  Supports 36-bit real addresses on Xeon processors Address Windowing Extensions (AWE)  Permits processes to address real memory > 4 GB

 Demand Technology, Inc. 32-bit VM Constraints 9 Virtual Memory Manager Boot.ini /3 GB switch  Virtual storage constraint relief for some server applications SQL Server Exchange 2000 Etc.  /userva=SizeInMB subparameter, where SizeinMB can be any value between 2048 and 3072

 Demand Technology, Inc. 32-bit VM Constraints 10 Physical Address Extension PAE is supported on most recent Intel servers  Physical addresses are 36-bits  Up to 64 GB of RAM can be installed Page Table Entries (PTEs) remain 32-bits Process address spaces are still limited to 4 GB. How can they exploit 64 GB of RAM?  Expand sideways, using multiple address spaces  Large Memory Enabled (LME) device drivers  Address Windowing Extensions (AWE) Boot.ini /pae switch  Limitation: system addresses can be no higher than 16 GB

 Demand Technology, Inc. 32-bit VM Constraints 11 Physical Address Extension  Expand sideways, using multiple address spaces e.g., MS SQL Server 16 GB 4 GB SQL Server 4 GB SQL Server 4 GB SQL Server

 Demand Technology, Inc. 32-bit VM Constraints 12 Address Windowing Extensions (AWE) API that allows processes to address real memory locations outside their 4 GB virtual addressing range  Create and manage memory Overlays  Used in conjunction with PAE AllocateUserPhysicalPages MapUserPhysicalPages Virtual Alloc

 Demand Technology, Inc. 32-bit VM Constraints 13 AWE Note: Frequent unmapping and remapping of Physical Memory blocks is expensive!

 Demand Technology, Inc. 32-bit VM Constraints 14 AWE support MS Exchange: /3 GB support only MS SQL Server 2000: /3 GB, PAE, & AWE  Multiple process instances  awe enabled = 1  Also Set max server Oraclememory Oracle: /3 GB, PAE, & AWE  AWE_WINDOW_MEMORY SAS: /3 GB, PAE  Work library can be placed in PAE memory

 Demand Technology, Inc. 32-bit VM Constraints 15 Win64 Virtual Memory

 Demand Technology, Inc. 32-bit VM Constraints 16 Virtual Memory Manager Real Memory allocation Counters Available Bytes Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes  Instantaneous Counters (all reported as Bytes)  Trimmed Pages on the Modified List are not counted (assumed to be small)  Cache Bytes is actually the (pageable) System Working Set = Pool Paged Resident Bytes + System Cache Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes

 Demand Technology, Inc. 32-bit VM Constraints 17 Real Memory allocation Counters

 Demand Technology, Inc. 32-bit VM Constraints 18 Virtual Memory Manager Memory allocation Counters Available Bytes Available KBytes Available MBytes Pool non-paged Bytes, Pool Paged Resident Bytes, System Cache Resident Bytes, System Code Resident Bytes, System Driver Resident Bytes

 Demand Technology, Inc. 32-bit VM Constraints 19 What is the difference between Pool Paged Bytes and Pool Nonpaged Bytes? Which applications would use which VM pool?

 Demand Technology, Inc. 32-bit VM Constraints 20 Virtual Memory map

 Demand Technology, Inc. 32-bit VM Constraints 21 Virtual Memory map Paged  Memory mapped files  Shared DLLs  File Server  IIS (html, jpg, gif)  Page tables nonPaged:  I/O buffers used by device drivers  TCP Session data  Kernel threads (win32k.sys per process)

 Demand Technology, Inc. 32-bit VM Constraints 22 Virtual Memory map (Win2K) Virtual memory map features overflow areas for the file cache, PTEs, and Paged/nonPaged Pools One reserved area of virtual memory can fill up faster than the others; No hard limits on the sizes of the pools

 Demand Technology, Inc. 32-bit VM Constraints 23 Virtual Memory map (Win2K) No hard limits on the sizes of the pools One reserved area of virtual memory can fill up faster than the others; AreaMax Size file cache960 MB PTEs660 MB Paged pool470 MB nonPaged Pools 256 MB

 Demand Technology, Inc. 32-bit VM Constraints 24 Virtual Memory map (Win2003) Virtual memory map is dynamic, subject to adjustment by the OS One reserved area of virtual memory can fill up faster than the others When any one area fills, VMM routines can empty the file cache to acquire more vm for the paged/nonpaged pools Paged Pool nonPaged Pool PTEs Cache

 Demand Technology, Inc. 32-bit VM Constraints 25 Nonpaged pool size (ref. Q126402): Obsolete in Windows 2003 Dynamic adjustment of system virtual memory when one pool is exhausted. NonPagedPoolSize, NonPagedPoolSize, and SystemPages can also be set explicitly. But how can you tell when you are running out of space in one of these pools?

 Demand Technology, Inc. 32-bit VM Constraints 26 Default Paged and Nonpaged pool size can be overridden:

 Demand Technology, Inc. 32-bit VM Constraints 27 Default Paged and Nonpaged pool size can be overridden (Win 2003): System Pages = x’ffff ffff’ 16 or -1  Maximizes the number of PTEs that can be built  Potentially useful with /PAE or Terminal Services PagedPoolSize = x’ffff ffff’ 16 or -1  Allows the OS maximum flexibility to determine the size of the Paged Pool  Or PagedPoolSize and NonPagedPoolSize can be set explicitly LargeSystemCache controls the maximum virtual address range of the file cache  512 MB – 1 GB

 Demand Technology, Inc. 32-bit VM Constraints 28 !vm Debugger command runs the Page Frame Number (PFN) database Available Pages: 4920 ( Kb) ResAvail Pages: 358 ( 1432 Kb) Locked IO Pages: 251 ( 1004 Kb) Free System PTEs: ( Kb) Free NP PTEs: ( Kb) Free Special NP: 0 ( 0 Kb) Modified Pages: 596 ( 2384 Kb) Modified PF Pages: 660 ( 2640 Kb) NonPagedPool Usage: 2750 ( Kb) NonPagedPool Max: ( Kb) PagedPool 0 Usage: 3544 ( Kb) PagedPool 1 Usage: 1359 ( 5436 Kb) PagedPool 2 Usage: 1340 ( 5360 Kb) PagedPool Usage: 6243 ( Kb) PagedPool Maximum: ( Kb) Shared Commit: 6842 ( Kb) Special Pool: 0 ( 0 Kb) Shared Process: 3688 ( Kb) PagedPool Commit: 6398 ( Kb) Driver Commit: 1630 ( 6520 Kb) Committed pages: ( Kb) Commit limit: ( Kb)

 Demand Technology, Inc. 32-bit VM Constraints 29 !poolused Debugger command accounts for all system pool allocations (plus pooltags.txt documentation) lkd> !poolused 2 Sorting by NonPaged Pool Consumed Pool Used: NonPaged Paged Tag Allocs Used Allocs Used LSwi NV File MmCm LSwr Devi Thre PcNw Irp

 Demand Technology, Inc. 32-bit VM Constraints 30 Poolmon utility (from DDK)

 Demand Technology, Inc. 32-bit VM Constraints 31 Virtual Memory constraints Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM)  4 GB  2 GB private area is not enough virtual memory for some applications e.g., SQL Server, Exchange database (store.exe) Due to fragmentation, it is typically not possible to allocate all 2 GB  2 GB system area is not enough virtual memory for some applications File cache for a conventional IIS-managed web site with many static.htm, jpg, gif, etc., files to retrieve

 Demand Technology, Inc. 32-bit VM Constraints 32 Virtual Memory constraints Virtual Memory constraints in 32-bit Windows tend to appear when sizeof (RAM)  4 GB  Ample RAM exists, but it is not possible for your applications to access it due to virtual memory addressing limitations  Large number of Available Bytes

 Demand Technology, Inc. 32-bit VM Constraints 33 Virtual Memory constraints So try the /3 GB switch  1 GB system area is not enough virtual memory for some applications Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool – Where Session data from TCP connections is stored  Due to fragmentation, it may not be possible to failover a 2 GB+ private address space (e.g., SQL Server, MS Exchange database – store.exe) using Microsoft Cluster Server (MCS) During address space recovery on the standby node, the entire virtual memory allocation is acquired at one time

 Demand Technology, Inc. 32-bit VM Constraints 34 Virtual Memory constraints So try the /3 GB switch  1 GB system area is not enough virtual memory for some applications  PagedPoolSize and NonPagedPoolSize defaults are cut in ½ Possible shortage of Free System Page Table Entries Possible shortage of Nonpaged Pool – Where Session data from TCP connections is stored – Where kernel threads are created per process for calls to win32k.sys (especially impacts large Terminal Server environments)

 Demand Technology, Inc. 32-bit VM Constraints 35 Exchange 2000 memory tuning Exchange default memory allocation parameters are “self-tuning,” but may not be optimal on servers with > 1 GB RAM  Adjust HKLM\SYSTEM\CurrentControlSet\Services\SMTPSVC\Queuin g MsgHandleThreshold & MsgHandleAsyncThreshold HKLM\SYSTEM\CurrentControlSet\Services\Inetinfo\Parameters FileCacheMaxHandles

 Demand Technology, Inc. 32-bit VM Constraints 36 Exchange 2000 memory tuning Store DB cache  Store cache normally expands to fill RAM  But, maximum cache = 900 MB  This value can be adjusted using ADSI Edit tool: msExchESEParamCacheSizeMax msExchESEParamCacheSizeMin  Also, consider adjusting msExchESEParamLogBuffers attribute for active, back-end servers

 Demand Technology, Inc. 32-bit VM Constraints 37 Detecting memory leaks Processes that allocate virtual memory, but later “forget” to free it. MS says leaks won’t happen in.Net managed code due to automatic garbage collection But, meanwhile…, Where to look depends on whether process or system addresses are being allocated:  Per Process: Virtual Bytes, Private Bytes, Pool Paged Bytes, Handle Count  System level: Memory Pool Paged Bytes, Pool Nonpaged Bytes and the Objects Object

 Demand Technology, Inc. 32-bit VM Constraints 38 Detecting memory leaks Look for a steady increase or a sharp spike in process Virtual Bytes, or the System’s Pool Paged Bytes.  If RAM is not full, the leak may also be manifest in the Memory allocation counters and result in increased paging, if RAM fills up. For example:

 Demand Technology, Inc. 32-bit VM Constraints 39 Detecting memory leaks (1)

 Demand Technology, Inc. 32-bit VM Constraints 40 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 41 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 42 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 43 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 44 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 45 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 46 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 47 Detecting memory leaks (2)

 Demand Technology, Inc. 32-bit VM Constraints 48 Lab exercise. Open SUPPORT EXCHG2K.sum. smf, SUPPORT EXCHG2K.sum. smf, and SUPPORT EXCHG2K.sum. smf and investigate virtual memory usage. Can you find the application that is leaking?

 Demand Technology, Inc. 32-bit VM Constraints 49 Detecting Memory Leaks (3)

 Demand Technology, Inc. 32-bit VM Constraints 50 Detecting Memory Leaks (3)

 Demand Technology, Inc. 32-bit VM Constraints 51 Detecting Memory Leaks (3)

 Demand Technology, Inc. 32-bit VM Constraints 52 Detecting Memory Leaks (3)

 Demand Technology, Inc. 32-bit VM Constraints 53 Questions ?