1. 1 Virtual Memory

2. 2 Outline Pentium/Linux Memory System Core i7 Suggested reading: 9.6, 9.7

3. 3 bus interface unit DRAM external system bus (e.g. PCI) instruction fetch unit L1 i-cache L2 cache cache bus L1 d-cache inst TLB data TLB processor package 32 bit address space 4 KB page size L1, L2, and TLBs 4-way set associative inst TLB 32 entries 8 sets data TLB 64 entries 16 sets L1 i-cache and d-cache 16 KB 32 B line size 128 sets L2 cache unified 128 KB -- 2 MB 32 B line size P6 Memory System

4. 4 Review of Address Translation Basic Parameters –N = 2 n = Virtual address limit –M = 2 m = Physical address limit –P = 2 p = page size (bytes). Components of the virtual address (VA) –VPO: Virtual page offset –VPN: Virtual page number Components of the physical address (PA) –PPO: Physical page offset (same as VPO) –PPN: Physical page number

5. 5 Review of Address Translation virtual page numberpage offset virtual address physical page numberpage offset physical address 0p–1 address translation pm–1 n–10p–1p TLB tagTLB index Cache tagCache indexCache offset

6. 6 Review of Address Translation Components of the virtual address (VA) –VPO: Virtual page offset –VPN: Virtual page number –TLBI: TLB index –TLBT: TLB tag Components of the physical address (PA) –PPO: Physical page offset (same as VPO) –PPN: Physical page number –CO: Byte offset within cache line –CI: Cache index –CT: Cache tag

7. 7 Multi-Level Page Tables multi-level page tables e.g., 2-level page table –Level 1 table –Level 1 table: 1024 entries, each of which points to a Level 2 page table. –Level 2 table –Level 2 table: 1024 entries, each of which points to a page Level 1 Table... Level 2 Tables

8. 8 P6 Address Translation

9. 9 Page directory –1024 4-byte page directory entries (PDEs) that point to page tables –one page directory per process. –page directory must be in memory when its process is running –always pointed to by PDBR Page tables: –1024 4-byte page table entries (PTEs) that point to pages. –page tables can be paged in and out. page directory... Up to 1024 page tables 1024 PTEs 1024 PTEs 1024 PTEs... 1024 PDEs P6 Page Table

10. 10 Page table physical base addrAvailGPSACDWTU/SR/WP=1 Page table physical base address: 20 most significant bits of physical page table address (forces page tables to be 4KB aligned) Avail: available for system programmers G: global page (don’t evict from TLB on task switch) PS: page size 4K (0) or 4M (1) A: accessed (set by MMU on reads and writes, cleared by software) CD: cache disabled (1) or enabled (0) WT: write-through or write-back cache policy for this page table U/S: user or supervisor mode access R/W: read-only or read-write access P: page table is present in memory (1) or not (0) 3112119876543210 Available for OS (page table location in secondary storage)P=0 3101 P6 page directory entry (PDE)

11. 11 Page physical base addressAvailG0DACDWTU/SR/WP=1 Page base address: 20 most significant bits of physical page address (forces pages to be 4 KB aligned) Avail: available for system programmers G: global page (don’t evict from TLB on task switch) D: dirty (set by MMU on writes) A: accessed (set by MMU on reads and writes) CD: cache disabled or enabled WT: write-through or write-back cache policy for this page U/S: user/supervisor R/W: read/write P: page is present in physical memory (1) or not (0) 3112119876543210 Available for OS (page location in secondary storage)P=0 3101 P6 page table entry (PTE)

12. 12 PDE PDBR physical address of page table base (if P=1) physical address of page base (if P=1) physical address of page directory word offset into page directory word offset into page table page directorypage table VPN1 10 VPO 1012 VPN2 Virtual address PTE PPNPPO 20 12 Physical address word offset into physical and virtual page Page tables Translation

13. 13 P6 TLB Translation

14. 14 TLB entry (not all documented, so this is speculative): –V: indicates a valid (1) or invalid (0) TLB entry –PD: is this entry a PDE (1) or a PTE (0)? –tag: disambiguates entries cached in the same set –PDE/PTE: page directory or page table entry Structure of the data TLB: –16 sets, 4 entries/set PDE/PTETagPDV 111632 entry... set 0 set 1 set 2 set 15 P6 TLB

15. 15 1. Partition VPN into TLBT and TLBI. 2. Is the PTE for VPN cached in set TLBI? 3. Yes: then build physical address. 4. No: then read PTE (and PDE if not cached) from memory and build physical address. CPU VPNVPO 2012 TLBTTLBI 416 virtual address PDEPTE... TLB miss TLB hit page table translation PPNPPO 2012 physical address 1 2 3 4 Translating with the P6 TLB

16. 16 P6 Page Table Translation

17. 17 Case 1/1: page table and page present. MMU Action: –MMU build physical address and fetch data word. OS action –none VPN VPN1VPN2 PDE PDBR PPNPPO 2012 20 VPO 12 p=1PTEp=1 Data page data Page directory Page table Mem Disk Translating with the P6 page tables (case 1/1)

18. 18 Case 1/0: page table present but page missing. MMU Action: –page fault exception –handler receives the following args: VA that caused fault fault caused by non-present page or page-level protection violation read/write user/supervisor VPN VPN1VPN2 PDE PDBR 20 VPO 12 p=1PTE Page directory Page table Mem Disk Data page data p=0 Translating with the P6 page tables (case 1/0)

19. 19 OS Action: –Check for a legal virtual address. –Read PTE through PDE. –Find free physical page (swapping out current page if necessary) –Read virtual page from disk and copy to virtual page –Restart faulting instruction by returning from exception handler. VPN VPN1VPN2 PDE PDBR 20 VPO 12 p=1PTEp=1 Page directory Page table Data page data PPNPPO 2012 Mem Disk Translating with the P6 page tables (case 1/0, cont)

20. 20 Case 0/1: page table missing but page present. Introduces consistency issue. –potentially every page out requires update of disk page table. Linux disallows this –if a page table is swapped out, then swap out its data pages too. VPN VPN1VPN2 PDE PDBR 20 VPO 12 p=0 PTEp=1 Page directory Page table Mem Disk Data page data Translating with the P6 page tables (case 0/1)

21. 21 Case 0/0: page table and page missing. MMU Action: –page fault exception VPN VPN1VPN2 PDE PDBR 20 VPO 12 p=0 PTE Page directory Page table Mem Disk Data page datap=0 Translating with the P6 page tables (case 0/0)

22. 22 OS action: –swap in page table. –restart faulting instruction by returning from handler. Like case 0/1 from here on. VPN VPN1VPN2 PDE PDBR 20 VPO 12 p=1PTE Page directory Page table Mem Disk Data page data p=0 Translating with the P6 page tables (case 0/0, cont)

23. 23 P6 L1 Cache Access

24. 24 Partition physical address into CO, CI, and CT. Use CT to determine if line containing word at address PA is cached in set CI. If no: check L2. If yes: extract word at byte offset CO and return to processor. physical address (PA) data 32... CTCO 205 CI 7 L2 and DRAM L1 (128 sets, 4 lines/set) L1 hit L1 miss L1 cache access

25. 25 Core i7 –The 64-bit Nehalem microarchitecture –Current support 48-bit (256 TB) virtual address space and 52-bit (4 PB) physical address space –Compatibility mode support 32-bit (4 GB) virtual and physical address spaces Core i7 Summary

26. 26 Processor package Core i7 Memory System Core x 4 Registers Instruction fetch L1 d-cache 32 KB,8-way L1 i-cache 32 KB,8-way L2 unified cache 256 KB, 8-way L3 unified cache 8 MB, 16-way (shared by all cores) MMU (addr translation) L1 d-TLB 64 entries, 4-way L1 i-TLB 64 entries, 4-way L2 unified TLB 512 entries, 4-way QuickPath interconnect 4 links @ 25.6 GB/s 102.4 GB/s total DDR3 memory controller 3 x 64 bit @ 10.66 GB/s 32 GB/s total (shared by all cores) Main memory To other cores To I/O bridge

27. ... 27... CPU VPN VPO 3612 432 Virtual address (VA) L1 TLB (16 sets, 4 entries/set) PPN PPO 404012 TLB miss TLB hit physical address (PA) result 32/64 CT 404066 L2, L3, and main memory L1 d-cache (64 sets, 8 lines/set) L1 hit L1 miss TLBT TLBI Core i7 Address Translation CI CO VPN1 VPN2 VPN3 VPN4 PTE CR3 Page Tables 9 999

28. 28 Page table physical base addrUnusedGPSACDWTU/SR/WP=1 XD Disable or enable instruction fetches Base addr 40 most significant bits of base address of child page table (forces page tables to be 4KB aligned) G global page (don’t evict from TLB on task switch) PS Page size either 4K(0) or 4M(1) (defined for Level 1 PTEs only) A Reference bit (set by MMU on reads and writes, cleared by software) CD Cache disabled(1) or enabled(0) for child page table WT Write-through or write-back cache policy U/S User or supervisor(kernel) mode access permission R/W Read-only or read-write access permissiom P Child page table present in memory(1) or not(0) 515112119876543210 Available for OS (page table location in secondary storage)P=0 Level 1, Level 2 and Level 3 Page Table Entry Unused 52 XD 6263

29. 29 Page table physical base addrUnusedGDACDWTU/SR/WP=1 XD Disable or enable instruction fetches Base addr 40 most significant bits of base address of child page table (forces page tables to be 4KB aligned) G global page (don’t evict from TLB on task switch) D Dirty bit (Set by MMU on writes, cleared by software) A Reference bit (set by MMU on reads and writes, cleared by software) CD Cache disabled(1) or enabled(0) for child page table WT Write-through or write-back cache policy U/S User or supervisor(kernel) mode access permission R/W Read-only or read-write access permissiom P Child page table present in memory(1) or not(0) 515112119876543210 Available for OS (page table location in secondary storage)P=0 Level 4 Page Table Entry Unused 52 XD 6263

30. 30 Physical address of L1 PT physical address of page 512 GB region per entry L1 PT Page global directory 12 Virtual address 404012 Physical address Offset into physical and virtual page Page tables Translation VPN 1VPN 2VPN 3VPN 4VPO 9999 PPO L1 PTE CR3 1 GB region per entry L2 PT Page upper directory L2 PTE 2 MB region per entry L3 PT Page middle directory L3 PTE 4 KB region per entry L4 PT Page directory L4 PTE PPO 12 40 9999

31. Cute Trick for Speeding Up L1 Access Observation –Bits that determine CI identical in virtual and physical address –Can index into cache while address translation taking place –Generally we hit in TLB, so PPN bits (CT bits) available next –“Virtually indexed, physically tagged” –Cache carefully sized to make this possible Physical address (PA) CTCO 366 CI 6 Virtual address (VA) VPNVPO 3612 PPOPPN Address Translation No Change CI L1 Cache CT Tag Check

32. 32 Process-specific data structures (e.g. task and mm structs, page tables, kernel stack) Process-specific data structures (e.g. task and mm structs, page tables, kernel stack) Physical memory Kernel code and data User stack Memory mapped region for shared libraries Run-time heap (via malloc) Uninitialized data (.bss) Initialized data (.data) Program text (.text) Kernel virtual memory Process virtual memory Different for each process Identical for each process %esp brk x08048000 (32) x40000000 (64) Linux Virtual Memory System

33. Next Memory Mapping Suggested reading: 10.7, 10.8 33