CS 704 Advanced Computer Architecture Lecture 32 Memory Hierarchy Design (Main and Virtual Memories) Prof. Dr. M. Ashraf Chughtai Welcome to the 29th Lecture for the series of lectures on Advanced Computer Architecture

Lec. 32 Memory Hierarchy Design (8) Today’s Topics Recap: Memory Hierarchy and Cache performance Main Memory Performance Virtual Memory Performance Summary MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Recap: Memory Hierarchy design goal of memory system Low cost as of cheapest memory fast speed as of fastest memory MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Recap: Memory Hierarchy The fastest, smallest and most costly memories The slowest, biggest and cheapest memories MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Recap: Memory Hierarchy Average access speed Cost Cheapest technology Semiconductor memories Static and Dynamic RAMs Upper levels in the memory hierarchy MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) Recap: Caches Design The Caches use Static Random Access Memory Main Memory is Dynamic Random Access Memory (DRAM) (~8 ms, <5% time) The magnetic, optical or other medias virtual memory MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) Recap: Cache Design Cache and main memory are organized in equal sized blocks Word transfer Bock transfer The CPU requests contents of main memory Word transfer is fast MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Recap: Cache Performance If misses Miss penalty Cache design and the performance Techniques Miss rate Hit time MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Main Memory Organization Organizations of main memory Source for Caches Destination virtual memory MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

DRAM logical organization (4 M Bit) Column Decoder Sense Amps & I/O Memory Array (2,048 x 2,048) A0…A10 11 D Q Word Line Storage Cell Bit Line Data Out Data In Address Buffer Row Decoder MAC/VU-Advanced Computer Architecture Lec. 32– Memory Hierarchy Design (8) 10

Main Memory Performance Performance of DRAM 1: Fast page mode DRAM 2: Synchronous DRAM 3:Double Data Rate DRAM MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Main Memory Performance Fast page mode Optimizes sequential access Synchronous DRAM (SDRAM) Avoid handshaking Double Data Rate (DDR) DRAM Transmit data MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Main Memory Performance latency Average memory access time Bandwidth Number of bytes read/write per unit time Access Time Cycle Time MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Main Memory Performance Inputs/outputs and multiprocessors Low-latency memory Multiprocessor demand higher bandwidth 2nd level caches with larger block size MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Improving Main Memory Performance The most commonly used techniques are Wider Main Memory Simple Interleaved Memory Independent Memory Banks MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) 1: Wider Main Memory MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) 1: Wider Main Memory Main Memory L1 cache Wider L2 Cache MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

1: Wider Main Memory: Example 4 words (i.e. 32 byte) block Time to send address = 4 clock cycles Time to send the data word = 4 clock cycles Access time per word = 56 clock cycles Miss Penalty = No. of words x [time to: send address + send data word + access word] MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) 1: Wider Main Memory 1: For 1 word organization Miss Penalty = 4 x (4 +4+56) = 4 x (64) = 256 Clock Cycles; The memory bandwidth = bytes/clock cycle = 32/256 = 1/8 byte /cycle 2: For 4-word organization Miss Penalty = 1 x (4 +4+56) = 64 Clock Cycles; and Memory bandwidth = 32/64 = 1/2 bytes/cycle; MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

1: Wider Main Memory: Demerits L1 cache Wider L2 Cache MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) 2: Interleaved Memory MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) 2: Interleaved Memory MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) 2: Interleaved Memory bank 0 has all word whose: Address MOD 4 = 0 bank 1 has all word whose: Address MOD 4 = 1 bank 2 has all word whose: Address MOD 4 = 2 bank 3 has all word whose: Address MOD 4 = 3 Word address Bank 0 Bank 1 Bank 2 Bank 3 4 8 12 1 5 9 13 2 6 10 14 3 7 11 152 MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

2: Interleaved Memory: Example Bandwidth Calculation: bandwidth of 4 words interleaved memory using the time model as used in case of wider memory The miss penalty for 4-word interleave memory is: = time to send address + time to access + number of banks x time to send data = 4 + 56 + 4 x 4 =76 clock cycles Bandwidth = 32/76 = 0.4 byte per clock Bandwidth = 32/256= 1/8 = 0.125 byte per clock MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

3: Independent Memory Banks Memory banks offer independent accesses Multiprocessors I/O CPU with Hit under n Misses Non-blocking Caches MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

3: Independent Memory Banks Superbank Bank ……… MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

3: Independent Memory Banks An input device may use one controller and one bank The cache read may use another and The cache write still another MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Summary: Main Memory Bandwidth Using memory banks Making memory and its bus wider Doing both How many the banks should be there? MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Summary: Main Memory Bandwidth Enhancement This decision is essential to ensure that if memory is being accessed sequentially (e.g. when processing an array) then by the time you try to read a second word from a bank, the first access has finished Otherwise it will return to original bank before it has the next word ready MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Summary: Main Memory Bandwidth Enhancement 8 banks, each of 64-bit Access time of 10 clock cycle Clock cycle 1 Bank 0 after 10 clock cycles After 10 clock cycles, The bank 0 would fetch the next desired word 7 banks sequentially till the 18th clock cycle MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Summary: Main Memory Bandwidth Enhancement 18th clock Bank 0 CPU cannot start fetching Clock cycle 20 10 clock cycles again Number of bank ≥ Number of clock cycles to access word in bank MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) Virtual Memory Multiple processes Single process Exceed physical memory available MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) Virtual Memory System Increasing gap High cost of main memory Physical DRAM as a cache for the disk Single level store MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Virtual Memory system … Cont’d Single level storage Virtual Memory System Manages two levels of memory hierarchy Main memory and secondary storage Segments, named as a page MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Virtual Memory system … Cont’d Page Block Contiguous pages MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Virtual Memory System … Cont’d CPU D A B Physical Main Memory C Virtual Memory Address space Disk Virtual Addresses 4k 8k 12k 16k 20k : 24k 28k 32k 36k 40k MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Virtual Memory: Attributes Protection Relocation MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Virtual Memory: Attributes … Cont’d Protection Operate in different address space Different permissions Cannot access privileged information MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) Protection VP: Valid Page Page Tables Memory Physical Addr Read? Write? PP 9 Yes No PP 4 XXXXXXX VP 0: VP 1: VP 2: • 0: 1: N-1: Process i: Physical Addr Read? Write? PP 6 Yes PP 9 No XXXXXXX • VP 0: VP 1: VP 2: Process j: MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Virtual Memory: Attributes … Cont’d Relocation Simplifies loading of program Allows to place a program anywhere Hardware Software MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Cache verses Virtual memory Page or segment is used for block Page fault or address fault is used for miss CPU produces virtual address The virtual addresses are translated to the main memory or physical addresses MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Cache verses Virtual memory Address translation Mapping of virtual address to the physical address Page table Physical address of the segment or the page MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Cache verses Virtual memory Virtual Page No. Page offset Virtual Address Page Table Main Memory Physical Address MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Cache verses Virtual memory Replacement on cache miss Page fault The size of processor address Cache size is independent of the processor address MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Cache verses Virtual memory Secondary storage Lower-level backing store for main memory File system occupies the space on secondary storage MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Issues of Virtual Memory Design Line size Large, since disk better at transferring large blocks Associativity High, (fully associative) to minimize miss rate Write Strategy Write through or write back MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Issues of Virtual Memory Design miss rate: Extremely low. << 1% hit time: Must match cache/maim memory performance miss latency: Very high. ~20ms tag storage overhead: Low, relative to block size MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Typical System with Virtual Memory CPU 0: 1: N-1: Memory P-1: Page Table Disk Virtual Addresses Physical MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Typical System with Virtual Memory The CPU generates the Virtual Address Operating system manages a lookup table Location of the page or segment Virtual addresses to physical addresses MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Page Faults (like “Cache Misses”) Indicates virtual address not in memory OS exception handler invoked Current process suspends OS has full control over placement MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Page Faults (like “Cache Misses”) CPU Memory Page Table Disk Virtual Addresses Physical Before fault B A CPU Memory Page Table Disk Virtual Addresses Physical After fault B A MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Servicing a Page Fault: 3 steps disk Disk Memory-I/O bus Processor Cache Memory I/O controller Reg (1) Initiate Block Read MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Servicing a Page Fault: 3 steps disk Disk Memory-I/O bus Processor Cache Memory I/O controller Reg (2) DMA Transfer (1) Initiate Block Read MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Servicing a Page Fault: 3 Steps disk Disk Memory-I/O bus Processor Cache Memory I/O controller Reg (2) DMA Transfer (1) Initiate Block Read (3) Read Done MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) Summary Main memory design Methods to improve the bandwidth of main memory Concept of Virtual Memory Servicing the page fault in Virtual Memory MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)

Lec. 32 Memory Hierarchy Design (8) Allah Hafiz MAC/VU-Advanced Computer Architecture Lec. 32 Memory Hierarchy Design (8)