Computer Architecture Key Points John Morris Electrical & Computer Enginering/ Computer Science, The University of Auckland Iolanthe II drifts off Waiheke Island

Memory Bottleneck State-of-the-art processor Bulk semiconductor RAM f = 3 GHz tclock = 330ps 1-2 instructions per cycle ~25% memory reference Memory response 4 instructions x 330ps ~1.2ns needed! Bulk semiconductor RAM 100ns+ for a ‘random’ access! Processor will spend most of its time waiting for memory!

Memory Bottleneck Assume * Clock speed, f = 3GHz Cycle time, tcyc = 1/f = 330ps * 32-bit = 4 byte machine word Internal bandwidth = (bytes per word) * f = 4 * f = 12 GB/s * 64-bit PCI bus, fbus = 32 MHz !! Clearly a bottleneck! Needs to be 3 GB/s for 25% load store instructions! Arrow width (roughly) indicates data bandwidth

Cache Small, fast memory 2nd level of the memory hierarchy Typically ~50kbytes (1998) 2 cycle access time Same die as processor “Off-chip” cache possible Custom cache chip closely coupled to processor Use fast static RAM (SRAM) rather than slower dynamic RAM Several levels possible 2nd level of the memory hierarchy “Caches” most recently used memory locations “closer” to the processor closer = closer in time

Memory Bottleneck Assume * Clock speed, f = 3GHz Cycle time, tcyc = 1/f = 330ps * 32-bit = 4 byte machine word Internal bandwidth = (bytes per word) * f = 4 * f = 12 GB/s * 64-bit PCI bus, fbus = 32 MHz Cache provides small, very fast memory ‘close’ to processor ‘Close’ = close in time, ie with high bandwidth connection Arrow width (roughly) indicates data bandwidth

Memory Bottleneck Assume Small  fast * Clock speed, f = 3GHz Cycle time, tcyc = 1/f = 330ps * 32-bit = 4 byte machine word Internal bandwidth = (bytes per word) * f = 4 * f = 12 GB/s * 64-bit PCI bus, fbus = 32 MHz Small  fast Larger  slower Very large  very slow Arrow width (roughly) indicates data bandwidth

Memory hierarchy & performance Usual metric is machine cycle time, tcyc = 1/f Visible to programmer Registers < 1 cycle latency (respond in same cycle) Transparent to programmer Level 1 (L1) cache 2 cycle latency L2 cache 5-6 cycles L3 cache about 10 cycles Main memory 100+ cycles for a random access Disc > 1 ms or >106 cycles Effective memory access time, teff = S fi ti where fi = fraction of hits at level i, ti = access time at level i

Cache - organisation Direct-mapped cache Each word in the cache has a tag Assume cache size - 2k words machine words - p bits byte-addressed memory m = log2 ( p/8 ) bits not used to address words m = 2 for 32-bit machines Address format p bits p-k-m k m tag cache address byte address

Cache - organisation Direct-mapped cache A cache line data tag 2k lines memory p p-k-m Hit? p-k-m k m CPU tag cache address byte address Memory address

Cache - Conflicts Conflicts p bits p-k-m k m Two addresses separated by 2k+m will hit the same cache location p bits p-k-m k m cache address tag byte address Addresses in which these k bits are the same will map to the same cache line

Cache - Conflicts When a word is modified in cache Write-back cache Only writes data back when needed Misses Two memory accesses Write modified word back Read new word Write-through cache Low priority write to main memory is queued Processor is delayed by read only Memory write occurs in parallel with other work Instruction and necessary data fetches take priority

Cache - Write-through or write-back? Seems a good idea! but ... Multiple writes to the same location waste memory bus bandwidth Typical programs better with write-back caches however Often you can easily predict which will be best Some processors (eg PowerPC) allow you to classify memory regions as write-back or write-through

Cache - more bits Cache lines need some status bits Tag bits + .. Valid All set to false on power up Set to true as words are loaded into cache Dirty Needed by write-back cache Write- through cache always queues the write, so lines are never ‘dirty’ Cache line Tag V M Data p-k-m 1 1 p

Cache – Improving Performance Conflicts ( addresses 2k+m bytes apart ) Degrade cache performance Lower hit rate Murphy’s Law operates Addresses are never random! Some locations ‘thrash’ in cache Continually replaced and restored Alternatively Ideal cache performance depends on uniform access to all parts of memory Never happens in real programs!

Cache - Fully Associative All tags are compared at the same time Words can use any cache line

Cache - Fully Associative Each tag is compared at the same time Any match  hit Avoids ‘unnecessary’ flushing Replacement Least Recently Used - LRU Needs extra status bits Cycles since last accessed Hardware cost high Extra comparators Wider tags p-m bits vs p-k-m bits

Cache - Set Associative 2-way set associative Each line - two words two comparators only

Cache - Set Associative n-way set associative caches n can be small: 2, 4, 8 Best performance Reasonable hardware cost Most high performance processors Replacement policy LRU choice from n Reasonable LRU approximation 1 or 2 bits Set on access Cleared / decremented by timer Choose cleared word for replacement

Cache - Locality of Reference Temporal Locality Same location will be referenced again soon Access same data again Program loops - access same instruction again Caches described so far exploit temporal locality Spatial Locality Nearby locations will be referenced soon Next element of an array Next instruction of a program

Cache - Line Length Spatial Locality Use very long cache lines Fetch one datum Neighbours fetched also PowerPC 601 (Motorola/Apple/IBM) first of the single chip Power processors 64 sets 8-way set associative 32 bytes per line 32 bytes (8 instructions) fetched into instruction buffer in one cycle 64 x 8 x 32 = 16k byte total

Cache - Separate I- and D-caches Unified cache Instructions and Data in same cache Two caches - * Instructions * Data Increases total bandwidth MIPS R10000 32Kbyte Instruction; 32Kbyte Data Instruction cache is pre-decoded! (32  36bits) Data 8-word (64byte) line, 2-way set associative 256 sets Replacement policy?

Computer Architecture Memory Management Units COMPSYS 304 Computer Architecture Memory Management Units Reefed down - heading for Great Barrier Island

Memory Management Unit Virtual Address Space Each user has a “private” address space User D’s Address Space

Virtual Addresses Mappings between user space and physical memory created by OS

Memory Management Unit (MMU) Responsible for VIRTUAL  PHYSICAL address mapping Sits between CPU and cache Cache operates on Physical Addresses (mostly - some research on VA cache) VA PA MMU PA Main Mem CPU Cache D or I D or I

MMU - operation q-k

MMU - Virtual memory space Page Table Entries can also point to disc blocks Valid bit Set: page in memory address is physical page address Cleared: page “swapped out” address is disc block address MMU hardware generates page fault when swapped out page is requested Allows virtual memory space to be larger than physical memory Only “working set” is in physical memory Remainder on paging disc

Page Fault q-k

MMU – Page faults Very expensive! Gap in access times Main memory ~100+ ns Disc ~1+ ms A factor of 104 slower!! May require write-back of old (but modified) page May require reading of Page Table Entries from disc! Good way to make a system thrash!

MMU – Access control Provides additional protection to programmer Pages can be marked Read only Execute only Can prevent wayward programmes from corrupting their own programme code or vital data Protection is hardware! MMU will raise exception if illegal access attempted OS traps the exception and process it

MMU Inverted page tables Sharing Scheme which saves memory for page tables One PTE per page of physical memory Hash function used Collisions probable Possibly slower Sharing Map virtual pages for several users to same physical page Good for sharing program code Data also (read/write control provided by OS) Saves physical memory Reduces pressure on main memory

MMU TLB TLB Coverage Cache for page table entries Enables MMU to translate VA  PA in time! Can be quite small: 50-100 entries Often fully associative Small size avoids one ‘cost’ of FA cache Only 50-100 comparators needed TLB Coverage Amount of memory covered by TLB entries Size of a program for which VA  PA translation will be fast

Memory Hierarchy - Operation